Patient Engagement

A/B Testing Health Behavior

Article · March 10, 2017

Medicine has been accused of implementing new interventions without a strong underlying evidence base, and that impulse for quick implementation is understandable, says Scott Halpern, Deputy Director for the University of Pennsylvania’s Center for Health Incentives and Behavioral Economics. “We have sick people. We need to do something. We can’t sit idly by,” he says. “But I wonder if that instinct to not just stand there — to do something — might have some unintended consequences, crowding out the potential for greater innovation that has better evidence to support it.” What can we learn from other industries that have grappled with this tension?

A decade ago, there was almost no A/B testing in the corporate sector, notes Harvard economist David Laibson. “The CEO or someone else had a great idea and they implemented it, and then they had anecdotes about how great it was — and that was the way you made progress,” he says, adding, “a lot of that was a huge waste of money.”

This attitude first started changing in Silicon Valley, where companies became experts at treatment and control. Today, they’ll immediately start randomizing a new idea and maybe within a day have enough data to know whether to proceed with it. “We can’t do that in health care, of course, because we don’t have the control and the systems are complicated,” says Laibson. “It’s not that we should aspire to 24-hour experiments, but we should aspire to experiments, to actually randomizing and figuring out what works before we roll things out nationally and embrace them as best practice.”

“If we take the treatment-and-control approach, we’re going to make a lot more progress quickly despite the fact that it’s frustrating,” says Laibson, “because when you commit to running randomized control trials, it slows down the legislation, it slows down the national adoption, because you’ve got to wait a year for those results to come in. But that’s a year that’s worth waiting.”

Those who are working to add more value to the health system may also need some training to help them understand what the best new design might be, adds Dartmouth College marketing expert Punam Keller. She has worked with the Centers for Disease Control and Prevention on HealthCommWorks, which uses an empirical algorithm to help you tailor your messages to your audience’s needs. The website also offers ProofWorks, which is “basically like PhD 101,” says Keller. It includes information on good design, control, random assignment, sample sizes, metrics validation, intervention and testing time, and outcome variables. “Those kinds of things are important if we are to improve the rigor of the data that we have in order to improve our collective understanding,” she says.

Watch the video:

From the NEJM Catalyst event Patient Engagement: Behavioral Strategies for Better Health at the University of Pennsylvania, February 25, 2016.

New Call for Submissions ­to NEJM Catalyst

Connect

A weekly email newsletter featuring the latest actionable ideas and practical innovations from NEJM Catalyst.

Learn More »

More From Patient Engagement
Face-to-Face Group Sessions Are the Most Useful Mode of Communication in Health Social Networks

Survey Snapshot: Patient Networks Are Important for Outcomes

NEJM Catalyst Insights Council members overwhelmingly agree that social networks can impact chronic diseases and healthy behaviors, but time commitment and funding prevent their prevalence.

Consumerism in Health Care

Will personalized interventions transform how we improve an individual’s health? A frequent-flyer expert believes that it will.

Chronic Disease Management and Healthy Behavior Promotion Are the Best Uses of Health Social Networks

Patient Engagement Survey: Social Networks to Improve Patient Health

Social networks — both face-to-face and virtual — have been shown to positively impact behaviors such as weight loss and smoking cessation. Is the health care industry ready to more widely leverage this mechanism for increasing healthy behaviors among patients?

Patient Inducements — High Graft or High Value?

Can health care be more like Uber and Lyft?

Consumer Engagement: New Tools and Capabilities for Health System Marketing

Novant Health is using research and data analytics to identify and engage new consumers, develop more relevant products and services, and support its mission and brand promise.

The Patient as Consumer and the Measurement of Bedside Manner

Many physicians do not believe patient satisfaction is a legitimate pursuit. Yet they must meet consumers of health care where they are — on Internet ratings sites.

Analysis of health care utilization cost and patient functioning in PCIC social interventions_new

Social Interventions Can Lower Costs and Improve Outcomes

We can better serve our under-resourced patients by helping them access treatments and social interventions we already know to be effective.

Transparency and Trust — Online Patient Reviews of Physicians

Transparency of validated data about delivery-system performance has the power to change the culture of health care.

Patient-Researchers and Physician-Consultants Confront Chronic Disease

Personal-use sensors and data applications allow patients with Type 1 diabetes and other chronic diseases to take charge of managing their health. This is the future of chronic disease management.

Geisinger’s Refund Promise: Where Things Stand After One Year

Geisinger’s radical refund promise at one year — no evidence of financial catastrophe but a good return on patient trust.

Connect

A weekly email newsletter featuring the latest actionable ideas and practical innovations from NEJM Catalyst.

Learn More »

Topics

Patient-Centered Care

190 Articles

In-Person Health Care as Option B

What if health care were designed so that in-person visits were the second, third, or…

Patient Incentives

56 Articles

Survey Snapshot: Patient Networks Are Important…

NEJM Catalyst Insights Council members overwhelmingly agree that social networks can impact chronic diseases and…

In-Person Health Care as Option B

What if health care were designed so that in-person visits were the second, third, or…

Insights Council

Have a voice. Join other health care leaders effecting change, shaping tomorrow.

Apply Now