Patient Engagement

Evaluating Complex Care Programs: Is It a Zero-Sum Game?

Article · May 15, 2017

Complex populations in health care are often defined as groups of individuals with co-occurring medical and behavioral health diagnoses as well as significant social challenges. Large sums of money have been devoted to developing interventions focused on “bending the cost curve” by attenuating the rapidly rising costs of caring for these populations. Whether such programs actually save health care dollars remains an open question because rigorous evaluation data are limited. In the absence of empirical evidence regarding program models that are worth replicating, new program models continue to abound and evaluators continue to evaluate.

The need for more rigorous evaluation presents a unique opportunity for policymakers and evaluators (and we include ourselves here) to shift focus away from traditional health system-centric outcomes such as emergency department (ED) visits, inpatient utilization, and associated costs of care or reimbursements. These outcomes may seem logical: complex populations generate high health care costs, so programs should be able to show a rapid return on investment. Yet while many programs for complex populations are implemented in health care settings, most are designed to address individuals’ needs both inside and outside of the health care system, and evaluations must take this consideration into account. Some programs and delivery systems targeting complex populations, including a Pay For Success project in Santa Clara County and the Washington State Health Care Authority (discussed in more detail below), link data across health and social domains in order to more comprehensively illustrate program impact. Yet, these programs are the exception rather than the rule. Why is this, and what can be done?

The Challenge of Defining Complex Populations and Interventions

Individuals who suffer from complex combinations of medical, behavioral, and social challenges are often defined first in economic or utilization terms as “superutilizers,” the “top 5%,” or the “top decile.” Yet despite these simple labels, complex populations are uniquely heterogeneous. In addition to chronic medical conditions, mental illness, and substance use disorders, these individuals face other important challenges, including mistrust of the medical system, lack of social support, homelessness and housing instability, cognitive decline, criminal justice involvement (including encounters with law enforcement, jail, and prison), a history of adverse childhood events (ACEs), and poverty.

Most individuals who work within complex care programs understand that heterogeneity is the most consistent characteristic of complex populations. Thus, we design programs to deliver a wide range of health and social services that can be tailored to the needs of each unique individual. For one person, the top priority might be finding permanent housing. For another, it might be fixing a broken wheelchair, handling an outstanding arrest warrant, or securing a detoxification bed. Blood sugar control and medication adherence may not be immediate goals, but they must also be addressed. The wide range of potential program interventions makes outcomes measurement difficult. Can we lump one individual’s outcome of obtaining permanent supportive housing with the other’s outcome of a repaired wheelchair? Does one outcome have a higher “value” than the other? Should we look at the sum of all of these parts in terms of the impact on a population as a whole, or should we instead evaluate the impacts of a specific arm of a program on a subset of patients? Furthermore, although these underlying social challenges can lead to health system use, will addressing them reduce such use in the near term? And should reduced use of health care resources be our only goal?

Measuring Comprehensive Impact Over Time

Using administrative health care data to evaluate complex care programs is inexpensive and relatively simple, and the results are easy to interpret. When ED visits and inpatient bed days go up or down, so do related costs or reimbursements. While the measurement of health service usage and related costs is necessary, restricting outcomes to these domains oversimplifies complex care programs and the unique circumstances facing the individuals who are enrolled in them. Individuals in such programs spend most of their days engaged in a wide range of activities that take place outside of the hospital or clinic, such as  obtaining a new form of identification, participating in a harm reduction program, reuniting with family, opening a bank account, attending a housing interview, or even taking part in discharge planning from jail.

Why is so little of this important work accounted for in program evaluations? First, these types of outcomes are not reflected in administrative data. They can be time-consuming and expensive to measure, and funders are often most interested in outcomes that point to rapid return on investment within health care. However, unrealistically short evaluation time frames will not capture longer-term program impacts and may lead to false conclusions about program effectiveness. We know firsthand that populations with complex needs have significant access barriers and health behaviors that can take many months — if not years — to change. In addition, pent-up demand for needed health and social services can drive up costs in the short term after program enrollment. For example, obtaining housing for a homeless individual with complex care needs often requires several months. In turn, health benefits and decreased use of acute health services allowed by a stable living environment will take additional time.

Second, unlike metrics such as blood pressure and cholesterol, outcomes related to social determinants of health often do not have the benefit of clearly defined numerical targets tied to meaningful clinical or longevity endpoints. Individuals who are enrolled in complex care programs may experience a higher quality of life via improved access to needed health and social services, such as palliative care, adult day programs, hospice, or housing, whereas individuals who are not enrolled in these programs may die earlier, with a lower lifetime cost to the health care system. Few evaluations of complex care programs, if any, have attempted to place a value on improved quality life and reduced mortality, but these metrics are routinely used to justify the use of costly oncological treatments and other pharmaceuticals. The evaluation of outcomes related to quality of life and the adherence to advanced directives regarding end-of-life care are critical considerations in this population.

Integrating Non-Medical Data to Form a More Person-Centered View

It is well documented that many individuals who qualify for complex care programs use services across multiple health and social care systems, including hospitals, clinics, substance use and mental health services, ambulance services, community-based agencies, and jail. In most jurisdictions, these systems are not interconnected, so evaluations historically have been unable to determine program impact across the full spectrum of affected services. Instead, many evaluations are restricted to one segment of care and do not evaluate the impact of a program on other important, and often costly, services. Variables such as the financial strain that an individual places on community-based social stabilization programs, the growth in (or lack of) an individual’s economic productivity, or the number of days that an individual spends in jail are all factors that a health insurer or delivery system cannot address alone (or financially benefit from).

While evaluating complex care programs is challenging, a few jurisdictions are ahead of the curve and are addressing the issues that have, to date, inhibited rigorous evaluation. The Washington State Health Care Authority has united more than 10 data sources to form a comprehensive data set for individuals insured by the state Medicaid program. This powerful tool has allowed the state to begin to evaluate multiple state programs, without having to restrict outcome measurement to the narrow lens of the health care system. Santa Clara County in California has taken the groundbreaking yet readily achievable step of merging county health care data with county data from behavioral health services (i.e., substance use and mental health services), housing services, and the criminal justice system. These actions have allowed evaluators to measure the impact of different programs (including one focused on providing permanent supportive housing for frequent users of county services who are chronically homeless) across multiple domains that include but are not restricted to health care.

Three Critical Principles for Complex Care Evaluations of the Future

As we look toward future evaluations of complex care programs, it will be necessary for delivery systems, evaluators, and policymakers to prioritize three critical principles:

  1. Allow adequate time to evaluate impact. In order to accurately capture program effects, outcomes related to service use and related costs must be measured over an extended time period. In our experience, evaluations will require at least 24 months of data for program enrollees. Evaluations must take a longer-term view.
  2. Look beyond dollars. Improving and prolonging the lives of individuals with advanced chronic illness and disability may cost more than allowing some of our society’s most vulnerable and expensive individuals to die. Evaluators can incorporate existing validated measures that assess such factors as quality of life, the presence of advanced directives (and whether they were followed), and the setting in which an individual dies (e.g., in hospice or at home versus in a hospital). These and other measures can provide policymakers and other stakeholders with important data that are difficult to put a price on.
  3. Link existing datasets to capture more comprehensive program effects. Finally, we must leverage existing administrative datasets to measure cross-sector spending and social There is increasing recognition among policymakers and health services researchers that these types of data connections will be necessary in order to make valid determinations about the impact of programs for complex populations.

As we reflect on the last decade of extensive work aimed at improving care for complex populations, we are struck by the limited body of scalable evidence that can guide future program implementation. But we are also encouraged by the momentum that exists among researchers, policymakers, providers, patients, and other stakeholders to more rigorously and comprehensively evaluate the impact of complex care programs. We can and must do better.


Acknowledgments: We thank the Center for Health Care Strategies for the opportunity to participate in the Complex Care Innovation Laboratory, which provides a peer forum to foster new ideas for improving outcomes for low-income populations with complex health and social needs.

Call for submissions:

Now inviting expert articles, longform articles, and case studies for peer review


A weekly email newsletter featuring the latest actionable ideas and practical innovations from NEJM Catalyst.

Learn More »

More From Patient Engagement

Data Graphic: CMOs and Staff Physicians Lead Patient Engagement Efforts

Survey respondents say CMOs and staff physicians head patient engagement efforts within health care organizations, but it’s important to note who does the ranking.

How Chronic Pain Treatment Falls Short of Patient-Centered Care

Healing the opioid epidemic requires nothing less than a comprehensive theory of patient health.

patient engagement vs patient experience

Patient Engagement versus Patient Experience

The difference between the two terms is muddled but important, says Cleveland Clinic’s Chief Experience Officer.

Toyin Ajayi Talk Still: The Difference Between Noncompliance and Defiance

The Difference Between Noncompliance and Defiance

How do we align our goals for patient engagement with even the most complex, difficult patients?

Hardwiring Patient Engagement to Deliver Better Health

Imagine a world in which health care was not “one size fits all,” but customized for each patient individually.

Health Care — A Final Frontier for Design

Design must move beyond narrow projects and encompass complex systems.

Engineering Social Incentives for Health

It’s important to engineer social engagements that promote health, but we must also test their acceptability and effectiveness.

When — and How — Can Incentives Actually Work?

Incentives can work if they’re designed a bit better.

My Favorite Slide: When Patients Are Prevented from Engaging

If patient engagement is about action, why don’t we let them speak?

Using a “Nursing Bundle” to Achieve Consistent Patient Experiences Across a Multi-Hospital System

How Geisinger addressed inconsistent patient experience by creating a nursing bundle that helped make patients’ experiences more consistent.


A weekly email newsletter featuring the latest actionable ideas and practical innovations from NEJM Catalyst.

Learn More »


Patient-Centered Care

278 Articles

Measuring Performance of OpenNotes Initiatives to…

How a New York safety-net health system used data science to identify obstacles to OpenNotes…

Information Asymmetry: The Untapped Value of…

The knowledge and preferences that patients could — and should — share with clinicians would…

Patient Incentives

75 Articles

Taxonomy of the Patient Voice

While health care pursues the important trend of putting patients at the center of care,…

Insights Council

Have a voice. Join other health care leaders effecting change, shaping tomorrow.

Apply Now