Care Redesign
Relentless Reinvention

How a Cancer Center Rapidly Developed Patient-Centered Outcome Measures

Case Study · February 5, 2017

In 2014, The University of Texas MD Anderson Cancer Center created a streamlined process for developing measure sets for patient-centered outcomes, including provider-generated and patient-reported outcomes, at an accelerated pace. These comprehensive sets are integrated with electronic health records and incorporated into clinical practice, and they will underpin internal quality-improvement and external benchmarking efforts.

Key Takeaways

  1. Use a streamlined methodology to accelerate the process of developing provider-generated and patient-reported outcome measures.

  2. Arrive at precise definitions of each patient population at the start of the process, avoiding a one-size-fits-all approach.

  3. Stratify outcomes by patient-specific factors such as age, pretreatment baseline status, and patient preference.

  4. Narrow the list of outcomes and prioritize them in order to minimize the burden of survey completion for patients and of data recording for providers.

  5. Develop outcomes with an eye toward both internal quality improvement and external benchmarking purposes.

The Challenge

Existing quality measures for cancer care focus primarily on process — i.e., whether a particular standard of care was followed — rather than outcomes. Attention on patient-reported outcome measures (PROMs) is increasing, although PROMs are still much more of a focus in clinical trials than in clinical practice. The International Consortium for Health Outcomes Measurement (ICHOM) is attempting to develop outcome measure sets for all medical conditions, but many are still forthcoming. At The University of Texas MD Anderson Cancer Center, we sought to create our own rapid process for developing provider-generated outcomes and PROMs, in the absence of ICHOM rubrics for all of the conditions we treat. We expect this effort to be of value to our patients, providers, and payers, especially as value-based care (and related payment models) gain prominence.

The Goal

We articulated our goal in May 2014: to develop, at an accelerated pace, comprehensive disease-specific outcome measure sets, including provider-generated outcomes and PROMs, that could be integrated with electronic health records (EHRs) and incorporated into clinical practice. We focused on six cancer sites: breast, colorectal, head and neck, lung, pancreatic, and prostate.

The Execution

We began with an outcomes framework, developed by Harvard Business School’s Michael E. Porter, that spans the entire cycle of care for all six of our targeted cancer sites. We structured the process of developing outcome measure sets in three phases:

  1. Concept-development phase (11 to 24 hours per disease site, each spanning 1 to 2 months). To identify outcome measure concepts by cancer site, four members of the development team sought the expertise, typically for 1 hour, of multidisciplinary providers (representing surgical, medical, and radiation oncology, and sometimes ancillary services). Initial meetings involved 1 to 3 physicians to ensure a high level of individual input; larger follow-up meetings secured broader buy-in. Diverse focus groups, comprising 8 to 12 patients, were also conducted (typically by an outside consulting firm) to help identify the most important outcomes for each cancer site. (Some focus groups met as early as 2011, before formal funding in 2014.) Members of the groups were asked broad questions about the outcomes information they would want upon starting treatment, rather than questions about specific outcomes. For example, we asked head-and-neck patients to prioritize survival versus quality-of-life (QOL) outcomes, but not for specific feedback on speaking and swallowing. Notably, patients consistently prioritized QOL over survival outcomes.

    Outcomes derived from the provider meetings and focus groups were categorized, using Porter’s basic framework, as follows: (a) survival and disease control; (b) degree of recovery and functional status; (c) treatment time and access; (d) disutility of care or treatment process; and (e) long-term consequences of therapy. At subsequent meetings, the list of outcome measure concepts was refined and eventually finalized, followed by associated discussion of items for the next phase: population parameters and stratification, measurement timelines, benchmarks, and data-collection methods (including PROM surveys). We completed this phase for the last of the 6 disease sites in August 2014.
  2. Specification and validation phase (6 to 8 hours per disease site). This phase began with the creation of concrete specifications for each outcome measure concept (using the expertise of large multidisciplinary provider teams). A numerator, denominator, and inclusion/exclusion criteria were specified for each measure concept. See, for example, the specifications for the “return to normal activities of daily living” measure for breast cancer, listed in the table.
    Measure Concept Example for Breast Cancer

    Measure Concept Example for Breast Cancer. Click To Enlarge.

    The provider teams then evaluated targets and benchmarks to set reasonable performance goals. For example, the Head and Neck Center established a target that 90% of patients would begin treatment within 21 business days after their first appointment. Outcome measures were then vetted for comprehensiveness and prioritized according to which would be most beneficial to patients. PROM surveys proposed in the concept phase were narrowed to one or two that would capture the outcomes most effectively while minimizing the administrative burden for patients. We completed this phase for the last of the 6 disease sites in January 2015.

  3. Implementation phase (ongoing). This phase focuses on data collection (integrated with the clinical workflow), storage and reporting via the EHR, and real-time access to PROMs for providers. For short-term data collection and reporting, we use an electronic data-capture system with an underlying database, integrated with the EHR and with automated, routine PROM survey administration. For example, existing EHR functionality was modified to administer a PROM for head-and-neck cancer via MD Anderson’s online patient portal before each appointment. Once submitted, patient responses were immediately viewable in the EHR, and PROMs could be calculated by aggregating data in the underlying database.Long-term data collection and reporting include an EHR-based solution, via our new Epic EHR, and PROM collection through an integrated web-based patient portal (see the screenshot of a sample survey). The data will be used for ongoing internal quality improvement and, eventually, for benchmarking (potentially related to pay for performance).
FACT B+4 Questionnaire

This survey from the FACIT Measurement System is owned and copyrighted by, and is the intellectual property of, David Cella, PhD. For complete copyright terms, see Click To Enlarge.

The Team

The development team included staff with expertise in outcomes measurement, value-based care, and project management. We recruited graduate research assistants with medicine and public health backgrounds. We also solicited an array of perspectives from faculty and clinicians who had expertise in specific cancer conditions; their participation was voluntary and uncompensated. Some providers were familiar with value-based care and eager to participate; others were skeptical and needed a primer on the topic. Most were enthusiastic about the idea of routinely collecting outcomes data, including provider-generated outcomes and PROMs, via the EHR.

The Results

Patient preferences: The concept-development phase revealed that patients did not fully understand the concept of “outcomes.” Nevertheless, their feedback showed that they placed the greatest value on provider experience (with certain procedures) and on the following outcomes: complication and success rates (for certain procedures) and QOL during and after treatment (e.g., ability to work and care for family). Patients placed secondary importance on receiving information about these elements at the time of diagnosis: prognosis of disease, survival and recurrence rates, percentage of patients cured, and effects secondary to specific diseases and treatment modalities (e.g., neuropathy after chemotherapy).

Time and effort: As noted, the entire concept-development phase spanned 1 to 2 months per disease site. (As a point of comparison, in 2013, ICHOM defined a set of PROM concepts for one site — localized prostate cancer — within approximately 7 months.) Of the 11 to 24 hours spent developing concept measures, research and literature review consumed roughly 20%, meetings and discussions 25%, content preparation 40%, and administrative tasks 15%. Of the 6 to 8 hours spent per disease site during the specification and validation phase, about three-quarters were dedicated to developing the specifications and the rest to narrowing down the list of measures and prioritizing them.

Implementation is an ongoing activity. The process for integrating the use of PROM surveys into clinical workflows was developed with providers and center staff, typically during 2 to 4 one-hour meetings. Efforts to further automate survey administration and develop robust reporting will require additional time from providers and from clinic and IT staff. We estimate that each PROM will take 2 weeks of IT staff time for complete automation. Estimates of operational efficiencies in the clinical workflow are forthcoming. Formal cost analyses have not yet been conducted, but existing staff were used to implement the entire effort (i.e., no new personnel were hired).

Data workflows: We use EHR functionality to administer PROM surveys at clinically relevant time points. A clinician can use MD Anderson’s new EHR to order that a patient be directed to a web-based PROM survey via the patient portal (survey responses are then stored in the EHR). Patients can complete surveys at home or in the waiting room before their clinic appointments, and then at predetermined intervals as they complete treatment. The goal of automating survey administration and integrating survey responses within the EHR is to streamline provider review of these outcomes and, thereby, reduce the burden to clinic staff.

Timeline comparison: MD Anderson has developed its outcome sets more quickly than ICHOM has done because we used a smaller group of in-house experts, leveraged patient feedback from previous focus groups, and obtained rapid consensus during in-person meetings. ICHOM, in contrast, assembled an international team and a larger expert workgroup, and it used a formal, modified Delphi method with a structured timeline to define and obtain consensus on its final outcome set. ICHOM’s outcomes-development process is appropriate to its role as a consortium with an international mission, and MD Anderson’s process is appropriate to its needs as a treatment center. MD Anderson’s example may be useful to other U.S. institutions that are seeking ideas on how to streamline and accelerate outcomes development.



This work was supported, in part, by the National Institutes of Health through MD Anderson’s Cancer Center Support Grant CA016672 and by grant funding from The Arthur Vining Davis Foundations.

We thank Tinisha L. Mayo, MPH, MBA and Seohyun Lee, MA for their significant efforts to develop outcome measure sets; Professor Michael Porter, of Harvard Business School, and Ronald S. Walters, MD, MBA, MHA, MS for their leadership on this initiative; and the following providers at The University of Texas MD Anderson Cancer Center for their participation in developing outcome measure sets: Randal Weber, Ehab Hanna, Charles Lu, Beth Beadle, Jan Lewin, Kate Hutcheson, Kelly Hunt, Henry Kuerer, Rosa Hwang, Sharon Giordano, Carlos Barcenas, Benjamin Smith, Steven Kronowitz, Jason Fleming, Matthew Katz, Gauri Varadhachary, Christopher Crane, John Skibber, Cathy Eng, Bruce Minsky, Prajnan Das, Stephen Swisher, Reza Mehran, Ara Vaporciyan, George Simon, Ritsuko Komaki, John Davis, Surena Matin, Jeri Kim, Steven Frank, Andrew Lee, and Deborah Kuban.

This case study originally appeared in NEJM Catalyst on August 17, 2016.

Call for submissions:

Now inviting expert articles, longform articles, and case studies for peer review


A weekly email newsletter featuring the latest actionable ideas and practical innovations from NEJM Catalyst.

Learn More »

More From Care Redesign

What Is Lean Healthcare?

Lean Healthcare is the application of “lean” ideas in healthcare to minimize waste with ongoing process improvement. Learn how to use Lean to improve patient satisfaction and care outcomes while reducing costs.

Zane05_pullquote3 hospital decentralization

Do Hospitals Still Make Sense? The Case for Decentralization of Health Care

The future is here: moving care out of the hospital and into the home and community.

Sociotechnical dimensions used to analyze EHR-related health IT safety concerns

Frustrated with Your EHR? Don’t Blame Your Vendor — Safety Is a Shared Responsibility

Two informatics experts urge individuals and organizations to work together toward safer EHR-enabled patient care.

Relative Health Care System Performance and Spending in 11 High-Income Countries

From Last to First — Could the U.S. Health Care System Become the Best in the World?

The United States could achieve the best-performing health care system in the world by undertaking coordinated efforts that address four challenges.

Addressing the Prescription Opioid Crisis: Advancing Provider Education and Collaborating with All Stakeholders

Providers have a large role to play in tackling the opioid overdose epidemic, but they can’t go it alone.

Collaboration Between Doctors and Computers - Machine Learning

Lost in Thought — The Limits of the Human Mind and the Future of Medicine

It’s ironic that just when clinicians feel that there’s no time in their daily routines for thinking, the need for deep thinking is more urgent than ever.

Root Cause Analysis: Typical Domains of Root Cause: Medical Errors

Does Every Hospital Admission Deserve a Root Cause Analysis?

Will applying the RCA rubric to hospital admissions better help define and manage care?

Uniquely Identified: The Impact of a National Health Index

What does the NHI mean to a New Zealand clinician, researcher, and health care consumer?

The Intersection of Home-Based Primary Care and Home-Based Palliative Care

My Favorite Slide: The Intersection of Home-Based Primary Care and Home-Based Palliative Care

What are the overlapping provider skill sets needed to care for homebound patients?

Simplifying Person-Centered Care with Use of the Personalized Perfect Care (PPC) Bundle

Personalized Perfect Care

The Personalized Perfect Care Bundle: Making quality metrics easier to understand and more patient-centered.


A weekly email newsletter featuring the latest actionable ideas and practical innovations from NEJM Catalyst.

Learn More »


Proactively Catching the Declining Patient

A coordinated effort by UCLA leaders to identify a high-cost population with chronic kidney disease…

Coordinated Care

145 Articles

Proactively Catching the Declining Patient

A coordinated effort by UCLA leaders to identify a high-cost population with chronic kidney disease…

Assessing and Addressing Social Needs in…

Lincoln Community Health Center improved care quality by measuring and responding to upstream social and…

Insights Council

Have a voice. Join other health care leaders effecting change, shaping tomorrow.

Apply Now