Care Redesign
Relentless Reinvention

How a Cancer Center Rapidly Developed Patient-Centered Outcome Measures

Case Study · February 5, 2017

In 2014, The University of Texas MD Anderson Cancer Center created a streamlined process for developing measure sets for patient-centered outcomes, including provider-generated and patient-reported outcomes, at an accelerated pace. These comprehensive sets are integrated with electronic health records and incorporated into clinical practice, and they will underpin internal quality-improvement and external benchmarking efforts.

Key Takeaways

  1. Use a streamlined methodology to accelerate the process of developing provider-generated and patient-reported outcome measures.

  2. Arrive at precise definitions of each patient population at the start of the process, avoiding a one-size-fits-all approach.

  3. Stratify outcomes by patient-specific factors such as age, pretreatment baseline status, and patient preference.

  4. Narrow the list of outcomes and prioritize them in order to minimize the burden of survey completion for patients and of data recording for providers.

  5. Develop outcomes with an eye toward both internal quality improvement and external benchmarking purposes.

The Challenge

Existing quality measures for cancer care focus primarily on process — i.e., whether a particular standard of care was followed — rather than outcomes. Attention on patient-reported outcome measures (PROMs) is increasing, although PROMs are still much more of a focus in clinical trials than in clinical practice. The International Consortium for Health Outcomes Measurement (ICHOM) is attempting to develop outcome measure sets for all medical conditions, but many are still forthcoming. At The University of Texas MD Anderson Cancer Center, we sought to create our own rapid process for developing provider-generated outcomes and PROMs, in the absence of ICHOM rubrics for all of the conditions we treat. We expect this effort to be of value to our patients, providers, and payers, especially as value-based care (and related payment models) gain prominence.

The Goal

We articulated our goal in May 2014: to develop, at an accelerated pace, comprehensive disease-specific outcome measure sets, including provider-generated outcomes and PROMs, that could be integrated with electronic health records (EHRs) and incorporated into clinical practice. We focused on six cancer sites: breast, colorectal, head and neck, lung, pancreatic, and prostate.

The Execution

We began with an outcomes framework, developed by Harvard Business School’s Michael E. Porter, that spans the entire cycle of care for all six of our targeted cancer sites. We structured the process of developing outcome measure sets in three phases:

  1. Concept-development phase (11 to 24 hours per disease site, each spanning 1 to 2 months). To identify outcome measure concepts by cancer site, four members of the development team sought the expertise, typically for 1 hour, of multidisciplinary providers (representing surgical, medical, and radiation oncology, and sometimes ancillary services). Initial meetings involved 1 to 3 physicians to ensure a high level of individual input; larger follow-up meetings secured broader buy-in. Diverse focus groups, comprising 8 to 12 patients, were also conducted (typically by an outside consulting firm) to help identify the most important outcomes for each cancer site. (Some focus groups met as early as 2011, before formal funding in 2014.) Members of the groups were asked broad questions about the outcomes information they would want upon starting treatment, rather than questions about specific outcomes. For example, we asked head-and-neck patients to prioritize survival versus quality-of-life (QOL) outcomes, but not for specific feedback on speaking and swallowing. Notably, patients consistently prioritized QOL over survival outcomes.

    Outcomes derived from the provider meetings and focus groups were categorized, using Porter’s basic framework, as follows: (a) survival and disease control; (b) degree of recovery and functional status; (c) treatment time and access; (d) disutility of care or treatment process; and (e) long-term consequences of therapy. At subsequent meetings, the list of outcome measure concepts was refined and eventually finalized, followed by associated discussion of items for the next phase: population parameters and stratification, measurement timelines, benchmarks, and data-collection methods (including PROM surveys). We completed this phase for the last of the 6 disease sites in August 2014.
  2. Specification and validation phase (6 to 8 hours per disease site). This phase began with the creation of concrete specifications for each outcome measure concept (using the expertise of large multidisciplinary provider teams). A numerator, denominator, and inclusion/exclusion criteria were specified for each measure concept. See, for example, the specifications for the “return to normal activities of daily living” measure for breast cancer, listed in the table.
    Measure Concept Example for Breast Cancer

    Measure Concept Example for Breast Cancer. Click To Enlarge.

    The provider teams then evaluated targets and benchmarks to set reasonable performance goals. For example, the Head and Neck Center established a target that 90% of patients would begin treatment within 21 business days after their first appointment. Outcome measures were then vetted for comprehensiveness and prioritized according to which would be most beneficial to patients. PROM surveys proposed in the concept phase were narrowed to one or two that would capture the outcomes most effectively while minimizing the administrative burden for patients. We completed this phase for the last of the 6 disease sites in January 2015.

  3. Implementation phase (ongoing). This phase focuses on data collection (integrated with the clinical workflow), storage and reporting via the EHR, and real-time access to PROMs for providers. For short-term data collection and reporting, we use an electronic data-capture system with an underlying database, integrated with the EHR and with automated, routine PROM survey administration. For example, existing EHR functionality was modified to administer a PROM for head-and-neck cancer via MD Anderson’s online patient portal before each appointment. Once submitted, patient responses were immediately viewable in the EHR, and PROMs could be calculated by aggregating data in the underlying database.Long-term data collection and reporting include an EHR-based solution, via our new Epic EHR, and PROM collection through an integrated web-based patient portal (see the screenshot of a sample survey). The data will be used for ongoing internal quality improvement and, eventually, for benchmarking (potentially related to pay for performance).
FACT B+4 Questionnaire

This survey from the FACIT Measurement System is owned and copyrighted by, and is the intellectual property of, David Cella, PhD. For complete copyright terms, see Click To Enlarge.

The Team

The development team included staff with expertise in outcomes measurement, value-based care, and project management. We recruited graduate research assistants with medicine and public health backgrounds. We also solicited an array of perspectives from faculty and clinicians who had expertise in specific cancer conditions; their participation was voluntary and uncompensated. Some providers were familiar with value-based care and eager to participate; others were skeptical and needed a primer on the topic. Most were enthusiastic about the idea of routinely collecting outcomes data, including provider-generated outcomes and PROMs, via the EHR.

The Results

Patient preferences: The concept-development phase revealed that patients did not fully understand the concept of “outcomes.” Nevertheless, their feedback showed that they placed the greatest value on provider experience (with certain procedures) and on the following outcomes: complication and success rates (for certain procedures) and QOL during and after treatment (e.g., ability to work and care for family). Patients placed secondary importance on receiving information about these elements at the time of diagnosis: prognosis of disease, survival and recurrence rates, percentage of patients cured, and effects secondary to specific diseases and treatment modalities (e.g., neuropathy after chemotherapy).

Time and effort: As noted, the entire concept-development phase spanned 1 to 2 months per disease site. (As a point of comparison, in 2013, ICHOM defined a set of PROM concepts for one site — localized prostate cancer — within approximately 7 months.) Of the 11 to 24 hours spent developing concept measures, research and literature review consumed roughly 20%, meetings and discussions 25%, content preparation 40%, and administrative tasks 15%. Of the 6 to 8 hours spent per disease site during the specification and validation phase, about three-quarters were dedicated to developing the specifications and the rest to narrowing down the list of measures and prioritizing them.

Implementation is an ongoing activity. The process for integrating the use of PROM surveys into clinical workflows was developed with providers and center staff, typically during 2 to 4 one-hour meetings. Efforts to further automate survey administration and develop robust reporting will require additional time from providers and from clinic and IT staff. We estimate that each PROM will take 2 weeks of IT staff time for complete automation. Estimates of operational efficiencies in the clinical workflow are forthcoming. Formal cost analyses have not yet been conducted, but existing staff were used to implement the entire effort (i.e., no new personnel were hired).

Data workflows: We use EHR functionality to administer PROM surveys at clinically relevant time points. A clinician can use MD Anderson’s new EHR to order that a patient be directed to a web-based PROM survey via the patient portal (survey responses are then stored in the EHR). Patients can complete surveys at home or in the waiting room before their clinic appointments, and then at predetermined intervals as they complete treatment. The goal of automating survey administration and integrating survey responses within the EHR is to streamline provider review of these outcomes and, thereby, reduce the burden to clinic staff.

Timeline comparison: MD Anderson has developed its outcome sets more quickly than ICHOM has done because we used a smaller group of in-house experts, leveraged patient feedback from previous focus groups, and obtained rapid consensus during in-person meetings. ICHOM, in contrast, assembled an international team and a larger expert workgroup, and it used a formal, modified Delphi method with a structured timeline to define and obtain consensus on its final outcome set. ICHOM’s outcomes-development process is appropriate to its role as a consortium with an international mission, and MD Anderson’s process is appropriate to its needs as a treatment center. MD Anderson’s example may be useful to other U.S. institutions that are seeking ideas on how to streamline and accelerate outcomes development.



This work was supported, in part, by the National Institutes of Health through MD Anderson’s Cancer Center Support Grant CA016672 and by grant funding from The Arthur Vining Davis Foundations.

We thank Tinisha L. Mayo, MPH, MBA and Seohyun Lee, MA for their significant efforts to develop outcome measure sets; Professor Michael Porter, of Harvard Business School, and Ronald S. Walters, MD, MBA, MHA, MS for their leadership on this initiative; and the following providers at The University of Texas MD Anderson Cancer Center for their participation in developing outcome measure sets: Randal Weber, Ehab Hanna, Charles Lu, Beth Beadle, Jan Lewin, Kate Hutcheson, Kelly Hunt, Henry Kuerer, Rosa Hwang, Sharon Giordano, Carlos Barcenas, Benjamin Smith, Steven Kronowitz, Jason Fleming, Matthew Katz, Gauri Varadhachary, Christopher Crane, John Skibber, Cathy Eng, Bruce Minsky, Prajnan Das, Stephen Swisher, Reza Mehran, Ara Vaporciyan, George Simon, Ritsuko Komaki, John Davis, Surena Matin, Jeri Kim, Steven Frank, Andrew Lee, and Deborah Kuban.

This case study originally appeared in NEJM Catalyst on August 17, 2016.

New Call for Submissions ­to NEJM Catalyst


A weekly email newsletter featuring the latest actionable ideas and practical innovations from NEJM Catalyst.

Learn More »

More From Care Redesign
Relentless Reinvention

The Move to Value-Based Care in Navy Medicine

Achieving the mission of Navy Medicine to “keep the Navy and Marine Corps family ready, healthy, and on the job” requires rethinking current health care delivery models.

Relentless Reinvention

Improving Access to Specialist Expertise via eConsult in a Safety-Net Health System

Electronic referral system supports communication between primary care and specialty providers.

Relentless Reinvention

Redesigning the Delivery of Specialty Care Within Newly Formed Hospital Networks

As the trend toward hospital mergers and consolidations continues, how can newly formed health care networks optimize their delivery of specialty care? They will need to consider a redesign of service lines that includes both centralizing and decentralizing strategies.

Relentless Reinvention

Survey Snapshot: Genomic Data Is Far from Clinical Use

NEJM Catalyst Insights Council members say that clinical and cost data will continue to be the most useful data sources.

Relentless Reinvention

Why Every Health Care Organization Needs a Data Science Strategy

Data science strategy can help providers tap into the power of their data, improve its quality, and keep it safe.

Relentless Reinvention

Learning to Drive — Early Exposure to End-of-Life Conversations in Medical Training

The importance of listening to the patient at the end of life.

Relentless Reinvention

Care Redesign Survey: What Data Can Really Do for Health Care

NEJM Catalyst Insights Council members are shifting from disillusionment over the unfulfilled promises of big data to a more realistic vision of how sophisticated analytics can transform health care delivery.

Relentless Reinvention

Can We Achieve Scale in Innovation?

Innovation and scale are inextricably tied to the future success and sustainability of health care providers.

Relentless Reinvention

Measures Only Get Better When You Use Them

Optimism, innovation, and how the two go together.

Relentless Reinvention

P4 Medicine and the Democratization of Health Care

“P4 health care” — predictive, preventive, personalized, and participatory — will use the astonishing power of systems medicine and big data to bring cutting-edge scientific wellness to everyone, improving health and saving money.


A weekly email newsletter featuring the latest actionable ideas and practical innovations from NEJM Catalyst.

Learn More »


Focusing on High-Cost Patients — The…

A focus on high-cost patients may not only fail to contain health care spending, but…

Care Integration

48 Articles

How Multi-Specialty Hubs Fill a Major…

Kaiser Permanente, Mid-Atlantic States identified a niche for patients seeking immediate care and found a…

Quality Management

88 Articles

The Move to Value-Based Care in…

Achieving the mission of Navy Medicine to “keep the Navy and Marine Corps family ready,…

Insights Council

Have a voice. Join other health care leaders effecting change, shaping tomorrow.

Apply Now