Patient Engagement

Better Physician Ratings from Discussing PROs with Patients

Case Study · June 20, 2018

The UPMC orthopaedic service line began routine collection of patient-reported outcomes (PROs) from all patients seeking orthopaedic care in 2017. In a survey of patients who had seen a foot and ankle specialist, greater shares of patients who report that their physician discussed their PROs with them reported top-box scores for physician communication and shared decision-making from the Clinician and Group Consumer Assessment of Healthcare Providers and Systems (CG-CAHPS) survey.

There is a correlation between PROs and patient satisfaction; patients who reported that physicians were discussing their PRO responses during office visits reported significantly better ratings of physician communication and shared decision-making.

Key Takeaways

  1. Patient-reported outcomes can be routinely collected and reported in the electronic health record.

  2. Simply collecting patient-reported outcomes is not sufficient — clinicians must also discuss the responses during visits.

  3. On average, patients who report that their patient-reported outcome responses were discussed during a visit provide higher ratings for doctor communication and shared decision-making.

  4. Physicians may need training to discuss patient-reported outcome results with patients.

The Challenge

As part of the federal Comprehensive Care for Joint Replacement episode-based payment program, UPMC began collecting PROs from patients undergoing elective joint replacement in 2015. These data were used both to fulfill mandatory reporting to the Centers for Medicare and Medicaid Services and to allow clinicians to track improvement in hip and knee function from the patients’ perspective. Because of the operational and financial success of this program, the UPMC orthopaedic service line leadership expanded PRO collection to all patients presenting for care in 35 orthopaedic practices across all orthopaedic specialties in June of 2017.

The use of PROs originated in academic research arenas, but their use has expanded into both clinical care and population health management. Recently, PROs have become a central focus in health care outcomes with a movement toward patient-centeredness as embodied by the Patient-Centered Outcome Research Institute.

The widespread use of electronic health records (EHRs) has reduced the burden of capturing, analyzing, and acting upon PROs. For example, a recent randomized control trial found that patients undergoing chemotherapy for solid tumor cancers who self-reported symptoms electronically to a nursing team were less likely to visit the emergency room, remained on chemotherapy regimens longer, and survived longer than the standardized care group of patients.

PROs have also been shown to improve patient-physician communication and may help individualize treatment for patient-specific concerns or symptoms; these findings are primarily in oncology and behavioral health contexts.

With the expansion of PRO collection, and recognizing the potential to improve patient outcomes, UPMC stakeholders performed a quality improvement project to evaluate the impact that routine electronic collection of PROs has on patient perceptions of their orthopaedic care, using the foot and ankle subspecialty as a case study.

The Goal

Our goal was to better understand the association between PRO clinical use with patient engagement and patient experience in a general orthopaedic population.

The Execution

The UPMC Patient-Reported Outcomes Center creates systems for collecting PRO questionnaires from patients through EpicCare. UPMC began PRO collection through Epic in 2012 with formal creation of the PRO center in 2017, though many of the center functions were informally in place before that time. The Center’s purpose is to improve the management of individual patients, clinical services, and population health using patient-reported outcomes.

The Center’s services include advising on the selection and collection of PROs, managing the integration of PROs into the EHR, and developing methods and reports for clinical use, quality assurance/improvement, and population management. UPMC has been at the national forefront of implementing this functionality with 58 clinical questionnaires in use and 273 active clinic locations live with PRO collection. Patients enter data directly into their EHR by completing questionnaires via the MyUPMC patient portal or an office-based tablet computer.

An electronic PRO implementation has many advantages, including:

  1. patient data flow instantaneously into the EHR, eliminating the need for staff time dedicated to data entry and the potential of data entry error;
  2. questionnaires can be completed in advance of a scheduled visit so the time spent filling out questionnaires in the office is reduced;
  3. providers can trend patient responses to see improvements or declining health during treatment;
  4. these data fulfill requirements from regulatory agencies and payers, such as measures required before and after a procedure or surgery; and
  5. remote administration of questionnaires via the MyUPMC patient portal allows collection of follow-up questionnaires outside of regular office visits at predetermined intervals.

The UPMC orthopaedic service line was an early adopter for the electronic collection of PROs in the spine, foot and ankle, and adult joint reconstruction subspecialties. In 2017, the department proposed to expand its collection and use of PROs to include all general orthopaedic care and its subspecialties, such as sports medicine, hand and upper extremity, orthopaedic trauma, and musculoskeletal oncology. For this study, we focused on patients presenting to foot and ankle specialists.

All patients age 18 and older who saw a foot and ankle specialist between June 2017 and September 2017 with an email recorded in the EHR were sent an invitation to complete an online survey about their orthopaedic office visit. The survey included:

  1. the Altarum Consumer Engagement (ACE) questionnaire, which measures a patient’s commitment to maintaining daily health behaviors, ability to navigate the health care system, and skill at looking up medical information;
  2. two CG-CAHPS survey subsections, which measure patient perceptions of physician communication and shared decision-making;
  3. a question assessing if the patient remembered completing PRO questions (which included a screen shot reminder), and, if they did complete the PRO questions, then a follow-up question; and
  4. asking if their doctor discussed the PRO responses with them.

The Team

The evaluation team comprised the study authors (Medical Director of UPMC Patient-Reported Outcomes; Director of Innovative Quality Care Programs, UPMC; Senior Director of Quality Analytics and Performance, UPMC) in collaboration with the UPMC Chief Quality Officer and key orthopaedic stakeholders (Vice Chairmen of Education, Orthopaedics; Vice Chair of Clinical Outcomes Research, Department of Orthopaedic Surgery).

PROs Associated with Better Top-Box Satisfaction Scores at UPMC

  Click To Enlarge.

The Metrics

Invitations were sent to 4,455 patients who had a visit with one of 63 providers; 558 people completed the survey (response rate 12.5%) of whom 150 reported not completing the PRO measures, 154 reported completing the PRO measures but not discussing them with their doctor, and 254 reported both completing the PRO measures and discussing them with their doctor.

ACE scores that measured the patient’s commitment to health and ability to look up medical information were not statistically different between groups. But ACE scores that measured the patient’s ability to navigate the health care system were statistically significantly different and slightly higher in the group that reported completing PROs and discussing them with their doctor compared to the group that reported completing PROs but not discussing them with their doctor.

The percent of CG-CAHPS scores with top box of physician communication were statistically different between groups. Greater shares of respondents who reported completing PROs and discussing them with their doctor gave top-box scores for provider communication (94%) compared to those who completed but did not discuss PROs (83%) and those who did not complete the PROs (84%). The group not completing PROs and the group who completed PROs but did not discuss them with their doctor were not statistically different.

A similar pattern was seen in CG-CAHPS shared decision-making scores. Greater shares of respondents who reported completing PROs and discussing them with their doctor gave positive scores for shared decision-making (91%) compared to those who completed but did not discuss PROs (82%) and those who did not complete the PROs (81%). The group not completing PROs and the group who completed PROs but did not discuss them with their doctor were not statistically different.

Where to Start

Success requires both the efficient collection of PROs from patients and the ability for physicians to address their responses during the visit. Based on our experience, for efficient collection of PROs, we recommend direct integration into the EHR, the ability to complete questionnaires before a visit, and tablet-based collection in waiting rooms. This enables physicians to immediately access responses and scores in the electronic record.

Next Steps

Future work will focus on educating physicians to explicitly discuss PRO results during visits.

Call for submissions:

Now inviting expert articles, longform articles, and case studies for peer review


A weekly email newsletter featuring the latest actionable ideas and practical innovations from NEJM Catalyst.

Learn More »

More From Patient Engagement
PROMs Usage Is Limited but Growing

Patient Engagement Buzz Survey: PROMs Use Is Growing, but Implementation Takes Effort

While clinical outcomes lend themselves to measurement, quantifying how an individual patient experiences symptoms or a loss of function can be more challenging. One promising approach is the use of Patient-Reported Outcome Measures (PROMs).

Taxonomy of the Patient Voice Table

Taxonomy of the Patient Voice

While health care pursues the important trend of putting patients at the center of care, the terms used to describe this goal are proliferating — and muddying the discourse. This taxonomy attempts to classify some of these terms and make some distinctions.

Agarwal01_pullquote handcrafting the patient experience

Handcrafting the Patient Experience

Health care organizations can take cues from consumer-facing companies like Airbnb to creatively insert convenience and surprise into patient encounters.

Screenshot of Tidepool daily diabetes data

A Taxonomy to Engage Patients: Objectives, Design, and Patient Activation

Health information initiatives will succeed only if they focus on patients’ motivation to engage and reflect the type of engagement they seek.

Ghafur01_pullquote -digital health health care consumer patient experience

Engaging Patients Using Digital Technology — Learning from Other Industries

Providers can benefit patients and disrupt health care by learning from the experience of other industries.

Mission Hospital Virtual Sitter - Drawing of the Virtual Bed Zone and Rails

Reducing Inpatient Falls and Injury Rates by Integrating New Technology with Workflow Redesign

How Mission Hospital scaled a virtual sitter pilot and reduced unassisted falls by 44% and fall-related injuries by 40%.

Health Care Providers Should Incentivize Patients

Survey Snapshot: Patient Financial Incentives — There Are No Quick Fixes

The NEJM Catalyst Insights Council agrees that while financial incentives are a common strategy to engage patients in healthy behaviors, they are not necessarily effective.

Support of Family and Friends Is More Effective Than Clinician Support in Realizing Health Goals - From the Patient Engagement Insights Report: Why No Single Health Incentive Works.

Patient Engagement Survey: Why No Single Health Incentive Works

Initiatives to improve patient engagement come in a variety of forms. While insurers, employers, and health care providers are all involved in using financial incentives and penalties for engagement efforts, improvement in health outcomes has been elusive. Achieving that ultimate goal will usually require a combination of financial and social approaches.

Health Systems Attending the Nudge Units in Health Care Symposium - Penn Medicine

Key Insights on Launching a Nudge Unit within a Health Care System

Leaders are finding that making higher-value choices easier through subtle changes to choice architecture can have an outsized impact on medical decision-making.

Barriers to Providing an Oustanding Patient Experience

Buzz Survey Report: Patient Experience

An independent NEJM Catalyst report sponsored by University of Utah Health on barriers to achieving an excellent patient experience.


A weekly email newsletter featuring the latest actionable ideas and practical innovations from NEJM Catalyst.

Learn More »


Patient Incentives

75 Articles

Taxonomy of the Patient Voice

While health care pursues the important trend of putting patients at the center of care,…

Ripe for Disruption: Why and How…

For big tech companies like Amazon, Apple, and Google, the health care sector looks ripe…

Patients As Customers

141 Articles

Information Asymmetry: The Untapped Value of…

The knowledge and preferences that patients could — and should — share with clinicians would…

Insights Council

Have a voice. Join other health care leaders effecting change, shaping tomorrow.

Apply Now