Patient Engagement

Better Physician Ratings from Discussing PROs with Patients

Case Study · June 20, 2018

The UPMC orthopaedic service line began routine collection of patient-reported outcomes (PROs) from all patients seeking orthopaedic care in 2017. In a survey of patients who had seen a foot and ankle specialist, greater shares of patients who report that their physician discussed their PROs with them reported top-box scores for physician communication and shared decision-making from the Clinician and Group Consumer Assessment of Healthcare Providers and Systems (CG-CAHPS) survey.

There is a correlation between PROs and patient satisfaction; patients who reported that physicians were discussing their PRO responses during office visits reported significantly better ratings of physician communication and shared decision-making.

Key Takeaways

  1. Patient-reported outcomes can be routinely collected and reported in the electronic health record.

  2. Simply collecting patient-reported outcomes is not sufficient — clinicians must also discuss the responses during visits.

  3. On average, patients who report that their patient-reported outcome responses were discussed during a visit provide higher ratings for doctor communication and shared decision-making.

  4. Physicians may need training to discuss patient-reported outcome results with patients.

The Challenge

As part of the federal Comprehensive Care for Joint Replacement episode-based payment program, UPMC began collecting PROs from patients undergoing elective joint replacement in 2015. These data were used both to fulfill mandatory reporting to the Centers for Medicare and Medicaid Services and to allow clinicians to track improvement in hip and knee function from the patients’ perspective. Because of the operational and financial success of this program, the UPMC orthopaedic service line leadership expanded PRO collection to all patients presenting for care in 35 orthopaedic practices across all orthopaedic specialties in June of 2017.

The use of PROs originated in academic research arenas, but their use has expanded into both clinical care and population health management. Recently, PROs have become a central focus in health care outcomes with a movement toward patient-centeredness as embodied by the Patient-Centered Outcome Research Institute.

The widespread use of electronic health records (EHRs) has reduced the burden of capturing, analyzing, and acting upon PROs. For example, a recent randomized control trial found that patients undergoing chemotherapy for solid tumor cancers who self-reported symptoms electronically to a nursing team were less likely to visit the emergency room, remained on chemotherapy regimens longer, and survived longer than the standardized care group of patients.

PROs have also been shown to improve patient-physician communication and may help individualize treatment for patient-specific concerns or symptoms; these findings are primarily in oncology and behavioral health contexts.

With the expansion of PRO collection, and recognizing the potential to improve patient outcomes, UPMC stakeholders performed a quality improvement project to evaluate the impact that routine electronic collection of PROs has on patient perceptions of their orthopaedic care, using the foot and ankle subspecialty as a case study.

The Goal

Our goal was to better understand the association between PRO clinical use with patient engagement and patient experience in a general orthopaedic population.

The Execution

The UPMC Patient-Reported Outcomes Center creates systems for collecting PRO questionnaires from patients through EpicCare. UPMC began PRO collection through Epic in 2012 with formal creation of the PRO center in 2017, though many of the center functions were informally in place before that time. The Center’s purpose is to improve the management of individual patients, clinical services, and population health using patient-reported outcomes.

The Center’s services include advising on the selection and collection of PROs, managing the integration of PROs into the EHR, and developing methods and reports for clinical use, quality assurance/improvement, and population management. UPMC has been at the national forefront of implementing this functionality with 58 clinical questionnaires in use and 273 active clinic locations live with PRO collection. Patients enter data directly into their EHR by completing questionnaires via the MyUPMC patient portal or an office-based tablet computer.

An electronic PRO implementation has many advantages, including:

  1. patient data flow instantaneously into the EHR, eliminating the need for staff time dedicated to data entry and the potential of data entry error;
  2. questionnaires can be completed in advance of a scheduled visit so the time spent filling out questionnaires in the office is reduced;
  3. providers can trend patient responses to see improvements or declining health during treatment;
  4. these data fulfill requirements from regulatory agencies and payers, such as measures required before and after a procedure or surgery; and
  5. remote administration of questionnaires via the MyUPMC patient portal allows collection of follow-up questionnaires outside of regular office visits at predetermined intervals.

The UPMC orthopaedic service line was an early adopter for the electronic collection of PROs in the spine, foot and ankle, and adult joint reconstruction subspecialties. In 2017, the department proposed to expand its collection and use of PROs to include all general orthopaedic care and its subspecialties, such as sports medicine, hand and upper extremity, orthopaedic trauma, and musculoskeletal oncology. For this study, we focused on patients presenting to foot and ankle specialists.

All patients age 18 and older who saw a foot and ankle specialist between June 2017 and September 2017 with an email recorded in the EHR were sent an invitation to complete an online survey about their orthopaedic office visit. The survey included:

  1. the Altarum Consumer Engagement (ACE) questionnaire, which measures a patient’s commitment to maintaining daily health behaviors, ability to navigate the health care system, and skill at looking up medical information;
  2. two CG-CAHPS survey subsections, which measure patient perceptions of physician communication and shared decision-making;
  3. a question assessing if the patient remembered completing PRO questions (which included a screen shot reminder), and, if they did complete the PRO questions, then a follow-up question; and
  4. asking if their doctor discussed the PRO responses with them.

The Team

The evaluation team comprised the study authors (Medical Director of UPMC Patient-Reported Outcomes; Director of Innovative Quality Care Programs, UPMC; Senior Director of Quality Analytics and Performance, UPMC) in collaboration with the UPMC Chief Quality Officer and key orthopaedic stakeholders (Vice Chairmen of Education, Orthopaedics; Vice Chair of Clinical Outcomes Research, Department of Orthopaedic Surgery).

PROs Associated with Better Top-Box Satisfaction Scores at UPMC

  Click To Enlarge.

The Metrics

Invitations were sent to 4,455 patients who had a visit with one of 63 providers; 558 people completed the survey (response rate 12.5%) of whom 150 reported not completing the PRO measures, 154 reported completing the PRO measures but not discussing them with their doctor, and 254 reported both completing the PRO measures and discussing them with their doctor.

ACE scores that measured the patient’s commitment to health and ability to look up medical information were not statistically different between groups. But ACE scores that measured the patient’s ability to navigate the health care system were statistically significantly different and slightly higher in the group that reported completing PROs and discussing them with their doctor compared to the group that reported completing PROs but not discussing them with their doctor.

The percent of CG-CAHPS scores with top box of physician communication were statistically different between groups. Greater shares of respondents who reported completing PROs and discussing them with their doctor gave top-box scores for provider communication (94%) compared to those who completed but did not discuss PROs (83%) and those who did not complete the PROs (84%). The group not completing PROs and the group who completed PROs but did not discuss them with their doctor were not statistically different.

A similar pattern was seen in CG-CAHPS shared decision-making scores. Greater shares of respondents who reported completing PROs and discussing them with their doctor gave positive scores for shared decision-making (91%) compared to those who completed but did not discuss PROs (82%) and those who did not complete the PROs (81%). The group not completing PROs and the group who completed PROs but did not discuss them with their doctor were not statistically different.

Where to Start

Success requires both the efficient collection of PROs from patients and the ability for physicians to address their responses during the visit. Based on our experience, for efficient collection of PROs, we recommend direct integration into the EHR, the ability to complete questionnaires before a visit, and tablet-based collection in waiting rooms. This enables physicians to immediately access responses and scores in the electronic record.

Next Steps

Future work will focus on educating physicians to explicitly discuss PRO results during visits.

Call for submissions:

Now inviting expert articles, longform articles, and case studies for peer review


A weekly email newsletter featuring the latest actionable ideas and practical innovations from NEJM Catalyst.

Learn More »

More From Patient Engagement
Barriers to Providing an Oustanding Patient Experience

Buzz Survey Report: Patient Experience

An independent NEJM Catalyst report sponsored by University of Utah Health on barriers to achieving an excellent patient experience.

The Patient Engagement Capacity Framework

The Patient Engagement Capacity Model: What Factors Determine a Patient’s Ability to Engage?

Patient engagement assessments often don’t dig deep enough to identify why patients don’t participate in their own health care. We present a new model to help providers pinpoint the reasons for lack of engagement and address them more effectively.

Organizational Mindset Is the Biggest Barrier to Engaging Patients as Consumers

Survey Snapshot: The Patient-Physician Relationship Is Key

Both parties involved in a consumer-facing transaction have access to important information about the product or service — but this isn’t the case with health care.

What High-Need, High-Cost Patients Say About How to Reduce High Utilization of ED and Inpatient Services

High-Need, High-Cost Patients Offer Solutions for Improving Their Care and Reducing Costs

More home health care and after-hours clinics, telemedicine, and home delivery of medications are among top solutions.

Top Physician Pain Points Identified by Chronic Patients

Unmet Needs: Hearing the Challenges of Chronic Patients with Artificial Intelligence

With natural language processing and machine learning, researchers are identifying patient emotional and medical needs that are not being met by clinicians and patient advocacy groups.

Health Care Can Learn a Lot from Other Consumer-Facing Industries - Especially About Customer Service - Consumerization of Health Care

Patient Engagement Survey: Health Care Has a Lot to Learn from Consumer-Friendly Industries

The consumerization of health care continues to reshape the way that patients engage with providers and experience care. Most providers see this fundamental change in the health care model as a necessary response to changing patient demands, and have embraced the need to learn from other industries.

Bilazarian01_pullquote patient engagement both sides of the bed

Patient Engagement from Both Sides of the Bed

When patients and families are included in medical rounds as valued members of the team, the quality of care improves.

Ten Guiding Principles for Patient-Centered Care as Described by Mountain States Health Alliance to consider with patient-centered outcomes research (PCOR)

Complementary Approaches to Patient Engagement in Patient-Centered Outcomes Research

It’s important for providers to understand and apply the key principles foundational to patient-centered outcomes research efforts.

Selected Advantages of Improved Patient Electronic Access to Health Data

Patient-Led Data Sharing — A New Paradigm for Electronic Health Data

The pieces are in place for a truly disruptive shift in how patients can access and use their own clinical data to improve their health.

Collapse of Two Online Health Communities as Result of Removal of Users Starting from Superusers

Online “Superusers” as Allies of the Health Care Workforce

Three proposed steps for integrating peer-driven online health communities with traditional services to improve health outcomes for people with chronic conditions.


A weekly email newsletter featuring the latest actionable ideas and practical innovations from NEJM Catalyst.

Learn More »


Patient Incentives

70 Articles

Survey Snapshot: Is Transparency the Answer…

NEJM Catalyst Insights Council members say that while transparency might be necessary, we have to…

Primary Care

184 Articles

New Evidence on Stemming Low-Value Prescribing

New research suggests that deploying strong and surprising messages could help to address overprescribing of…

Digital Health Care in China: Benefits…

China’s experience offers insights for integrating digital and non-digital health care services.

Insights Council

Have a voice. Join other health care leaders effecting change, shaping tomorrow.

Apply Now