Care Redesign

Measuring Performance of OpenNotes Initiatives to Target Improvement Efforts

Case Study · October 22, 2019

When launching and implementing OpenNotes (sharing visit notes with patients through an electronic patient portal), health systems face the challenges of devising ways to increase patient use and measure impact. At the country’s largest safety-net health system, New York City Health + Hospitals, we developed data cascade visualizations and metrics to guide efforts to improve both note sharing and viewing. Data cascades allowed us to identify root causes of underutilization, address technical barriers to default note sharing, design targeted behavioral interventions to engage patients in viewing notes, and begin measuring effectiveness.

Key Takeaways

  1. Health systems can improve implementation of OpenNotes by using data cascade visualizations and metrics to uncover the different factors affecting patient engagement.

  2. Evidence-based behavioral techniques, such as the EAST framework, offer a useful approach to creating targeted interventions to improve performance in a safety-net setting like ours.

  3. Improving patient engagement will require strong interdisciplinary teams with a range of skills such as population health, user experience design, marketing, and data science.


OpenNotes is a movement to make clinical notes available to patients and make health care more transparent. Begun in 2010 as a demonstration project at three U.S. health care centers, OpenNotes now enables more than 40 million patients worldwide to read their outpatient progress notes online. The benefits of making notes “open” are well established, including increased trust in providers, patient satisfaction, adherence, and shared decision making. As OpenNotes is adopted by more systems, the most pressing questions will shift from the feasibility of launching, which is often a “non-event,” to how health systems can optimize and continually improve note sharing so that more patients experience its benefits. Other questions revolve around how best to measure the tool’s impact on patient engagement.

OpenNotes implementation is a multi-step process that requires numerous stakeholders to be successful — senior administrative and clinical leaders, internal technical specialists, external patient portal vendors, individual providers, and of course, patients. Health systems with OpenNotes typically share clinical notes with patients by default via the patient portal while still giving providers the option of not sharing individual notes in cases where an individual note may pose a risk or cause undue harm.

Having adequate data is critical to spreading and scaling interventions in a safety-net system like ours. New York City Health + Hospitals is the largest public health care system in the U.S., serving about 1.2 million patients from many racial and ethnic groups across 70 locations. Research indicates that OpenNotes are especially valuable in helping people of color and those with less education feel informed about their medical care. We want to empower all our patients and their families, particularly the most vulnerable, to be active participants in maintaining their health.

Our Office of Population Health initially launched OpenNotes among NYC Health + Hospitals’ first wave of facilities transitioning to the Epic electronic medical record. These sites included three hospitals and their affiliated ambulatory care sites. We received grant funding from New York State Health Foundation to implement this first wave of sites over 20 months, from February 2017 to October 2018. During this time, we enabled default note sharing via Epic’s patient portal, MyChart, for all outpatient specialties except behavioral health.

The Challenge

To ensure that we were effectively implementing OpenNotes in a way that supported our diverse population, we needed a rigorous system for measuring performance. However, we found that metrics used in the literature and by most health systems to evaluate OpenNotes performance were inadequate for informing targeted actions.

The Goal

To use data science to target strategies for increasing the number and proportion of patients using OpenNotes to engage with their health care. By applying core metrics and data cascades to dissect OpenNotes implementation, we aimed to identify opportunities for technical and behavioral interventions to influence provider and patient behavior and to measure the effectiveness of interventions.

The Execution

Generating Data Cascades

When deploying OpenNotes at NYC Health+ Hospitals, we found that understanding our performance required different and more granular data than normally captured. Health systems typically reference two data fields to calculate clinical note view rate: whether a note was shared, and whether a patient viewed it. While note view rate is important, it was insufficient in helping us understand where and why problems were occurring. For example, when we first examined this data, we could not discern whether our low view rate was due to low provider share rates, patient enrollment issues with MyChart, or navigation barriers to finding visit notes within MyChart.

To answer our questions, we worked with our IT colleagues to produce a monthly report using existing data fields in MyChart. Our data fell into three categories: patient/provider identifiers, actions taken in Epic, and demographic data. Having more robust information allowed us to create data cascades, an analytic tool used in contexts as diverse as HIV treatment programs, consumer marketing, and software design. Data cascades display the cumulative effect of consecutively introduced steps in an intervention. By breaking down a process into its components, we could readily identify potential root causes by studying various drop-off points in clinical note activity. Based on our review of the literature, other organizations implementing OpenNotes did not measure their performance to the level of specificity seen in our data cascades.

Data cascades became a core part of how we communicated OpenNotes performance. We developed a digital dashboard (Figure 1) that featured our data cascades and key process and outcome metrics. We circulated the dashboard regularly during implementation to engage numerous stakeholders, such as the clinical providers who signed and shared notes, our IT team who supported patient portal configuration and design, and clinic staff who granted patient portal access during outpatient visits. The data cascades’ visual design helped make the dashboard easily digestible. The dashboard formalized OpenNotes as an important system-wide initiative and provided consistent language by which to communicate performance.

NYC Health + Hospitals OpenNotes Performance Dashboard February 10, 2017 - October 19, 2018

Figure 1 Click To Enlarge.


Designing Targeted Interventions

Identifying key drop-off points created opportunities to collaborate with relevant stakeholders to improve note view rates. On the technical side, data cascades helped us make specific requests for IT fixes. For example, the notes cascade (Figure 1, Cascade Chart [Notes Perspective]) illustrates several drop-offs between the cumulative total number of signed notes (785,577) and total notes viewed by patients (4,295). We observed a steep drop between signed and shared notes and quickly pinpointed at least two technical causes. First, default note sharing was mistakenly not enabled for an entire group of providers: resident physicians. As a result, over 20% of notes written were not accessible to our patients. Second, we found that the average share rate by department was 75%, with the majority of departments sharing at or above 90%. However, a few were sharing at 0%, a nearly impossible rate when default note sharing is enabled. In both cases, we worked with our IT colleagues to ensure it was enabled correctly across providers and specialties, except behavioral health.

Even after removing those technical barriers, we saw that many patients still were not viewing their clinical notes (Figure 1, Cascade Chart [Patient Perspective]). We knew influencing patient behavior would not be a simple technical fix, so we turned to behavioral economics principles. We partnered with the global Behavioural Insights Team to design interventions and enhance user experience using their EAST framework, which involves applying Easy, Attractive, Social, and Timely (EAST) measures to influence behavior change. “Easy” interventions try to reduce the effort needed to achieve a specific outcome. Not surprisingly, even a small effort can have a disproportionately large effect on outcomes. For example, to increase patient portal enrollment, we enabled self-signup capability—eliminating the need for patients to wait for a staff member to enroll. We distilled the MyChart enrollment process into a simple checklist distributed to patients at the end of their appointment. We also made notes easier to find by adding a new button for viewing notes on the MyChart homepage and creating instructional banners. Figure 2 shows these and other examples of our behavioral approaches.

NYC Health + Hospitals EAST Interventions to Increase Engagement with OpenNotes

Figure 2 Click To Enlarge.

Measuring Impact of Targeted Interventions

We also created specific data cascades and related metrics to evaluate the impact of our efforts. For instance, we developed a new data cascade to gauge patients’ user experience after they received an email prompt about their notes. We saw that even among patients who were motivated enough to log into the portal, many still did not view their notes. We had already exhausted available MyChart customization options to improve note viewing, so this suggested an underlying design flaw with MyChart. Using the insights from our data cascades, we successfully lobbied Epic to change its native MyChart design for all clients across the country. In the future, users will see better alerts and additional access points from other sections of MyChart’s web-based patient portal and mobile app.

The Behavioural Insights Team also helped us launch randomized controlled trials (RCT) to test the effectiveness of specific interventions. In the world of consumer marketing and product management, such trials (often called A-B tests) involve comparing two versions of marketing content for relative impact. In one trial, we wanted to understand whether the language of an email encouraging patients to access their notes affected view rate. We emailed one cohort in English and another in either English or Spanish, depending on their preferred language. Among Spanish speakers, translated outreach emails were associated with a 22.9% increase in odds of clicking the email link to view notes, and a 33.7% increase in odds of actually viewing notes after clicking open the email (Figure 3). The RCT framework helped us validate our intuitions and quantify the impact of an intervention at each step of note viewing.

OpenNotes Epic Patient Email Cascade Chart February 1, 2018 - August 14, 2018

Figure 3 Click To Enlarge.

We are also exploring interventions to understand how providers could influence patient view rates. As an example, we designed another RCT to evaluate the impact of emailing providers about patient view rates. In treatment group A, clinicians received an email showing the note view rate for the NYC Health + Hospitals system and for high-performing hospitals as a benchmark. Treatment group B received a similar email, but instead of the system’s overall view rate, it compared the individual provider’s view rate for their patient panel with the high-performing hospitals benchmark. Group B members also received a list of their own patients who viewed their note. A control group of providers received no emails about patient view rate. We could not detect any differences in patient view rates across the two treatment groups or control group. Despite the results, the RCT allowed us to quickly test an idea before committing to a full-scale implementation.

Metrics and Results

At the end of the 20-month grant period (February 2017 to October 2018), 129,579 notes were shared with 23,282 patients by 1,758 providers (including residents and all outpatient specialties except behavioral health). As illustrated in Figure 1 (Key Performance Metrics), 74.3% of all notes available were shared with patients via MyChart; 4,295 notes (3.3%) were viewed; and 2,079 patients (8.9%) read at least one note. The number of providers sharing notes grew from 157 to 1,758 (an 11-fold increase), and there was a 10-fold increase in the number of patients using OpenNotes.

We have since incorporated OpenNotes into the standard EHR implementation so that every site in New York City Health + Hospitals that transitions to Epic will automatically have it configured. From February 2017 to August 2019, 4,207 providers shared over 1.4 million notes, and nearly 4,420 patients have read at least one note. We are well poised to extend OpenNotes to all 11 hospitals and over 70 community-based clinics by the end of 2019.

To augment our quantitative analysis, in spring 2018 we conducted an online survey and interviews with patients to solicit their impressions of OpenNotes. This provided invaluable insights into patient engagement. Participants said limited language capabilities within the EHR prevent them from maximizing the benefits of OpenNotes. We found that most patients took some sort of action after reading their clinical note; a majority looked up more information online or felt they could better follow their treatment plan. One patient shared, “I welcome that I can now read the Doctor’s notes on all my visits so I can better understand what’s going on with my health and I can make the necessary changes for a healthier lifestyle.” This echoes our patients’ overall appreciation for access to notes, even among non-English speakers.

A Selection of OpenNotes Patient Testimonials

General Feedback

“I love this Mychart feature … especially if I was not hearing everything he said, I can come on there and go through it again.”

“The notes online makes it so easy to find out what you need to know — especially when you’re waiting for results!”

“I welcome that I can now read the Doctor’s notes on all my visits so I can better understand what’s going on with my health and I can make the necessary changes for a healthier lifestyle.”

“I would encourage everyone to sign up for MyChart and to keep informed by reading and following up on their doctor’s notes.”

“I love MyChart. I have a MyChart account for myself and my daughter. My husband also signed up because he also thought it was very useful…. At the beginning, one is afraid of the unknown whether it be due [to] being a first time mom or because you are new to this country and don’t know how it works. The consistency in what my doctor told me during the visit and what they would write down in their notes is what generated my trust in my doctors and in the care that I was receiving. I use Google Translate to translate the notes, or if I have a question I would ask my husband to translate for me. I share my notes and the pediatrician’s notes with my husband, as he likes to be informed.”

A Selection of OpenNotes Patient Testimonials

Language and Accessibility

“I would like there to be a translator integrated into the clipboard so as to translate the notes directly without making a copy and paste [into] usado [online translation tool].”

“I would like for the notes to be more extensive, as sometimes they only write simple things or use terminology that one does not understand. Sometimes, even [online translation tools] would not be able to translate because some of the terminology is very technical. Sometimes they write with abbreviations and it leaves you kind of lost.”

“It is important that there be doctor’s notes for ALL visits.”

Despite the fact that clinical notes are available only in English, we incorporated preferred language in other aspects of the OpenNotes experience. For Spanish speakers, we translated elements of MyChart, such as the link to OpenNotes, for easier navigation. We also tested a Spanish language email to remind patients that a note is available to read. Our patients frequently turn to friends and family to help translate, so we give relatives and care partners proxy MyChart access when requested by the patient. Care partners can play an important role in helping patients access information and in fostering productive discussions between patients and care teams.


A fundamental question that health systems implementing OpenNotes and other digital interventions must ask and answer is, “What does success mean?” Answering this question should guide decisions around goals and metrics to use, including benchmarking with different systems. Merely launching OpenNotes as a feature, while a vital step, is not sufficient if patients are not accessing and making meaningful use of their notes.

Data cascades and their related metrics allowed us to quantify our performance at each step of implementation and engage appropriate stakeholders to address any challenges. Our performance dashboard clearly communicated our progress across clinical teams and encouraged an iterative approach to problem solving. In addition, data cascades helped us design targeted interventions to achieve specific outcomes.

As we consider what success means in OpenNotes implementation, we must also use both quantitative and qualitative data to understand and improve patient engagement. Just because patients view their post-encounter note does not always mean they understand it or derive any benefit.

Finally, as health systems consider how to implement patient-engagement initiatives, it is critical to assemble interdisciplinary teams with the right mix of skill sets, including some not included in traditional hospital administration and population health management. In our case, data science skills were important for mining data, creating compelling graphs, developing our randomized control trials methodology, and evaluating their impact. We also partnered with marketing and communications teams for patient outreach. Our training teams helped engage and educate our providers. Our technical teams and behavioral science partner refined the patient and clinician user experience. All these different skill sets were united through our Office of Population Health, which often works across disciplines and care settings in crafting new care models and enabling patient engagement.

Next Steps

Our efforts constituted an initial step toward measuring and optimizing patient engagement with OpenNotes. We will continue to ask our patients about their direct experience to better understand what they value. We look to further collaborate with patients and EHR vendors to improve patient engagement.

In the future, we aim to create a more comprehensive dashboard that captures additional information about MyChart use and other digital features available to patients and providers. The need for such measurement will grow as digital applications become more integrated in the care process, and as health organizations compete for patients who demand consumer-friendly and transparent care.


The authors are grateful to the New York State Health Foundation, who funded our OpenNotes pilot; the OpenNotes organization; The Behavioural Insights Team (BIT); and Get My Health Data, who funded our patient-facing survey.

Call for submissions:

Now inviting expert articles, longform articles, and case studies for peer review


A weekly email newsletter featuring the latest actionable ideas and practical innovations from NEJM Catalyst.

Learn More »

More From Care Redesign
Community Resource Referral Type

Assessing and Addressing Social Needs in Primary Care

Lincoln Community Health Center improved care quality by measuring and responding to upstream social and economic risk factors disproportionately affecting low-income households.

Time Driven Activity Based Costing for ECMO

Achieving Value in Highly Complex Acute Care: Lessons from the Delivery of Extra Corporeal Life Support

To improve both the value and outcomes of ECLS, Cedars-Sinai Medical Center created guidelines for ECLS delivery and explored opportunities for more efficient care.

ARISE and SFHN BHVS Collaboration

The “Behavioral Health Vital Signs” Initiative

A safety net system’s trauma-informed approach to integrating interpersonal violence into behavioral health programs in primary care.

Mapping a Technology Strategy for Bundled Payment Care Using a Value-Driven Framework

Harnessing Emerging Information Technology for Bundled Payment Care Using a Value-Driven Framework

A four-part framework developed by physicians at Partners HealthCare provides a stepwise process for assessing and integrating technologies to effectively use data through a continuous patient experience.

UCLA Health CKD Risk Stratification and Management

Proactively Catching the Declining Patient

A coordinated effort by UCLA leaders to identify a high-cost population with chronic kidney disease and to modify care processes and personnel has led to improved health and reduced utilization.

Telehealth and remote monitoring are little used and ineffective for chronic disease care

Survey Snapshot: Treating Chronic Disease Proactively

Though survey respondents don’t indicate strong use of telehealth and remote monitoring, NEJM Catalyst Insights Council members discuss the ways they’re using these tools to monitor chronic disease, with good results.

Platforming Health Care Operations - Consumer-Driven Health Care - Business-Minded Optimizations

Platforming Health Care to Transform Care Delivery

Health care leaders need to focus less on ownership and control of the delivery process, and more on outcomes, cost efficiency, and customer experience.

Shah05_ integrated systems innovation pullquote

Build vs. Buy: What Should Health Systems Do?

The consolidation craze continues, but vertical integration has yet to demonstrate real progress toward the Triple Aim. Health care leaders would do well to consider innovative approaches that are working in other industries, including the tech-enabled full stack model.

Diagram Illustrating the COPD Care Pathway at Allegheny General Hospital

End-to-End Care for COPD Patients that Improves Outcomes and Lowers Costs

Allegheny General Hospital created a comprehensive solution for patients with chronic obstructive pulmonary disease (COPD) that led to improved clinical outcomes, reduced hospital admissions and readmissions, and a resultant decrease in the total cost of care.

David Blumenthal and Bob Galvin head shots

Ripe for Disruption: Why and How Big Players in the Private Sector Are Taking on Health Care

For big tech companies like Amazon, Apple, and Google, the health care sector looks ripe for disruption. Two executives working in different parts of the health care ecosystem discuss what this means for patients and doctors, including the positives and unintended consequences.


A weekly email newsletter featuring the latest actionable ideas and practical innovations from NEJM Catalyst.

Learn More »


Coordinated Care

145 Articles

Proactively Catching the Declining Patient

A coordinated effort by UCLA leaders to identify a high-cost population with chronic kidney disease…

Care Integration

77 Articles

Platforming Health Care to Transform Care…

Health care leaders need to focus less on ownership and control of the delivery process,…

Assessing and Addressing Social Needs in…

Lincoln Community Health Center improved care quality by measuring and responding to upstream social and…

Insights Council

Have a voice. Join other health care leaders effecting change, shaping tomorrow.

Apply Now