Care Redesign

Ensuring Clinical Quality in Telemedicine

Case Study · August 2, 2018

In September 2014, Cleveland Clinic partnered with the Online Care Group, a nationwide system of physicians who provide telemedicine services through the American Well online platform, to deliver care via video visits under the Cleveland Clinic name.

Cleveland Clinic’s original aims in introducing telemedicine were to increase patient access to care, support the transition to value-based care, improve system efficiency, and expand the system’s reach to new patients. To ensure that the Online Care Group physicians practiced at or above the clinical standards set by the Cleveland Clinic, the two organizations embarked on a multiyear collaboration to improve the quality of care delivered through the online platform.

Key Takeaways

  1. Health systems and third-party providers of telemedicine clinical services can successfully redesign their processes to improve the quality of care provided by a nationwide system of physicians.

  2. Establishing regular and frequent communication between the leadership of health systems and that of third-party telemedicine providers allows for quick identification of problems and development of solutions.

  3. Both parties must be willing and able to work on quality initiatives and must be equally engaged in identifying and improving clinical metrics.

  4. Technology assets should be used to their fullest potential. In our case, following the creation of standardized templates that prompted physicians across the country to adhere to consistent clinical standards, the percentage of notes that met full documentation standards increased from 45% to 85%.

  5. When communicating with a group of distributed physicians working virtually, it is important to communicate through multiple channels (e.g., email, e-newsletters, webinars, online discussion boards, phone calls, and individual calls) to ensure that all physicians are aware of quality improvement measures and guideline changes. In our case, following the implementation of several measures designed to improve communication, the rate of inappropriate pediatric telemedicine visits decreased from 67% to 1% within 6 months.

The Challenge

Professional organizations such as the American Medical Association and the American Academy of Pediatrics have raised quality concerns around telemedicine, citing the lack of care standards and nationally recognized clinical guidelines. As the urgent care clinical services provided by the Online Care Group were delivered under the Cleveland Clinic name, our challenge was to ensure that Online Care Group physicians met the same high-quality clinical standards that Cleveland Clinic upholds across the organization.

The Goal

Our goal was to create an effective quality improvement collaboration between the Cleveland Clinic and the Online Care Group with a focus on ensuring visit-type appropriateness, documentation completion, and appropriate antibiotic prescriptions.

The Execution

The Platform and Partnership

In September 2014, the Cleveland Clinic began virtual visits and, in June 2015, it launched Express Care Online, an on-demand urgent care telemedicine service (hosted on the American Well platform) that provides 24/7/365 care for patients >2 years of age. Recognizing that it did not have the internal infrastructure or capacity to independently support this new service, Cleveland Clinic partnered with physicians of the Online Care Group.

Off-hours telephone coverage and triage provided by an external group is not uncommon in health care; however, establishing an on-demand urgent care service that is staffed by physicians and that enables providers to diagnose and treat patients requires more stringent clinical quality protocols. Clinical quality equal to or exceeding the metrics established for brick-and-mortar facilities was paramount to this program. Although the Online Care Group monitored quality with use of well-established metrics that had been developed and tested since the group’s inception, these metrics required alignment with existing Cleveland Clinic standards to meet quality goals.

Establishing Quality Review Meetings

Early on, both organizations determined that frequent and open communication was necessary to reach our quality goals. To that end, we established monthly quality review meetings that allowed for the bidirectional sharing of information regarding quality metrics and outcomes. Attendance was mandatory for the Medical and Quality Directors of both organizations; however, the entire clinical team was also invited to attend and participate in the conversation. We designed the initial quality improvement plan around four interrelated categories: Organizational Alignment, Clinical Guidelines, Collaborative Partnerships, and Quality Assurance and Improvement.

Telemedicine Online Care Group Cleveland Clinic Express Care Online Quality Improvement - Next Steps

Chart summarizing the initial quality improvement plan, showing progressive goals and feedback loops. Click To Enlarge.

Determining the Appropriate Scope of Telemedicine

As telemedicine was a new type of care delivery for Cleveland Clinic, the first goal was to establish guidelines to help determine which visits were appropriate for telemedicine without an in-person referral. At each meeting, the Quality Directors of both organizations reviewed the audit of approximately 30% of the total number of adult patient visits and 100% of pediatric visits from the preceding month for clinical appropriateness.

The operational definition of an “inappropriate visit” was “a completed visit leading to a treatment plan in the absence of one or more physical examination components considered to be necessary to arrive at the diagnosis or clinical decision.” Examples included the diagnosis and treatment of acute otitis media in a patient in whom the tympanic membrane had not been visualized and the diagnosis and treatment of a urinary tract infection in a pediatric patient without a urine analysis and culture.

Initial audits by the Cleveland Clinic’s Express Care Online Quality Director revealed that up to 67% of the previous month’s pediatric visits had been inappropriate. As the Medical and Quality Directors from both organizations were committed to driving down the number of inappropriate visits and facilitating high-quality telehealth care, both organizations quickly committed to a quality improvement plan designed to eliminate inappropriate visits.

Transparency and Communication          

Over the next 6 months, both organizations implemented an intensive program to train physicians on appropriate and inappropriate visit types. Both organizations were transparent with their provider teams regarding the results of the chart audit and the goal of the initiative. The Online Care Group used email campaigns and online training to increase provider awareness of the initiative, including updated visit guidelines and remediation plans if the guidelines were not followed.

Overcoming Barriers to Adoption of New Guidelines

Initially, the intensive campaign showed little reduction in inappropriate visit types. Both organizations quickly worked together and identified three system barriers impeding the timely and consistent distribution of and adherence to guidelines. First, the Online Care Group had low attendance at live webinars because its physicians practiced in different time zones and had busy in-person clinical schedules. Second, the physicians had no online forum where they could gather and discuss policy changes. Third, the Online Care Group works with other health systems in addition to Cleveland Clinic and had not yet developed an easy method for communicating different health system standards for visit appropriateness to their physicians.

To overcome these barriers, the Online Care Group developed enhancements to its electronic medical record, including the use of templates to prompt the physician on the standards for an appropriate visit type. The Online Care Group also developed standing virtual team meetings; a weekly email-based newsletter; a private, online physician discussion board; and guidelines on when to trigger one-on-one remediation conversations. Within 6 months, the baseline measure of inappropriate visits declined from 67% to 1%. The significant and rapid improvement in adherence to visit appropriateness guidelines was encouraging to both groups and provided a solid foundation for further quality improvement initiatives.

Appropriate Documentation

Based on the successes of the initial quality improvement initiative, the two organizations decided to tackle appropriate documentation as a second quality measure in mid-2016. All parties agreed that minimum documentation elements required for an online telehealth visit should mirror in-person requirements, including patient identifiers, allergies, medical and surgical history, medications, the history of the current problem, physical examination findings, and a decision-making and treatment plan (including patient instructions regarding when and where to follow-up).

The Cleveland Clinic team was accustomed to using templates and had developed several such tools for the care that they were already providing, whereas the telehealth partner physicians were typically free-texting clinical documentation without a template. Both organizations agreed that templates would likely improve documentation, and, over the next 2 months, templates were developed and shared internally within the Online Care Group. The use of these templates increased steadily, driving the rate of full documentation from 45% initially to 85% after 6 months; currently, the rate of full documentation hovers around 95% to 97%.

Antibiotic Stewardship

The third quality measure, initiated in January 2018, focused on appropriate antibiotic prescriptions. In the spirit of embracing antibiotic stewardship, we identified a coordinated set of strategies that were designed both to treat conditions appropriately and to minimize the potential for antimicrobial resistance.

Both organizations shared and aligned their clinical practice guidelines regarding antibiotic appropriateness and first-line medications. If a physician deviated from the first-line antimicrobial, he or she needed to provide a clear reason for the clinical decision, such as a documented patient allergy to the first-line medication. Baseline data showed that we initially achieved antimicrobial stewardship 69% of the time. By 6 months after our intervention, our antibiotic stewardship had improved to 97%.

The Team

The leadership of this quality improvement initiative consisted of the Medical and Quality Directors of both the Cleveland Clinic and the Online Care Group as well as the President of the Online Care Group.

Metrics Summary

The quality improvement metrics described above can be summarized as follows:

  • The rate of inappropriate pediatric visits decreased from 67% to 1% within 14 months.
  • The rate of full documentation increased from 45% to 85% within 6 months (and currently hovers around 95% to 97%).
  • The rate of appropriate antibiotic prescribing increased from 69% to 97% within 6 months.

Lessons Learned

  • The partnership required both organizations to be fully committed to the goal of becoming the highest-quality medical providers in the on-demand urgent care telehealth space. This commitment helped both organizations to persevere in the collaboration despite initial setbacks in the effort to ensure quality in this evolving field.
  • Both organizations were required to enhance their own digital infrastructures to monitor quality in real time and to provide timely feedback to medical providers.
  • To practice effectively with the constantly evolving technology that supports telemedicine, medical providers must be able to frequently and easily communicate with the quality and information technology teams.
  • Respectful open dialogue, coupled with a clearly articulated vision of providing high-quality medical care in the on-demand urgent care space, has allowed these two organizations to rapidly iterate quality improvement strategies and thrive as telehealth standards and metrics continue to be developed at a national level.

Call for submissions:

Now inviting expert articles, longform articles, and case studies for peer review


A weekly email newsletter featuring the latest actionable ideas and practical innovations from NEJM Catalyst.

Learn More »

More From Care Redesign
Mapping a Technology Strategy for Bundled Payment Care Using a Value-Driven Framework

Harnessing Emerging Information Technology for Bundled Payment Care Using a Value-Driven Framework

A four-part framework developed by physicians at Partners HealthCare provides a stepwise process for assessing and integrating technologies to effectively use data through a continuous patient experience.

UCLA Health CKD Risk Stratification and Management

Proactively Catching the Declining Patient

A coordinated effort by UCLA leaders to identify a high-cost population with chronic kidney disease and to modify care processes and personnel has led to improved health and reduced utilization.

Telehealth and remote monitoring are little used and ineffective for chronic disease care

Survey Snapshot: Treating Chronic Disease Proactively

Though survey respondents don’t indicate strong use of telehealth and remote monitoring, NEJM Catalyst Insights Council members discuss the ways they’re using these tools to monitor chronic disease, with good results.

Platforming Health Care Operations - Consumer-Driven Health Care - Business-Minded Optimizations

Platforming Health Care to Transform Care Delivery

Health care leaders need to focus less on ownership and control of the delivery process, and more on outcomes, cost efficiency, and customer experience.

Shah05_ integrated systems innovation pullquote

Build vs. Buy: What Should Health Systems Do?

The consolidation craze continues, but vertical integration has yet to demonstrate real progress toward the Triple Aim. Health care leaders would do well to consider innovative approaches that are working in other industries, including the tech-enabled full stack model.

Diagram Illustrating the COPD Care Pathway at Allegheny General Hospital

End-to-End Care for COPD Patients that Improves Outcomes and Lowers Costs

Allegheny General Hospital created a comprehensive solution for patients with chronic obstructive pulmonary disease (COPD) that led to improved clinical outcomes, reduced hospital admissions and readmissions, and a resultant decrease in the total cost of care.

David Blumenthal and Bob Galvin head shots

Ripe for Disruption: Why and How Big Players in the Private Sector Are Taking on Health Care

For big tech companies like Amazon, Apple, and Google, the health care sector looks ripe for disruption. Two executives working in different parts of the health care ecosystem discuss what this means for patients and doctors, including the positives and unintended consequences.

Top challenges facing chronic disease management care - insufficient time and care coordination

Care Redesign Survey: To Improve Chronic Disease Care, Change the Payment Model

Many health care organizations are reasonably effective in treating chronic diseases, but they are limited from doing better by fee-for-service payment, which remains the predominant payment model in the United States. This report serves as a snapshot in time, showing the intent of health care providers to be proactive in treating chronic disease, but limitations in their ability to address population health.

Nirav Shah head shot Stanford - AI in medicine and team care

What AI Means for Doctors and Doctoring

Physicians must hone the “four Cs” — critical thinking, communication, collaboration, and creativity — when leveraging AI as a new partner in their care teams.

End of Life EOL Palliative Care in the ED for Patients with Advanced Cancer - Process Map - MD Anderson Cancer Center

Patient-Centered Care at the End of Life in the ED

How MD Anderson Cancer Center is improving end-of-life care in an unlikely place: the emergency department.


A weekly email newsletter featuring the latest actionable ideas and practical innovations from NEJM Catalyst.

Learn More »


Design Thinking

20 Articles

Taxonomy of the Patient Voice

While health care pursues the important trend of putting patients at the center of care,…

How Artificial Intelligence Is Changing Health…

The development of intelligent machines holds great promise for making health care delivery more accurate,…

Proactively Catching the Declining Patient

A coordinated effort by UCLA leaders to identify a high-cost population with chronic kidney disease…

Insights Council

Have a voice. Join other health care leaders effecting change, shaping tomorrow.

Apply Now