Care Redesign

Ensuring Clinical Quality in Telemedicine

Case Study · August 2, 2018

In September 2014, Cleveland Clinic partnered with the Online Care Group, a nationwide system of physicians who provide telemedicine services through the American Well online platform, to deliver care via video visits under the Cleveland Clinic name.

Cleveland Clinic’s original aims in introducing telemedicine were to increase patient access to care, support the transition to value-based care, improve system efficiency, and expand the system’s reach to new patients. To ensure that the Online Care Group physicians practiced at or above the clinical standards set by the Cleveland Clinic, the two organizations embarked on a multiyear collaboration to improve the quality of care delivered through the online platform.

Key Takeaways

  1. Health systems and third-party providers of telemedicine clinical services can successfully redesign their processes to improve the quality of care provided by a nationwide system of physicians.

  2. Establishing regular and frequent communication between the leadership of health systems and that of third-party telemedicine providers allows for quick identification of problems and development of solutions.

  3. Both parties must be willing and able to work on quality initiatives and must be equally engaged in identifying and improving clinical metrics.

  4. Technology assets should be used to their fullest potential. In our case, following the creation of standardized templates that prompted physicians across the country to adhere to consistent clinical standards, the percentage of notes that met full documentation standards increased from 45% to 85%.

  5. When communicating with a group of distributed physicians working virtually, it is important to communicate through multiple channels (e.g., email, e-newsletters, webinars, online discussion boards, phone calls, and individual calls) to ensure that all physicians are aware of quality improvement measures and guideline changes. In our case, following the implementation of several measures designed to improve communication, the rate of inappropriate pediatric telemedicine visits decreased from 67% to 1% within 6 months.

The Challenge

Professional organizations such as the American Medical Association and the American Academy of Pediatrics have raised quality concerns around telemedicine, citing the lack of care standards and nationally recognized clinical guidelines. As the urgent care clinical services provided by the Online Care Group were delivered under the Cleveland Clinic name, our challenge was to ensure that Online Care Group physicians met the same high-quality clinical standards that Cleveland Clinic upholds across the organization.

The Goal

Our goal was to create an effective quality improvement collaboration between the Cleveland Clinic and the Online Care Group with a focus on ensuring visit-type appropriateness, documentation completion, and appropriate antibiotic prescriptions.

The Execution

The Platform and Partnership

In September 2014, the Cleveland Clinic began virtual visits and, in June 2015, it launched Express Care Online, an on-demand urgent care telemedicine service (hosted on the American Well platform) that provides 24/7/365 care for patients >2 years of age. Recognizing that it did not have the internal infrastructure or capacity to independently support this new service, Cleveland Clinic partnered with physicians of the Online Care Group.

Off-hours telephone coverage and triage provided by an external group is not uncommon in health care; however, establishing an on-demand urgent care service that is staffed by physicians and that enables providers to diagnose and treat patients requires more stringent clinical quality protocols. Clinical quality equal to or exceeding the metrics established for brick-and-mortar facilities was paramount to this program. Although the Online Care Group monitored quality with use of well-established metrics that had been developed and tested since the group’s inception, these metrics required alignment with existing Cleveland Clinic standards to meet quality goals.

Establishing Quality Review Meetings

Early on, both organizations determined that frequent and open communication was necessary to reach our quality goals. To that end, we established monthly quality review meetings that allowed for the bidirectional sharing of information regarding quality metrics and outcomes. Attendance was mandatory for the Medical and Quality Directors of both organizations; however, the entire clinical team was also invited to attend and participate in the conversation. We designed the initial quality improvement plan around four interrelated categories: Organizational Alignment, Clinical Guidelines, Collaborative Partnerships, and Quality Assurance and Improvement.

Telemedicine Online Care Group Cleveland Clinic Express Care Online Quality Improvement - Next Steps

Chart summarizing the initial quality improvement plan, showing progressive goals and feedback loops. Click To Enlarge.

Determining the Appropriate Scope of Telemedicine

As telemedicine was a new type of care delivery for Cleveland Clinic, the first goal was to establish guidelines to help determine which visits were appropriate for telemedicine without an in-person referral. At each meeting, the Quality Directors of both organizations reviewed the audit of approximately 30% of the total number of adult patient visits and 100% of pediatric visits from the preceding month for clinical appropriateness.

The operational definition of an “inappropriate visit” was “a completed visit leading to a treatment plan in the absence of one or more physical examination components considered to be necessary to arrive at the diagnosis or clinical decision.” Examples included the diagnosis and treatment of acute otitis media in a patient in whom the tympanic membrane had not been visualized and the diagnosis and treatment of a urinary tract infection in a pediatric patient without a urine analysis and culture.

Initial audits by the Cleveland Clinic’s Express Care Online Quality Director revealed that up to 67% of the previous month’s pediatric visits had been inappropriate. As the Medical and Quality Directors from both organizations were committed to driving down the number of inappropriate visits and facilitating high-quality telehealth care, both organizations quickly committed to a quality improvement plan designed to eliminate inappropriate visits.

Transparency and Communication          

Over the next 6 months, both organizations implemented an intensive program to train physicians on appropriate and inappropriate visit types. Both organizations were transparent with their provider teams regarding the results of the chart audit and the goal of the initiative. The Online Care Group used email campaigns and online training to increase provider awareness of the initiative, including updated visit guidelines and remediation plans if the guidelines were not followed.

Overcoming Barriers to Adoption of New Guidelines

Initially, the intensive campaign showed little reduction in inappropriate visit types. Both organizations quickly worked together and identified three system barriers impeding the timely and consistent distribution of and adherence to guidelines. First, the Online Care Group had low attendance at live webinars because its physicians practiced in different time zones and had busy in-person clinical schedules. Second, the physicians had no online forum where they could gather and discuss policy changes. Third, the Online Care Group works with other health systems in addition to Cleveland Clinic and had not yet developed an easy method for communicating different health system standards for visit appropriateness to their physicians.

To overcome these barriers, the Online Care Group developed enhancements to its electronic medical record, including the use of templates to prompt the physician on the standards for an appropriate visit type. The Online Care Group also developed standing virtual team meetings; a weekly email-based newsletter; a private, online physician discussion board; and guidelines on when to trigger one-on-one remediation conversations. Within 6 months, the baseline measure of inappropriate visits declined from 67% to 1%. The significant and rapid improvement in adherence to visit appropriateness guidelines was encouraging to both groups and provided a solid foundation for further quality improvement initiatives.

Appropriate Documentation

Based on the successes of the initial quality improvement initiative, the two organizations decided to tackle appropriate documentation as a second quality measure in mid-2016. All parties agreed that minimum documentation elements required for an online telehealth visit should mirror in-person requirements, including patient identifiers, allergies, medical and surgical history, medications, the history of the current problem, physical examination findings, and a decision-making and treatment plan (including patient instructions regarding when and where to follow-up).

The Cleveland Clinic team was accustomed to using templates and had developed several such tools for the care that they were already providing, whereas the telehealth partner physicians were typically free-texting clinical documentation without a template. Both organizations agreed that templates would likely improve documentation, and, over the next 2 months, templates were developed and shared internally within the Online Care Group. The use of these templates increased steadily, driving the rate of full documentation from 45% initially to 85% after 6 months; currently, the rate of full documentation hovers around 95% to 97%.

Antibiotic Stewardship

The third quality measure, initiated in January 2018, focused on appropriate antibiotic prescriptions. In the spirit of embracing antibiotic stewardship, we identified a coordinated set of strategies that were designed both to treat conditions appropriately and to minimize the potential for antimicrobial resistance.

Both organizations shared and aligned their clinical practice guidelines regarding antibiotic appropriateness and first-line medications. If a physician deviated from the first-line antimicrobial, he or she needed to provide a clear reason for the clinical decision, such as a documented patient allergy to the first-line medication. Baseline data showed that we initially achieved antimicrobial stewardship 69% of the time. By 6 months after our intervention, our antibiotic stewardship had improved to 97%.

The Team

The leadership of this quality improvement initiative consisted of the Medical and Quality Directors of both the Cleveland Clinic and the Online Care Group as well as the President of the Online Care Group.

Metrics Summary

The quality improvement metrics described above can be summarized as follows:

  • The rate of inappropriate pediatric visits decreased from 67% to 1% within 14 months.
  • The rate of full documentation increased from 45% to 85% within 6 months (and currently hovers around 95% to 97%).
  • The rate of appropriate antibiotic prescribing increased from 69% to 97% within 6 months.

Lessons Learned

  • The partnership required both organizations to be fully committed to the goal of becoming the highest-quality medical providers in the on-demand urgent care telehealth space. This commitment helped both organizations to persevere in the collaboration despite initial setbacks in the effort to ensure quality in this evolving field.
  • Both organizations were required to enhance their own digital infrastructures to monitor quality in real time and to provide timely feedback to medical providers.
  • To practice effectively with the constantly evolving technology that supports telemedicine, medical providers must be able to frequently and easily communicate with the quality and information technology teams.
  • Respectful open dialogue, coupled with a clearly articulated vision of providing high-quality medical care in the on-demand urgent care space, has allowed these two organizations to rapidly iterate quality improvement strategies and thrive as telehealth standards and metrics continue to be developed at a national level.

New call for submissions ­to NEJM Catalyst

Now inviting longform articles


A weekly email newsletter featuring the latest actionable ideas and practical innovations from NEJM Catalyst.

Learn More »

More From Care Redesign
Summary of Comprehensive Approach to Physician Behavior and Practice Change

Engaging Stakeholders to Produce Sustainable Change in Surgical Practice

How an initiative designed to improve patient outcomes and satisfaction while containing costs led to sustainable change in surgical practice and physician behavior.

Myths and Realities of Opioid Use Disorder Treatment.

Primary Care and the Opioid-Overdose Crisis — Buprenorphine Myths and Realities

There is a realistic, scalable solution for reaching the millions of Americans with opioid use disorder: mobilizing the primary care physician (PCP) workforce to offer office-based addiction treatment with buprenorphine, as other countries have done.

Coffey02_pullquote family-centered care in medical and surgical procedures

What If Family-Centered Care Were Extended to Medical and Surgical Procedures?

Though the concerns are valid, early experiences suggest that family member engagement may be an effective tool for improving the value of care.

Evidence Needed for Health Systems Change to Address Social Determinants of Health and Obesity and Diet-Related Diseases in Turn

Better Clinical Care for Obesity and Diet-Related Diseases Requires a Focus on Social Determinants of Health

To more effectively treat the problems of obesity and diet-related conditions, health systems need to restructure the traditional medical model of care delivery to address the social determinants of health.

People Living with Dementia Around the World - Value-Based Chronic Illness and Dementia Care

Value-Based Care Must Strengthen Focus on Chronic Illnesses

To effectively control costs and improve value, new models must address our increasingly older patients and chronic care patients, especially those with Alzheimer’s and related dementias.

The Barriers to Excellent Care Vary Widely Across Geographic Regions - both Rural Health Care and Urban Health Care

Survey Snapshot: Rural Health Innovations Born from Challenges

According to NEJM Catalyst Insights Council members, every health system has to develop its own definition of what is meant by “rural” health.

Same-Day Breast Biopsy Workflow at Baylor College of Medicine

How Care Redesign and Process Improvement Can Reduce Patient Fear

Seeing how clinicians take care of their own when they are in frightening situations was the epiphany that led to a same-day breast biopsy program.

Rural Health Care Is Rated Comparable or Worse Across Quadruple Aim Aspects

Care Redesign Survey: Lessons Learned from and for Rural Health

Although care delivery models in rural and urban/suburban areas are distinct, by virtue of geographic density and resource availability, each locale affords lessons for the other.

Comprehensive Intervention Review at Lurie Childrens Hospital - improving patient flow and length of stay

Reducing Length of Stay in the ED

A comprehensive redesign of triage and ED care.

Pumonary Nurse Post-Discharge Follow-Up Note for Patients with COPD

TOPS: Telephonic Outreach in the Pulmonary Service at VA Boston Healthcare System

A nurse-directed intervention targeting veterans who had been hospitalized for COPD resulted in improved access to ambulatory care and a reduced rate of readmissions.


A weekly email newsletter featuring the latest actionable ideas and practical innovations from NEJM Catalyst.

Learn More »


Social Needs

88 Articles

Better Clinical Care for Obesity and…

To more effectively treat the problems of obesity and diet-related conditions, health systems need to…

A Successful Pilot to Improve Access…

Actionable data and modest financial incentives can help motivate clinicians to adjust their behavior around…

Coordinated Care

129 Articles

The Evolution of Primary Care: Embracing…

Primary care must leverage disruptive innovations to ensure that patients receive first-access, comprehensive, coordinated, continuous…

Insights Council

Have a voice. Join other health care leaders effecting change, shaping tomorrow.

Apply Now