Care Redesign

Ensuring Clinical Quality in Telemedicine

Case Study · August 2, 2018

In September 2014, Cleveland Clinic partnered with the Online Care Group, a nationwide system of physicians who provide telemedicine services through the American Well online platform, to deliver care via video visits under the Cleveland Clinic name.

Cleveland Clinic’s original aims in introducing telemedicine were to increase patient access to care, support the transition to value-based care, improve system efficiency, and expand the system’s reach to new patients. To ensure that the Online Care Group physicians practiced at or above the clinical standards set by the Cleveland Clinic, the two organizations embarked on a multiyear collaboration to improve the quality of care delivered through the online platform.

Key Takeaways

  1. Health systems and third-party providers of telemedicine clinical services can successfully redesign their processes to improve the quality of care provided by a nationwide system of physicians.

  2. Establishing regular and frequent communication between the leadership of health systems and that of third-party telemedicine providers allows for quick identification of problems and development of solutions.

  3. Both parties must be willing and able to work on quality initiatives and must be equally engaged in identifying and improving clinical metrics.

  4. Technology assets should be used to their fullest potential. In our case, following the creation of standardized templates that prompted physicians across the country to adhere to consistent clinical standards, the percentage of notes that met full documentation standards increased from 45% to 85%.

  5. When communicating with a group of distributed physicians working virtually, it is important to communicate through multiple channels (e.g., email, e-newsletters, webinars, online discussion boards, phone calls, and individual calls) to ensure that all physicians are aware of quality improvement measures and guideline changes. In our case, following the implementation of several measures designed to improve communication, the rate of inappropriate pediatric telemedicine visits decreased from 67% to 1% within 6 months.

The Challenge

Professional organizations such as the American Medical Association and the American Academy of Pediatrics have raised quality concerns around telemedicine, citing the lack of care standards and nationally recognized clinical guidelines. As the urgent care clinical services provided by the Online Care Group were delivered under the Cleveland Clinic name, our challenge was to ensure that Online Care Group physicians met the same high-quality clinical standards that Cleveland Clinic upholds across the organization.

The Goal

Our goal was to create an effective quality improvement collaboration between the Cleveland Clinic and the Online Care Group with a focus on ensuring visit-type appropriateness, documentation completion, and appropriate antibiotic prescriptions.

The Execution

The Platform and Partnership

In September 2014, the Cleveland Clinic began virtual visits and, in June 2015, it launched Express Care Online, an on-demand urgent care telemedicine service (hosted on the American Well platform) that provides 24/7/365 care for patients >2 years of age. Recognizing that it did not have the internal infrastructure or capacity to independently support this new service, Cleveland Clinic partnered with physicians of the Online Care Group.

Off-hours telephone coverage and triage provided by an external group is not uncommon in health care; however, establishing an on-demand urgent care service that is staffed by physicians and that enables providers to diagnose and treat patients requires more stringent clinical quality protocols. Clinical quality equal to or exceeding the metrics established for brick-and-mortar facilities was paramount to this program. Although the Online Care Group monitored quality with use of well-established metrics that had been developed and tested since the group’s inception, these metrics required alignment with existing Cleveland Clinic standards to meet quality goals.

Establishing Quality Review Meetings

Early on, both organizations determined that frequent and open communication was necessary to reach our quality goals. To that end, we established monthly quality review meetings that allowed for the bidirectional sharing of information regarding quality metrics and outcomes. Attendance was mandatory for the Medical and Quality Directors of both organizations; however, the entire clinical team was also invited to attend and participate in the conversation. We designed the initial quality improvement plan around four interrelated categories: Organizational Alignment, Clinical Guidelines, Collaborative Partnerships, and Quality Assurance and Improvement.

Telemedicine Online Care Group Cleveland Clinic Express Care Online Quality Improvement - Next Steps

Chart summarizing the initial quality improvement plan, showing progressive goals and feedback loops. Click To Enlarge.

Determining the Appropriate Scope of Telemedicine

As telemedicine was a new type of care delivery for Cleveland Clinic, the first goal was to establish guidelines to help determine which visits were appropriate for telemedicine without an in-person referral. At each meeting, the Quality Directors of both organizations reviewed the audit of approximately 30% of the total number of adult patient visits and 100% of pediatric visits from the preceding month for clinical appropriateness.

The operational definition of an “inappropriate visit” was “a completed visit leading to a treatment plan in the absence of one or more physical examination components considered to be necessary to arrive at the diagnosis or clinical decision.” Examples included the diagnosis and treatment of acute otitis media in a patient in whom the tympanic membrane had not been visualized and the diagnosis and treatment of a urinary tract infection in a pediatric patient without a urine analysis and culture.

Initial audits by the Cleveland Clinic’s Express Care Online Quality Director revealed that up to 67% of the previous month’s pediatric visits had been inappropriate. As the Medical and Quality Directors from both organizations were committed to driving down the number of inappropriate visits and facilitating high-quality telehealth care, both organizations quickly committed to a quality improvement plan designed to eliminate inappropriate visits.

Transparency and Communication          

Over the next 6 months, both organizations implemented an intensive program to train physicians on appropriate and inappropriate visit types. Both organizations were transparent with their provider teams regarding the results of the chart audit and the goal of the initiative. The Online Care Group used email campaigns and online training to increase provider awareness of the initiative, including updated visit guidelines and remediation plans if the guidelines were not followed.

Overcoming Barriers to Adoption of New Guidelines

Initially, the intensive campaign showed little reduction in inappropriate visit types. Both organizations quickly worked together and identified three system barriers impeding the timely and consistent distribution of and adherence to guidelines. First, the Online Care Group had low attendance at live webinars because its physicians practiced in different time zones and had busy in-person clinical schedules. Second, the physicians had no online forum where they could gather and discuss policy changes. Third, the Online Care Group works with other health systems in addition to Cleveland Clinic and had not yet developed an easy method for communicating different health system standards for visit appropriateness to their physicians.

To overcome these barriers, the Online Care Group developed enhancements to its electronic medical record, including the use of templates to prompt the physician on the standards for an appropriate visit type. The Online Care Group also developed standing virtual team meetings; a weekly email-based newsletter; a private, online physician discussion board; and guidelines on when to trigger one-on-one remediation conversations. Within 6 months, the baseline measure of inappropriate visits declined from 67% to 1%. The significant and rapid improvement in adherence to visit appropriateness guidelines was encouraging to both groups and provided a solid foundation for further quality improvement initiatives.

Appropriate Documentation

Based on the successes of the initial quality improvement initiative, the two organizations decided to tackle appropriate documentation as a second quality measure in mid-2016. All parties agreed that minimum documentation elements required for an online telehealth visit should mirror in-person requirements, including patient identifiers, allergies, medical and surgical history, medications, the history of the current problem, physical examination findings, and a decision-making and treatment plan (including patient instructions regarding when and where to follow-up).

The Cleveland Clinic team was accustomed to using templates and had developed several such tools for the care that they were already providing, whereas the telehealth partner physicians were typically free-texting clinical documentation without a template. Both organizations agreed that templates would likely improve documentation, and, over the next 2 months, templates were developed and shared internally within the Online Care Group. The use of these templates increased steadily, driving the rate of full documentation from 45% initially to 85% after 6 months; currently, the rate of full documentation hovers around 95% to 97%.

Antibiotic Stewardship

The third quality measure, initiated in January 2018, focused on appropriate antibiotic prescriptions. In the spirit of embracing antibiotic stewardship, we identified a coordinated set of strategies that were designed both to treat conditions appropriately and to minimize the potential for antimicrobial resistance.

Both organizations shared and aligned their clinical practice guidelines regarding antibiotic appropriateness and first-line medications. If a physician deviated from the first-line antimicrobial, he or she needed to provide a clear reason for the clinical decision, such as a documented patient allergy to the first-line medication. Baseline data showed that we initially achieved antimicrobial stewardship 69% of the time. By 6 months after our intervention, our antibiotic stewardship had improved to 97%.

The Team

The leadership of this quality improvement initiative consisted of the Medical and Quality Directors of both the Cleveland Clinic and the Online Care Group as well as the President of the Online Care Group.

Metrics Summary

The quality improvement metrics described above can be summarized as follows:

  • The rate of inappropriate pediatric visits decreased from 67% to 1% within 14 months.
  • The rate of full documentation increased from 45% to 85% within 6 months (and currently hovers around 95% to 97%).
  • The rate of appropriate antibiotic prescribing increased from 69% to 97% within 6 months.

Lessons Learned

  • The partnership required both organizations to be fully committed to the goal of becoming the highest-quality medical providers in the on-demand urgent care telehealth space. This commitment helped both organizations to persevere in the collaboration despite initial setbacks in the effort to ensure quality in this evolving field.
  • Both organizations were required to enhance their own digital infrastructures to monitor quality in real time and to provide timely feedback to medical providers.
  • To practice effectively with the constantly evolving technology that supports telemedicine, medical providers must be able to frequently and easily communicate with the quality and information technology teams.
  • Respectful open dialogue, coupled with a clearly articulated vision of providing high-quality medical care in the on-demand urgent care space, has allowed these two organizations to rapidly iterate quality improvement strategies and thrive as telehealth standards and metrics continue to be developed at a national level.

Call for submissions:

Now inviting expert articles, longform articles, and case studies for peer review


A weekly email newsletter featuring the latest actionable ideas and practical innovations from NEJM Catalyst.

Learn More »

More From Care Redesign
The Assessment of Care Tool - Consisting of Six Visual Analog Scales Corresponding with the IOM Six Dimensions of Perfect Care

Real-Time Pursuit of Outcomes That Matter to Patients

A simple and affordable tool to use at the point of care to drive value creation within clinical microsystems.

Many Patients Who Would Benefit from Palliative - End-of-Life Care Do Not Receive It

Care Redesign Survey: The Power of Palliative Care

Our NEJM Catalyst Insights Council survey on palliative care reveals an interesting dichotomy: While the great majority of organizations have a palliative or end-of-life care program, 60% of patients who would benefit from such services don’t receive them.

Heart Safe Motherhood and Way to Health Two-Way Texting for Blood Pressure Monitoring for Postpartum Women with Preeclampsia

Heart Safe Motherhood: Applying Innovation Methodology for Improved Maternal Outcomes

At the Hospital of the University of Pennsylvania, a text message–based blood pressure surveillance program for postpartum women with preeclampsia improved blood pressure management, reduced readmissions, and increased patient and provider satisfaction.

VHA Whole Health System diagram

Finding the Cause of the Crises: Opioids, Pain, Suicide, Obesity, and Other “Epidemics”

Until we redesign our health care system to address our patients’ personal determinants of health, we will continue to inadequately address our multiple chronic disease crises.

Leff06_pullquote home-based medical care for homebound patients

Using Quality to Shine a Light on Homebound Care

How two thought leaders in the fields of home-based medical care, geriatrics, and palliative medicine advanced a quality-of-care agenda for homebound adults.

Charlotte Yeh head shot - hearing aids hearing loss

“You’re Old Without Hearing Aids”— Addressing the Silent Epidemic of Hearing Loss

Hearing loss isn’t a normal consequence of aging. But it is associated with a higher risk of dementia, depression, and falls. The Chief Medical Officer for AARP Services talks about combating this huge but silent epidemic that impacts all ages.

Dentzer01_pullquote - Stone-Age Policies Stifle Modern Virtual Care Solutions

Stone-Age Policies Stifle Modern Solutions

Health care leaders must advocate for regulatory and reimbursement changes to unlock the potential of innovative technology and care team approaches to Parkinson’s and other suitable conditions.

Idiopathic Pulmonary Fibrosis IPF Multidisciplinary Collaborative Care Model

From Consulting to Caring: Care Redesign in Idiopathic Pulmonary Fibrosis

A multidisciplinary collaborative model to address the palliative care needs of patients with idiopathic pulmonary fibrosis resulted in improved end-of-life care and decreased hospital deaths.

Impact of PCSP on Patient Satisfaction at Providence Heart Clinic

Transforming Specialty Practice in Pursuit of Value-Based Care: Results from an Integrated Cardiology Practice

Despite significant primary care reform around patient-centered medical home models, specialty care remains fragmented, with poor communication between primary care and specialists. How should specialty practices be reformed to deliver more coordinated, patient-centered care?

Michael Bennick Yale New Haven Hospital Medical Director of the Patient Experience - Yale Living History Project

The Living History Project: Open-Ended Patient Interviews Create a Therapeutic Bridge

A program at Yale has students conduct open-ended interviews with patients about their lives, their hopes, their values, and what they most want their medical team to know — creating the opportunity for human connection and a better care experience.


A weekly email newsletter featuring the latest actionable ideas and practical innovations from NEJM Catalyst.

Learn More »


Coordinated Care

136 Articles

Care Redesign Survey: The Power of…

Our NEJM Catalyst Insights Council survey on palliative care reveals an interesting dichotomy: While the…

Primary Care

188 Articles

China’s Health Care: Public Benefit, Enormous…

Though demand for health services in China has outpaced the development of management systems to…

Mental Health

30 Articles

Finding the Cause of the Crises:…

Until we redesign our health care system to address our patients’ personal determinants of health,…

Insights Council

Have a voice. Join other health care leaders effecting change, shaping tomorrow.

Apply Now