Care Redesign

Ensuring Clinical Quality in Telemedicine

Case Study · August 2, 2018

In September 2014, Cleveland Clinic partnered with the Online Care Group, a nationwide system of physicians who provide telemedicine services through the American Well online platform, to deliver care via video visits under the Cleveland Clinic name.

Cleveland Clinic’s original aims in introducing telemedicine were to increase patient access to care, support the transition to value-based care, improve system efficiency, and expand the system’s reach to new patients. To ensure that the Online Care Group physicians practiced at or above the clinical standards set by the Cleveland Clinic, the two organizations embarked on a multiyear collaboration to improve the quality of care delivered through the online platform.

Key Takeaways

  1. Health systems and third-party providers of telemedicine clinical services can successfully redesign their processes to improve the quality of care provided by a nationwide system of physicians.

  2. Establishing regular and frequent communication between the leadership of health systems and that of third-party telemedicine providers allows for quick identification of problems and development of solutions.

  3. Both parties must be willing and able to work on quality initiatives and must be equally engaged in identifying and improving clinical metrics.

  4. Technology assets should be used to their fullest potential. In our case, following the creation of standardized templates that prompted physicians across the country to adhere to consistent clinical standards, the percentage of notes that met full documentation standards increased from 45% to 85%.

  5. When communicating with a group of distributed physicians working virtually, it is important to communicate through multiple channels (e.g., email, e-newsletters, webinars, online discussion boards, phone calls, and individual calls) to ensure that all physicians are aware of quality improvement measures and guideline changes. In our case, following the implementation of several measures designed to improve communication, the rate of inappropriate pediatric telemedicine visits decreased from 67% to 1% within 6 months.

The Challenge

Professional organizations such as the American Medical Association and the American Academy of Pediatrics have raised quality concerns around telemedicine, citing the lack of care standards and nationally recognized clinical guidelines. As the urgent care clinical services provided by the Online Care Group were delivered under the Cleveland Clinic name, our challenge was to ensure that Online Care Group physicians met the same high-quality clinical standards that Cleveland Clinic upholds across the organization.

The Goal

Our goal was to create an effective quality improvement collaboration between the Cleveland Clinic and the Online Care Group with a focus on ensuring visit-type appropriateness, documentation completion, and appropriate antibiotic prescriptions.

The Execution

The Platform and Partnership

In September 2014, the Cleveland Clinic began virtual visits and, in June 2015, it launched Express Care Online, an on-demand urgent care telemedicine service (hosted on the American Well platform) that provides 24/7/365 care for patients >2 years of age. Recognizing that it did not have the internal infrastructure or capacity to independently support this new service, Cleveland Clinic partnered with physicians of the Online Care Group.

Off-hours telephone coverage and triage provided by an external group is not uncommon in health care; however, establishing an on-demand urgent care service that is staffed by physicians and that enables providers to diagnose and treat patients requires more stringent clinical quality protocols. Clinical quality equal to or exceeding the metrics established for brick-and-mortar facilities was paramount to this program. Although the Online Care Group monitored quality with use of well-established metrics that had been developed and tested since the group’s inception, these metrics required alignment with existing Cleveland Clinic standards to meet quality goals.

Establishing Quality Review Meetings

Early on, both organizations determined that frequent and open communication was necessary to reach our quality goals. To that end, we established monthly quality review meetings that allowed for the bidirectional sharing of information regarding quality metrics and outcomes. Attendance was mandatory for the Medical and Quality Directors of both organizations; however, the entire clinical team was also invited to attend and participate in the conversation. We designed the initial quality improvement plan around four interrelated categories: Organizational Alignment, Clinical Guidelines, Collaborative Partnerships, and Quality Assurance and Improvement.

Telemedicine Online Care Group Cleveland Clinic Express Care Online Quality Improvement - Next Steps

Chart summarizing the initial quality improvement plan, showing progressive goals and feedback loops. Click To Enlarge.

Determining the Appropriate Scope of Telemedicine

As telemedicine was a new type of care delivery for Cleveland Clinic, the first goal was to establish guidelines to help determine which visits were appropriate for telemedicine without an in-person referral. At each meeting, the Quality Directors of both organizations reviewed the audit of approximately 30% of the total number of adult patient visits and 100% of pediatric visits from the preceding month for clinical appropriateness.

The operational definition of an “inappropriate visit” was “a completed visit leading to a treatment plan in the absence of one or more physical examination components considered to be necessary to arrive at the diagnosis or clinical decision.” Examples included the diagnosis and treatment of acute otitis media in a patient in whom the tympanic membrane had not been visualized and the diagnosis and treatment of a urinary tract infection in a pediatric patient without a urine analysis and culture.

Initial audits by the Cleveland Clinic’s Express Care Online Quality Director revealed that up to 67% of the previous month’s pediatric visits had been inappropriate. As the Medical and Quality Directors from both organizations were committed to driving down the number of inappropriate visits and facilitating high-quality telehealth care, both organizations quickly committed to a quality improvement plan designed to eliminate inappropriate visits.

Transparency and Communication          

Over the next 6 months, both organizations implemented an intensive program to train physicians on appropriate and inappropriate visit types. Both organizations were transparent with their provider teams regarding the results of the chart audit and the goal of the initiative. The Online Care Group used email campaigns and online training to increase provider awareness of the initiative, including updated visit guidelines and remediation plans if the guidelines were not followed.

Overcoming Barriers to Adoption of New Guidelines

Initially, the intensive campaign showed little reduction in inappropriate visit types. Both organizations quickly worked together and identified three system barriers impeding the timely and consistent distribution of and adherence to guidelines. First, the Online Care Group had low attendance at live webinars because its physicians practiced in different time zones and had busy in-person clinical schedules. Second, the physicians had no online forum where they could gather and discuss policy changes. Third, the Online Care Group works with other health systems in addition to Cleveland Clinic and had not yet developed an easy method for communicating different health system standards for visit appropriateness to their physicians.

To overcome these barriers, the Online Care Group developed enhancements to its electronic medical record, including the use of templates to prompt the physician on the standards for an appropriate visit type. The Online Care Group also developed standing virtual team meetings; a weekly email-based newsletter; a private, online physician discussion board; and guidelines on when to trigger one-on-one remediation conversations. Within 6 months, the baseline measure of inappropriate visits declined from 67% to 1%. The significant and rapid improvement in adherence to visit appropriateness guidelines was encouraging to both groups and provided a solid foundation for further quality improvement initiatives.

Appropriate Documentation

Based on the successes of the initial quality improvement initiative, the two organizations decided to tackle appropriate documentation as a second quality measure in mid-2016. All parties agreed that minimum documentation elements required for an online telehealth visit should mirror in-person requirements, including patient identifiers, allergies, medical and surgical history, medications, the history of the current problem, physical examination findings, and a decision-making and treatment plan (including patient instructions regarding when and where to follow-up).

The Cleveland Clinic team was accustomed to using templates and had developed several such tools for the care that they were already providing, whereas the telehealth partner physicians were typically free-texting clinical documentation without a template. Both organizations agreed that templates would likely improve documentation, and, over the next 2 months, templates were developed and shared internally within the Online Care Group. The use of these templates increased steadily, driving the rate of full documentation from 45% initially to 85% after 6 months; currently, the rate of full documentation hovers around 95% to 97%.

Antibiotic Stewardship

The third quality measure, initiated in January 2018, focused on appropriate antibiotic prescriptions. In the spirit of embracing antibiotic stewardship, we identified a coordinated set of strategies that were designed both to treat conditions appropriately and to minimize the potential for antimicrobial resistance.

Both organizations shared and aligned their clinical practice guidelines regarding antibiotic appropriateness and first-line medications. If a physician deviated from the first-line antimicrobial, he or she needed to provide a clear reason for the clinical decision, such as a documented patient allergy to the first-line medication. Baseline data showed that we initially achieved antimicrobial stewardship 69% of the time. By 6 months after our intervention, our antibiotic stewardship had improved to 97%.

The Team

The leadership of this quality improvement initiative consisted of the Medical and Quality Directors of both the Cleveland Clinic and the Online Care Group as well as the President of the Online Care Group.

Metrics Summary

The quality improvement metrics described above can be summarized as follows:

  • The rate of inappropriate pediatric visits decreased from 67% to 1% within 14 months.
  • The rate of full documentation increased from 45% to 85% within 6 months (and currently hovers around 95% to 97%).
  • The rate of appropriate antibiotic prescribing increased from 69% to 97% within 6 months.

Lessons Learned

  • The partnership required both organizations to be fully committed to the goal of becoming the highest-quality medical providers in the on-demand urgent care telehealth space. This commitment helped both organizations to persevere in the collaboration despite initial setbacks in the effort to ensure quality in this evolving field.
  • Both organizations were required to enhance their own digital infrastructures to monitor quality in real time and to provide timely feedback to medical providers.
  • To practice effectively with the constantly evolving technology that supports telemedicine, medical providers must be able to frequently and easily communicate with the quality and information technology teams.
  • Respectful open dialogue, coupled with a clearly articulated vision of providing high-quality medical care in the on-demand urgent care space, has allowed these two organizations to rapidly iterate quality improvement strategies and thrive as telehealth standards and metrics continue to be developed at a national level.

Call for submissions:

Now inviting expert articles, longform articles, and case studies for peer review


A weekly email newsletter featuring the latest actionable ideas and practical innovations from NEJM Catalyst.

Learn More »

More From Care Redesign
Murali01_pullquote Home Recovery Care patient satisfaction

No Place Like Home: Bringing Inpatient Care to the Patient

Providing home-based acute care improves patient satisfaction and care quality while reducing costs.

Rating the Raters - Strengths and Weaknesses Assessment of the Four Public Hospital Quality Rating Systems - 2a

Rating the Raters: An Evaluation of Publicly Reported Hospital Quality Rating Systems

Some promising innovation is taking place among organizations that rate hospital performance, but major systemic change is needed in the field to ensure access to meaningful comparisons through better data and relevant metrics, and to establish integrated oversight through robust audits and peer review.

McKee01_pullquote - the need for coordinated care IPUs for Parkinson's disease

Creating “One-Stop Shop” Care for Parkinson’s

Integrated Practice Units (IPUs) can revolutionize the care of specialty disease conditions, and Parkinson’s disease is a good place to start.

Good Shepherd culturally competent hospice care home visit to the widow of a recently deceased patient

Strangers No More: Culturally Competent Add-On Programs for Diverse Seniors

Creating specialized culturally competent programs to improve patient satisfaction and address the unique health care needs of older immigrants.

Kimberly Dennis and David Newton head shots - SunCloud Mental Health Services

Addressing the Lack of Continuity of Care in Mental Health Services

Co-founders of an integrative outpatient treatment center for mental health and addiction discuss the problematic lack of continuity of care between inpatient and outpatient and physical and mental health services.

Epic OB Hemorrhage PPH Risk Assessment Tool and Alerts at PSJH - Preventing Maternal Death

Improving Maternal Safety Through an EMR Risk Assessment

After a mother died of postpartum hemorrhage, Providence St. Joseph Health made organization-wide changes to prevent future maternal deaths and injuries.

Most Health Care Organizations Have Palliative or End-of-Life Care Programs

Survey Snapshot: Challenging the Resistance to “Palliative”

NEJM Catalyst Insights Council members agree that palliative care is gaining traction, but one of many barriers is getting providers over their resistance to the word “palliative.”

The Assessment of Care Tool - Consisting of Six Visual Analog Scales Corresponding with the IOM Six Dimensions of Perfect Care

Real-Time Pursuit of Outcomes That Matter to Patients

A simple and affordable tool to use at the point of care to drive value creation within clinical microsystems.

Many Patients Who Would Benefit from Palliative - End-of-Life Care Do Not Receive It

Care Redesign Survey: The Power of Palliative Care

Our NEJM Catalyst Insights Council survey on palliative care reveals an interesting dichotomy: While the great majority of organizations have a palliative or end-of-life care program, 60% of patients who would benefit from such services don’t receive them.

Heart Safe Motherhood and Way to Health Two-Way Texting for Blood Pressure Monitoring for Postpartum Women with Preeclampsia

Heart Safe Motherhood: Applying Innovation Methodology for Improved Maternal Outcomes

At the Hospital of the University of Pennsylvania, a text message–based blood pressure surveillance program for postpartum women with preeclampsia improved blood pressure management, reduced readmissions, and increased patient and provider satisfaction.


A weekly email newsletter featuring the latest actionable ideas and practical innovations from NEJM Catalyst.

Learn More »


Coordinated Care

141 Articles

Creating “One-Stop Shop” Care for Parkinson’s

Integrated Practice Units (IPUs) can revolutionize the care of specialty disease conditions, and Parkinson’s disease…

Physician Incentives: Importance of the Why

When moving health care providers forward in the world of value-based care and population health,…

Primary Care

196 Articles

No Place Like Home: Bringing Inpatient…

Providing home-based acute care improves patient satisfaction and care quality while reducing costs.

Insights Council

Have a voice. Join other health care leaders effecting change, shaping tomorrow.

Apply Now