Real-World Advice for Generating Real-World Evidence

Article · September 4, 2019

More than a decade ago, the Institute of Medicine advanced the imperative for “a healthcare system that . . . adds to learning throughout the delivery of care, and leads to improvements in the nation’s health.” Achievement of this learning health care system requires “a new clinical research paradigm” based on “stronger synchrony of efforts” between clinical researchers and clinical delivery systems “so that evidence is available when it is needed.” In this new paradigm, the goal would be to leverage a greater proportion of routine care for creating generalizable knowledge. The clinical researchers who contribute to this synchrony might be full-time employees of the delivery systems (a model that would be most effective in the very largest systems) or they might be externally based — often in academic research departments.

For the past 10 years, our two institutions have collaborated in pursuit of this vision. HCA Healthcare (HCA) operates 189 hospitals in 22 U.S. states and in London. HCA maintains a resource to coordinate pragmatic research consisting of an administrative leader, four analysts, a research coordinator, a medical writer, and administrative support. Senior leadership, including the two HCA authors, allocate a portion of their efforts to this partnership, and it accounts for part of the corporate research compliance officer’s effort.

This group works to incorporate identified priorities for pragmatic research into routine clinical operations, in partnership with both academic centers and public agencies. A research partnership with the Harvard Medical School Department of Population Medicine at the Harvard Pilgrim Health Care Institute conducts observational and interventional research on an array of topics including the detection and prevention of health care–associated infections, assessing the safety and effectiveness of medical products, clinician behavior change, and the impact of insurance benefit design on health care utilization and outcomes. Many of these initiatives also are operational priorities at HCA. The partnership is led by the authors and information sharing occurs on the basis of a formal memorandum of understanding and data use agreements.

The partnership has allowed us to pursue ambitious, large-scale clinical trials, positioned within the context of routine clinical care, predominantly in the community hospital setting. Thus far, our successful collaboration includes the REDUCE-MRSA study, a cluster-randomized clinical trial involving 43 HCA hospitals and 74,000 patients that provided strong evidence for universal decolonization in intensive care units to reduce the risk of methicillin resistant S. aureus infections and all-cause bloodstream infections. This study prompted an immediate change in practice throughout HCA, and was a major driver of a change in national practice. A follow-up randomized in 53 HCA hospitals evaluated 340,000 patients outside the critical care environment. It demonstrated that although universal colonization did not provide overall benefit, it did contribute to reduced all-cause bloodstream infections by 31% in the 10% of patients with central vascular catheters or drains who accounted for 56% of these infections. HCA is currently implementing decolonization as a standard procedure for all such patients.

We are now conducting or planning cluster-randomized and stepped-wedge trials to improve antimicrobial stewardship and opioid prescribing in emergency departments, improve glycemic control for inpatients with diabetes, and reduce the risk of readmission for patients with mental illness and multiple medical comorbidities. These pragmatic clinical trials will involve millions of episodes of routine care. In addition, we have partnered to incorporate HCA’s clinical data in nationwide efforts to improve the surveillance of the safety of medical products through the FDA Sentinel Initiative, and to develop better methods for identifying sepsis.

The nation’s health care delivery system could support many more such partnerships between health care providers and researchers. While HCA is one of the largest health care providers in the United States, it represents less than 6% of the nation’s inpatient care delivery. As the trend toward consolidation of provider organizations continues, there are likely an increasing number of entities capable with the scale to support pragmatic research. There are also many opportunities for partnerships involving other sectors of the health care system, such as dialysis centers and nursing homes.

Our lived experience is that a successful partnership requires commitment, shared vision, willingness to understand and accommodate different priorities, and the ability to engage for years. We have learned from our work together that certain “rules of the road” foster success, regardless of the specific nature of the investigation. To encourage more such partnerships, we summarize the following five principles.

Principle 1: Establish and maintain a durable partnership.

The partnership must be symmetric (“win-win”), feel equitable, and be broadly perceived as such. This is perhaps a self-evident principle of any true and equal partnership, but it is easily overlooked and important to address explicitly. While our partnership benefits from its formal governance and legal structure, it only succeeds because of mutual trust, cognizance of each other’s perspective, and prioritization based on shared goals and shared benefit.

Participation must provide benefits that are self-evident at the beginning of the study. Benefits to the health care system from clinical investigation can include the clinician engagement that accompanies a pragmatic study, reputational value, market recognition, and early access to new knowledge of best practices improving clinical and operational outcomes. Clinical investigators must likewise derive academic credit commensurate with their effort. We do not compensate clinicians for participation in pragmatic trials, but our experience is that clinicians will participate even when a trial represents a small additional burden, as long as the science and benefits of the trial are explained and every effort has been made to make participation align with routine clinical workflow.

From an overall financial perspective, the partnership needs to be mutually justifiable and viable. For the academic partner, there is the potential for direct benefit in the form of grant funding, research opportunities for fellows, and opportunity to advance a research portfolio.  For the provider organization, the benefit is largely indirect as the organization should not assume that extramural funding will cover the direct costs of participation in pragmatic research. Beyond the dedicated personnel listed above, HCA staff in various disciplines (e.g., pharmacy, clinical informatics, nursing) may be tasked with supporting design or deployment of a clinical trial in the same way that any other clinical priority may obligate a special effort. HCA justifies the relatively small cost of maintaining a research program through benefits such as clinician engagement, access to new knowledge, skill building, and institutional reputation.

Champion roles need to be identified and resourced before work begins. An empowered champion must provide enterprise leadership, and a clinical champion with the respect of the delivery system is necessary for the success of any investigation. Creativity may be required to engage the right clinical leaders to successfully serve as champions. HCA routinely relies on facility chief medical officers to champion study implementation, emphasizing the positive relationship between pragmatic research and physician engagement and recruitment, GME program status, and institutional reputation. Non-physician leaders may also be key champions, based on the nature of the investigation. Examples based on recent efforts within our program include leaders in critical care, nursing, infection prevention, behavioral health, and pharmacy.

Credit — in the form of authorship, presentations in public meetings, etc. — is equitably shared, and sharing is determined a priori. Any potential for development of intellectual property, such as patents or licensable products, needs to be addressed at the outset, and in a manner that strengthens the partnership. Our collaborations to date have focused on products such as tool kits, intended for public domain, and so issues relating to intellectual property have been theoretical but nevertheless important to address.

All participants must have the support needed to effectively function in their roles. Even fully trained health services researchers and academic clinical trialists require substantial acculturation and experience to function effectively as members of teams conducting systems-based research. In our experience, this process often requires 2 years. Even then, it is important to have clear mechanisms to identify and address situations in which one of the partners requests the other to operate beyond their current capability.

Principle 2: Select research topics of mutual interest.

The research question has to be clinically important in that it clarifies alternatives and provides an approach that is likely to be adopted. The clinical value of a study needs to be easily justified to both clinicians and leaders within the health care system, and in a domain with relevance to the patients of that system.

Research questions must be operationally and/or strategically important. Investigations within the delivery system work best when there is strong alignment between the research and the clinical imperatives of the system. If the system is merely a host, the study will be difficult to sustain. At the same time, any structured intervention or evaluation cannot be perceived to negatively impact routine clinical care.

Principle 3: Give precedence to operational imperatives.

Questions must be inserted into the operational cadence. Clinical research and clinical operations often function on different timetables. The health care system cannot delay implementation of operational priorities for clinical research, nor can it always adopt interventions according to the timetable required by an external research sponsor. Creative ways need to be employed to overlay the rigors of scientific investigation onto operational priorities.

Academic investigators should consider the need to accommodate changing institutional priorities and the risks this poses to competitive grants and peer-reviewed publications. This risk can be partly ameliorated, but not entirely eliminated, by careful selection of research topics.

Activity has to be largely transparent or similar to normal workflow. Clinicians typically enjoy participating in the generation of new knowledge and will embrace participation in a study, but only to the extent that their work processes are either undisturbed or improved.

Principle 4: Recognize that data security is essential.

In virtually all investigations, the health care system allows external investigators access to sensitive clinical data. Doing so requires airtight security systems that avoid transfer or duplication of data. For our collaboration, we conduct all research data analysis behind HCA’s firewall in a standalone research computing environment, with remote access for the investigator team that is in alignment with HIPAA requirements.

Principle 5: Build data systems for learning health care.

Traditional electronic health data are transactional systems and are generally customized by individual institutions, limiting the capacity for aggregation of data. Building on the basic standards put forward through the Centers for Medicare and Medicaid’s requirements for meaningful use of certified electronic health record technology, HCA designed its clinical data warehouse for incorporation of both structured and unstructured data across the continuum of care. This serves as a platform for rapid learning and flexible inquiry, allows us to study both anticipated and unanticipated questions, and improves data quality with every additional inquiry. While our collaborations would not be possible without this data warehouse, each of our investigations has required substantial and often time-consuming data curation in order to develop an analysis-quality data set.

Review and Revise the Collaborative Projects as Needed

We refer to these principles as we evaluate potential new projects and troubleshoot or revise existing projects. Yet these “rules” are dynamic, and we fully expect that they will grow and adapt alongside our partnership.

While pressures on health care delivery systems and the research community for efficiency and productivity continue to grow, our commitment to partnering in pragmatic research to improve health outcomes remains strong. Attention to the principles for successful collaboration is an important contributor to our success and will likely be useful for other organizations looking to replicate such partnerships.

Call for submissions:

Now inviting expert articles, longform articles, and case studies for peer review


A weekly email newsletter featuring the latest actionable ideas and practical innovations from NEJM Catalyst.

Learn More »

More From Leadership
Fiscus01_pullquote - humanizing physician performance review

Humanizing the Annual Physician Performance Review

Transforming the review process from a punitive, deflating experience to a valuable one that strengthens the relationship between physician and organization.

People Believe Strongly That Leadership Can Be Taught

Leadership Survey: Leadership Skills Are Teachable and Vital

Leadership is teachable, and leadership development and training are important, according to our survey on the topic. Yet the same survey reveals that more than half of respondents think their organizations’ efforts to develop and train leaders are lacking in quality and time commitment.

A Preliminary Model of Determinants and Consequences of Unhurried Conversations with Patients

Careful and Kind Care Requires Unhurried Conversations

Health care providers must have time to know their patients in “high definition” to best meet their needs.

From the Commonwealth to Obamacare: Reflections on 10+ Years of Expanding Health Insurance Coverage

The former Executive Director of the Commonwealth Health Insurance Connector — a model for the Affordable Care Act and other state marketplaces — reflects on what worked, what didn’t, and what could be done differently in both Massachusetts and at the federal level.

Time Spent Engaging Directly with 16 Camden RESET Participants or Coordinating Care on Their Behalf

“Putting All the Pieces Back”: Lessons from a Health Care–Led Jail Reentry Pilot

The Camden Coalition’s jail-based reentry program illuminated the necessity and challenges of engaging people with complex health and social needs and helping to transform the systems that serve them.

The Largest Share of Organizations Do Not Have a Formal Strategy for Clinician Engagement

Leadership Survey: Why Clinicians Are Not Engaged, and What Leaders Must Do About It

Clinician engagement is vital for improving clinical quality and patient satisfaction, as well as the job satisfaction of clinicians themselves. Yet nearly half of health care organizations are not very effective or not at all effective at clinician engagement.

Rowe01_pullquote - clinician well-being - fighting clinician burnout and creating culture of wellness takes all stakeholders

Defending the Term “Burnout”: A Useful Tool in the Quest to Ease Clinician Suffering

Health care leaders must take a preemptive approach to clinician well-being that is supported by all stakeholders and prioritized on an equal footing with essential clinical and financial measures.

Screenshot from the NewYork Quality Care Chronic Condition Dashboard

Success in a Hospital-Integrated Accountable Care Organization

How NewYork Quality Care achieved shared savings — by strengthening collaboration, enhancing care management with telehealth, and transparently sharing performance data.

Miller03_pullquote social determinants whole-person

How a State Advances Whole-Person Health Care

Pennsylvania addresses social determinants of health by bringing together managed care and social services organizations to expand access to vital resources.

Abigail Geisinger Scholars Program for Medical Students -Ryu02_pullquote

Why a Teaching Hospital Offers an Employment-Based Tuition Waiver Program

Geisinger Commonwealth School of Medicine subsidizes medical students’ education in exchange for their willingness to practice at Geisinger Health System.


A weekly email newsletter featuring the latest actionable ideas and practical innovations from NEJM Catalyst.

Learn More »


Leading Transformation

284 Articles

From the Commonwealth to Obamacare: Reflections…

The former Executive Director of the Commonwealth Health Insurance Connector — a model for the…

Physician Burnout

53 Articles

Humanizing the Annual Physician Performance Review

Transforming the review process from a punitive, deflating experience to a valuable one that strengthens…

From the Commonwealth to Obamacare: Reflections…

The former Executive Director of the Commonwealth Health Insurance Connector — a model for the…

Insights Council

Have a voice. Join other health care leaders effecting change, shaping tomorrow.

Apply Now