New Marketplace

Predicting the Future — Big Data, Machine Learning, and Clinical Medicine

Article · October 10, 2016

By now, it’s almost old news: big data will transform medicine. It’s essential to remember, however, that data by themselves are useless. To be useful, data must be analyzed, interpreted, and acted on. Thus, it is algorithms — not data sets — that will prove transformative. We believe, therefore, that attention has to shift to new statistical tools from the field of machine learning that will be critical for anyone practicing medicine in the 21st century.

First, it’s important to understand what machine learning is not. Most computer-based algorithms in medicine are “expert systems” — rule sets encoding knowledge on a given topic, which are applied to draw conclusions about specific clinical scenarios, such as detecting drug interactions or judging the appropriateness of obtaining imaging. Expert systems work the way an ideal medical student would: they take general principles about medicine and apply them to new patients.

Machine learning, conversely, approaches problems as a doctor progressing through residency might: by learning rules from data. Starting with patient-level observations, algorithms sift through vast numbers of variables, looking for combinations that reliably predict outcomes. In one sense, this process is similar to that of traditional regression models: there are outcomes, covariates, and statistical functions linking the two. But where machine learning shines is in handling enormous numbers of predictors — sometimes, remarkably, more predictors than observations — and combining them in nonlinear and highly interactive ways.1 This capacity allows us to use new kinds of data, whose sheer volume or complexity would previously have made analyzing them unimaginable.

Consider a chest radiograph. Some radiographic features might predict an important outcome, such as death. In a standard statistical model, we might use the radiograph’s interpretation — “normal,” “atelectasis,” “effusion” — as a variable. But instead, why not let the data speak for themselves? Leveraging dramatic advances in computational power, digital pixel matrixes underlying radiographs become millions of individual variables. Algorithms then go to work, clustering pixels into lines and shapes and ultimately learning contours of fracture lines, parenchymal opacities, and more. Even traditional insurance claims data can take on a new life: diagnostic codes trace an intricate, dynamic picture of patients’ medical histories, far richer than the static variables for coexisting conditions used in standard statistical models.

Of course, letting the data speak for themselves can be problematic. Algorithms might “overfit” predictions to spurious correlations in the data, or multiple collinear, correlated predictors could produce unstable estimates. Either possibility can lead to overly optimistic estimates of the accuracy of a model and exaggerated claims about real-world performance. These concerns are serious and must be addressed by testing models on truly independent validation data sets, from different populations or periods that played no role in model development. In this way, problems in the model-fitting stage, whatever their cause, will show up as poor performance in the validation stage. This principle is so important that in many data-science competitions, validation data are released only after teams upload their final algorithms built on another publicly available data set.

Another key issue is the quantity and quality of input data. Machine learning algorithms are highly data hungry, often requiring millions of observations to reach acceptable performance levels.2 In addition, biases in data collection can substantially affect both performance and generalizability. Lactate might be a good predictor of the risk of death, for example, but only a small, nonrepresentative sample of patients have their lactate levels checked. Private companies spend enormous resources to amass high-quality, unbiased data to feed their algorithms, and existing data in electronic health records (EHRs) or claims databases need careful curation and processing to become usable.

Finally, machine learning does not solve any of the fundamental problems of causal inference in observational data sets. Algorithms may be good at predicting outcomes, but predictors are not causes.3 The usual commonsense caveats about confusing correlation with causation apply; indeed, they become even more important as researchers begin including millions of variables in statistical models.

Machine learning has become ubiquitous and indispensable for solving complex problems in most sciences. In astronomy, algorithms sift through millions of images from telescope surveys to classify galaxies and find supernovas. In biomedicine, machine learning can predict protein structure and function from genetic sequences and discern optimal diets from patients’ clinical and microbiome profiles. The same methods will open up vast new possibilities in medicine. A striking example: algorithms can read cortical activity directly from the brain, transmitting signals from a paralyzed human’s motor cortex to hand muscles and restoring motor control.4 These advances would have been unimaginable without machine learning to process real-time, high-resolution physiological data.

Increasingly, the ability to transform data into knowledge will disrupt at least three areas of medicine. First, machine learning will dramatically improve the ability of health professionals to establish a prognosis. Current prognostic models (e.g., the Acute Physiology and Chronic Health Evaluation [APACHE] score and the Sequential Organ Failure Assessment [SOFA] score) are restricted to only a handful of variables, because humans must enter and tally the scores. But data could instead be drawn directly from EHRs or claims databases, allowing models to use thousands of rich predictor variables. Does doing so lead to better predictions? Early evidence from our own ongoing work, using machine learning to predict death in patients with metastatic cancer, provides some indication: we can precisely identify large patient subgroups with mortality rates approaching 100% and others with rates as low as 10%. Predictions are driven by fine grained information cutting across multiple organ systems: infections, uncontrolled symptoms, wheelchair use, and more. Better estimates could transform advance care planning for patients with serious illnesses, who face many agonizing decisions that depend on duration of survival. We predict that prognostic algorithms will come into use in the next 5 years — although prospective validation will take several more years of data collection.

Second, machine learning will displace much of the work of radiologists and anatomical pathologists. These physicians focus largely on interpreting digitized images, which can easily be fed directly to algorithms instead. Massive imaging data sets, combined with recent advances in computer vision, will drive rapid improvements in performance, and machine accuracy will soon exceed that of humans. Indeed, radiology is already partway there: algorithms can replace a second radiologist reading mammograms5 and will soon exceed human accuracy. The patient-safety movement will increasingly advocate the use of algorithms over humans — after all, algorithms need no sleep, and their vigilance is the same at 2 a.m. as at 9 a.m. Algorithms will also monitor and interpret streaming physiological data, replacing aspects of anesthesiology and critical care. The time scale for these disruptions is years, not decades.

Third, machine learning will improve diagnostic accuracy. A recent Institute of Medicine report highlighted the alarming frequency of diagnostic errors and the lack of interventions to reduce them. Algorithms will soon generate differential diagnoses, suggest high-value tests, and reduce overuse of testing. This disruption will happen more slowly, over the next decade, for three reasons: first, the standard for diagnosis is unclear in many conditions (e.g., sepsis, rheumatoid arthritis) — unlike binary judgments in radiology or pathology (e.g., malignant or benign) — making it harder to train algorithms. Second, high-value EHR data are often stored in unstructured formats that are inaccessible to algorithms without layers of preprocessing. Finally, models need to be built and validated individually for each diagnosis.

Clinical medicine has always required doctors to handle enormous amounts of data, from macro-level physiology and behavior to laboratory and imaging studies and, increasingly, “omic” data. The ability to manage this complexity has always set good doctors apart from the rest. Machine learning will become an indispensable tool for clinicians seeking to truly understand their patients. As patients’ conditions and medical technologies become more complex, the role of machine learning will grow, and clinical medicine will be challenged to grow with it. As in other industries, this challenge will create winners and losers in medicine. But we are optimistic that patients, whose lives and medical histories shape the algorithms, will emerge as the biggest winners as machine learning transforms clinical medicine.


From the Department of Emergency Medicine, Harvard Medical School and Brigham and Women’s Hospital, and the Department of Health Care Policy, Harvard Medical School, Boston (Z.O.); and the Department of Medical Ethics and Health Policy, Perelman School of Medicine, and the Department of Health Care Management, Wharton School, University of Pennsylvania, Philadelphia (E.J.E.).

1. Mullainathan S, Spiess J. Machine learning: an applied econometric approach. J Econ Perspect (in press).
2. Halevy A, Norvig P, Pereira F. The unreasonable effectiveness of data. IEEE Intell Syst 2009; 24(2): 8-12.
3. Kleinberg J, Ludwig J, Mullainathan S, Obermeyer Z. Prediction policy problems. Am Econ Rev 2015; 105: 491-5.
4. Bouton CE, Shaikhouni A, Annetta NV, et al. Restoring cortical control of functional movement in a human with quadriplegia. Nature 2016; 533: 247-50.
5. Gilbert FJ, Astley SM, Gillan MGC, et al. Single reading with computer-aided detection for screening mammography. N Engl J Med 2008; 359: 1675-84.


This Perspective article originally appeared in The New England Journal of Medicine.

New call for submissions ­to NEJM Catalyst

Now inviting longform articles


A weekly email newsletter featuring the latest actionable ideas and practical innovations from NEJM Catalyst.

Learn More »

More From New Marketplace
Strongwater08_pullquote primary care value proposition and disruptive innovation

The Evolution of Primary Care: Embracing Innovation While Protecting the Core Value

Primary care must leverage disruptive innovations to ensure that patients receive first-access, comprehensive, coordinated, continuous care that is woven into a seamlessly integrated system.

Berns01_pullquote nephrologists dialysis facility joint venture conflicts of interest

Dialysis-Facility Joint-Venture Ownership — Hidden Conflicts of Interest

Despite potential benefits, joint ventures between nephrologists and dialysis companies raise legal and ethical concerns because of participants’ conflicts of interest and lack of transparency.

Fee-for-Service Continues to Account for the Majority of Revenue

New Marketplace Survey: Transitioning Payment Models: Fee-for-Service to Value-Based Care

In a survey of the NEJM Catalyst Insights Council sponsored by Optum, respondents express enthusiasm for value-based care but have conflicting opinions about just how far along that path they should go.

Sample Report Comparing Individual Patient-Reported Outcome Measures with FORCE-TJR National Norms

The Essential Role of Patient-Centered Registries in an Era of Electronic Health Records

Smartly designed patient-centered registries capture longitudinal data to augment EHRs and enhance quality improvement, policy, and research efforts.

Murray02_pullquote surgical care bundled payments accountable care organizations

Surgical Value — Beyond Bundled Payments

The surgeon has a crucial role in defining value for patients in a population — and not just when that patient is in need of the surgeon’s knife.

Comparison Between Traditional and Proposed Health Reimbursement Plan for Primary Care Outcomes Model Across Five Key Domains

Payment Designed for People: Introducing the Primary Care Outcomes Model

A care and payment model that engages primary care physicians in an aligned model built on trust and value could result in better patient care at lower costs.

Figure 1. Differential Changes in Total Medicare Spending for Patients in Accountable Care Organizations (ACOs), According to the Type of ACO, Year of Entry, and Number of Years of Participation.

Medicare Spending after 3 Years of the Medicare Shared Savings Program

After 3 years of the MSSP, participation in shared-savings contracts by physician groups was associated with savings for Medicare that grew over the study period, whereas hospital-integrated ACOs did not produce savings (on average) during the same period.

Reason and Potential Cost Changes for Patients with Treament Recommendation Changes During MD Anderson Cancer Center Multidisciplinary Team Meetings

Great Minds Don’t Always Think Alike

How multidisciplinary team meetings improve cancer care.

Mechanic02_pullquote self-insured employers

Self-Insured Employers — The Payment-Reform Wild Card

Without more private-sector leadership, U.S. health care will remain stuck in a fee-for-service system for the foreseeable future.

Washington State Health Care Authority Center of Excellence joint replacement elements and outcomes of program produces strong patient experience scores

Improving Care by Redesigning Payment

A state-run center of excellence uses benefit design to improve outcomes while controlling cost.


A weekly email newsletter featuring the latest actionable ideas and practical innovations from NEJM Catalyst.

Learn More »


Value Based Care

190 Articles

The Evolution of Primary Care: Embracing…

Primary care must leverage disruptive innovations to ensure that patients receive first-access, comprehensive, coordinated, continuous…

Dialysis-Facility Joint-Venture Ownership — Hidden Conflicts…

Despite potential benefits, joint ventures between nephrologists and dialysis companies raise legal and ethical concerns…

Dialysis-Facility Joint-Venture Ownership — Hidden Conflicts…

Despite potential benefits, joint ventures between nephrologists and dialysis companies raise legal and ethical concerns…

Insights Council

Have a voice. Join other health care leaders effecting change, shaping tomorrow.

Apply Now