Health care providers who participate as an accountable care organization (ACO) in the voluntary Medicare Shared Savings Program (MSSP) have incentives to lower spending for Medicare patients while achieving high performance on a set of quality measures. Little is known about the extent to which early savings achieved by ACOs in the program have grown and been replicated by ACOs that entered the program in later years. ACOs that are physician groups have stronger incentives to lower spending than hospital-integrated ACOs.
Using fee-for-service Medicare claims from 2009 through 2015, we performed difference-in-differences analyses to compare changes in Medicare spending for patients in ACOs before and after entry into the MSSP with concurrent changes in spending for local patients served by providers not participating in the MSSP (control group). We estimated differential changes (i.e., the between-group difference in the change from the pre-entry period) separately for hospital-integrated ACOs and physician-group ACOs that entered the MSSP in 2012, 2013, or 2014.
MSSP participation was associated with differential spending reductions in physician-group ACOs. These reductions grew with longer participation in the program and were significantly greater than the reductions in hospital-integrated ACOs. By 2015, the mean differential change in per-patient Medicare spending was −$474 (−4.9% of the pre-entry mean, P<0.001) for physician-group ACOs that entered in 2012, −$342 (−3.5% of the pre-entry mean, P<0.001) for those that entered in 2013, and −$156 (−1.6% of the pre-entry mean, P=0.009) for those that entered in 2014. The corresponding differential changes for hospital-integrated ACOs were −$169 (P=0.005), −$18 (P=0.78), and $88 (P=0.14), which were significantly lower than for physician-group ACOs (P<0.001). Spending reductions in physician-group ACOs constituted a net savings to Medicare of $256.4 million in 2015, whereas spending reductions in hospital-integrated ACOs were offset by bonus payments.
After 3 years of the MSSP, participation in shared-savings contracts by physician groups was associated with savings for Medicare that grew over the study period, whereas hospital-integrated ACOs did not produce savings (on average) during the same period. (Funded by the National Institute on Aging.)
Since the first accountable care organizations (ACOs) entered the Medicare Shared Savings Program (MSSP) in 2012, it has expanded annually to include 561 ACOs covering approximately one third of the fee-for-service Medicare population.1 ACOs in the voluntary MSSP have incentives to lower fee-for-service spending for Medicare patients below a financial benchmark (spending target) while achieving high performance on a set of quality measures. Through 2015, 99.2% of ACOs had opted for so-called “one-sided” contracts that reward ACOs with shared-savings bonuses if spending is sufficiently below benchmarks but do not impose a risk of financial losses for spending above benchmarks.2
Estimates from the Centers for Medicare and Medicaid Services (CMS) suggest that reductions in spending by ACOs have been minimal and more than offset by shared-savings bonuses paid to ACOs. In 2015, per-beneficiary Medicare spending in ACOs was only $59 lower than ACO benchmarks, on average, which resulted in an apparent $216 million loss to Medicare after accounting for bonuses.3 However, an assessment of the effect of the MSSP on spending requires counterfactual estimates of spending for ACO patients (i.e., spending that would have occurred in the absence of the program). CMS benchmarks are not valid counterfactual estimates, and the method for their calculation has systematically underestimated savings.4 For example, until 2017, benchmarks were updated annually on the basis of concurrent growth in national Medicare spending, so any downward effects of ACOs on spending growth also lowered benchmarks. In addition, growth rates that were used to set benchmarks were based on the entire fee-for-service population, which included low-cost beneficiaries who received too little care to be attributed to an ACO.5
Although MSSP shared-savings contracts are standardized, the strength of incentives differs for ACOs with different organizational structures. ACOs that are large health systems or parts of organizations spanning many settings, specialties, and services have weaker incentives to lower spending than organizations that provide a narrower scope of services. If an ACO reduces its provision of services, the resulting shared-savings bonus is at least partially offset by the forgone fee-for-service profits. If all other factors are equal, it is more profitable for an ACO to limit the provision of services delivered by other providers. In addition, since large, integrated ACOs provide a range of care for patients who are not covered by their ACO contracts, they can lose substantial fee-for-service profits if they are unable to restrict reductions in utilization to ACO-covered patients.6,7 Thus, physician groups have stronger incentives as ACOs to limit utilization and pursue systemic strategies that affect all their patients, without having to rely on efforts that target specific patients.8 For example, physician groups do not lose revenue when they reduce unnecessary hospitalizations or outpatient hospital procedures and imaging for any patient, regardless of whether that patient is covered by an ACO contract.
Consistent with these expectations, early evaluations of the MSSP that used plausible counterfactual estimates suggested greater spending reductions than calculations based on benchmarks and greater reductions in physician groups than in larger ACOs that are financially integrated with hospitals,9-12 but the early reductions were small. In this study, we examined how program savings have evolved over multiple years for multiple entry cohorts of ACOs and whether differences in savings between hospital-integrated and physician-group ACOs have grown.
Study Design and Overview
We used Medicare claims and enrollment data from 2009 through 2015 and a difference-in-differences approach to compare fee-for-service Medicare beneficiaries served by ACOs in the MSSP with local groups of beneficiaries served by nonparticipating providers (control group) before and after the start of ACO contracts. This approach estimated the effects of the MSSP on utilization and spending by comparing changes for ACO patients from the period before entry to the period after entry with concurrent changes for the control group. We focused on cohorts of patients entering the MSSP in 2012, 2013, and 2014 to examine at least 2 years after entry. For the 2012 entry cohort, we prespecified 2012 to be a transition year because ACOs in that cohort entered the program in April or July instead of January.
Using website data for each ACO, we assessed whether the core medical groups that determined the organization’s attributed population were part of a larger organization or health system that also included hospitals, which suggested an ownership structure that spanned multiple settings and many services. We conducted prespecified subgroup comparisons between these hospital-integrated ACOs and the other, independent physician-group ACOs in the program. Using our estimates of changes in spending associated with MSSP participation and CMS data on shared-savings bonus payments,3 we calculated the net savings to Medicare for each group of ACOs.
For each year from 2009 through 2015, our study population included a random 20% sample of Medicare beneficiaries who were continuously enrolled in the fee-for-service program during the year (while alive in the case of decedents) and in the preceding year (to assess preexisting clinical conditions). Using previously described methods11,12 as well as CMS definitions that describe each ACO as a set of taxpayer identification numbers that were used by providers for billing (or CMS certification numbers for safety-net providers), we attributed each beneficiary in each year to the ACO or non-ACO taxpayer identification number (or CMS certification number) that accounted for the plurality of the beneficiary’s office visits with a primary care physician. Beneficiaries who were attributed to taxpayer identification numbers of nonparticipating providers constituted the control group. In each year, we excluded beneficiaries who did not have at least one office visit with a primary care physician. (Details regarding the study methods are provided in the Supplementary Appendix, available with the full text of this article at NEJM.org.)
We used ACO definitions from the first year of participation, holding them constant over the study period to minimize bias from changes in ACO composition in the post-entry period that may have favored providers with higher or lower spending. We conducted sensitivity analyses to address changes over the study period in the practices or physicians represented by the taxpayer identification numbers in each ACO.
We similarly attributed beneficiaries to organizations participating in the Pioneer ACO model implemented by the Center for Medicare and Medicaid Innovation in 2012. We excluded these beneficiaries from all study years to minimize the effects of Pioneer ACOs on the control group. Because of the substantial dropout of ACOs from the Pioneer model by 2015, we did not examine how previously described effects of Pioneer ACOs in their first 2 years may have evolved.13,14
Spending and Utilization
Our primary outcome was total annual Medicare spending for services covered by Part A and Part B. Anticipating potential increases in ACO savings over time,10 we prespecified 2015 as the primary year of interest. In secondary analyses, we examined spending according to the setting and type of service, along with measures of health care use, including hospitalizations, hospitalizations for ambulatory care–sensitive conditions (i.e., those for which appropriate ambulatory care could potentially reduce the need for inpatient care),15 30-day readmissions for any cause, emergency department visits not followed by admission, post-acute facility stays, days in post-acute facilities, and primary care visits.
Using the Medicare Master Beneficiary Summary File, the CMS Chronic Conditions Data Warehouse, diagnoses that were recorded in claims, and U.S. Census data,16 we assessed the following fixed and time-varying characteristics of beneficiaries: age, sex, race or ethnic group, whether disability was the original reason for Medicare eligibility, Medicaid enrollment, long-term residence in a nursing home,17 history of chronic conditions, Hierarchical Condition Category (HCC) risk score, and area-level sociodemographic characteristics. In our main analysis, for each beneficiary in each study year, we assessed the presence of chronic conditions on the basis of cumulative records in the Chronic Conditions Data Warehouse through the end of the preceding year, and we used claims from the preceding year to calculate HCC scores. ACOs in the MSSP had modest incentives to record more diagnoses in claims. According to the MSSP rules, decreases in HCC scores for attributed beneficiaries caused risk-adjusted benchmarks to fall, but increases in scores did not cause benchmarks to rise.18 In sensitivity analyses, we shifted assessments of chronic conditions and HCC scores back 2 years to eliminate potential effects of changes in coding practices. Thus, for study year 2015, these earlier assessments were based on diagnoses that occurred no later than December 31, 2012, before the first full year of MSSP operation.
For each outcome, we fit the following linear regression model for the years 2009 through 2015:
E(Yi,t,k,h) = β0 + β1ACOk + β2HRRh×Yeart + β3ACO_Cohortk×Postt + β4Covariatesi,t
where Y is the annual spending or utilization measure for beneficiary i in year t attributed to ACO k or a taxpayer identification number in the control group and residing in hospital referral region (HRR) h; “ACO” is a vector of indicators for each ACO with the control group as the reference group; “HRR×Year” is a vector of indicators for each hospital referral region–year combination omitting a reference combination; “ACO_Cohort×Post” is a vector of indicators for each entry cohort of ACOs (2012, 2013, or 2014 cohort) in each post-entry year; “Covariates” is a vector of the beneficiary characteristics listed in Table 1; and β1through β4 are the corresponding vectors of coefficients for each term. To obtain separate estimates for each ACO type, we re-estimated this model after alternately excluding beneficiaries attributed to hospital-integrated or physician-group ACOs. We used a robust variance estimator, specifying clusters as ACOs (for ACO-attributed beneficiaries) or hospital referral regions (for the control group).19
The inclusion of the ACO indicators adjusted for pre-entry differences between ACOs and the control group and for any changes in the distribution of ACO-attributed beneficiaries across ACOs. The inclusion of the hospital referral region–year indicators adjusted for geographic differences between ACOs and the control group and for regional changes in spending for the control group. Thus, the estimated effect of MSSP participation (β3) is the difference between spending for ACO-attributed patients in a post-entry year and spending that would be expected for ACO patients if the change from the pre-entry period to that year was equal to the change observed for patients in the same hospital referral region served by non-ACO providers. We report estimated effects as differential changes (i.e., the between-group difference in the change from the pre-entry period) in spending for patients in ACOs.
We conducted additional analyses to explore alternative explanations for our findings. First, we estimated differences in pre-entry trends between each ACO cohort and the control group and conducted falsification tests that treated pre-entry years as hypothetical post-entry years. Second, we tested for differential changes in the characteristics of patients served by ACOs from the pre-entry period to 2015, and we compared estimates of differential changes in spending with and without adjustment for patient characteristics. (Details regarding these sensitivity analyses are provided in the Supplementary Appendix.)
Third, we conducted falsification tests to gauge the contribution of documented hospital–physician consolidation to savings estimates for physician-group ACOs. We know that these organizations remained financially independent from hospitals in the post-entry period on the basis of online information about organizational structure that we could assess only after MSSP entry. Thus, we expected more hospital–physician consolidation in the control group, which would cause greater spending increases in hospital-owned outpatient settings in the control group than in physician–group ACOs (and lesser spending increases in the independent office setting). To estimate the empirical distribution of expected results for physician groups that remained independent but were not exposed to MSSP incentives, we took random draws from 378 taxpayer identification numbers of nonparticipating physician groups that were large enough to participate in the MSSP and billed Medicare in 2015 as independent practices (not hospital-owned). Treating each draw as hypothetical ACOs and the remaining nonparticipating providers (other than the 378 taxpayer identification numbers) as the control group, we then repeated our main analysis. We also conducted falsification tests treating ACOs in the 2015 entry cohort as hypothetical entrants in earlier years.
Fourth, we conducted a resampling analysis to assess the importance of organizational structure as a predictor of savings relative to other potential predictors20 or random variation. Finally, in a sensitivity analysis we used the Hochberg procedure21 to account for multiple testing in analyses of our primary outcome; P values that were reported for the main results were not adjusted for multiple testing.
The 20% study sample of Medicare beneficiaries included 29,616,964 beneficiary-years from 2009 through 2015 (Table 1), with an average of 19.6% of beneficiaries attributed to an ACO in the cohorts that entered the program from 2012 through 2014. In 2015, an average of 4021 beneficiaries were attributed to each hospital-integrated ACO and 1843 to each physician-group ACO (or 20,105 and 9215 patients expected in the full Medicare population, respectively). From the pre-entry period through 2015, differential changes in the characteristics of beneficiaries in the ACOs, as compared with the control group, were consistently small (Table S1 in the Supplementary Appendix). One exception was a differential increase in the mean HCC risk score of beneficiaries in physician groups in the 2013 entry cohort of ACOs (differential change adjusted for hospital referral region, 0.033, or 2.7% of the pre-entry mean). In an analysis of claims submitted 3 years before each study year, the differential change in HCC scores was smaller, indicating that either the health of beneficiaries in these ACOs differentially declined in the post-entry period or coding practices differentially changed.
Levels and Trends before MSSP Entry
During the pre-entry period, adjusted total annual Medicare spending per beneficiary and spending trends did not differ significantly between the ACOs and the control group (Tables S2 and S3 in the Supplementary Appendix). Spending trends before entry also did not differ significantly for most categories of spending and utilization measures in overall comparisons and comparisons stratified according to the organizational structure of ACOs. Exceptions included significant differences in annual trends in spending on outpatient care in hospital-owned facilities for patients in physician-group ACOs in the 2013 cohort ($13 per beneficiary lower than in the control group, P=0.04) and in the 2014 cohort ($19 per beneficiary lower, P<0.001) but not in the 2012 cohort ($9 per beneficiary lower, P=0.26). Trends in total outpatient spending before entry did not differ significantly between the ACOs and the control group (Table S3 in the Supplementary Appendix).
Differential Changes in Total Spending in 2015
Without accounting for bonus payments, we determined that by 2015, participation in the MSSP, as compared with nonparticipation, was associated with a mean differential reduction of $302 in total Medicare spending per beneficiary in the 2012 entry cohort of ACOs (or 3.1% of the pre-entry mean, P<0.001), a $139 differential reduction in the 2013 entry cohort (or 1.4%, P=0.009), and a $36 differential reduction in the 2014 entry cohort (or 0.4%, P=0.48). For physician-group ACOs, the differential reductions in 2015 spending in the 2012 cohort ($474, P<0.001), 2013 cohort ($342, P<0.001), and 2014 cohort ($156, P=0.009) were significantly larger than changes in the hospital-integrated ACOs in the 2012 cohort (−$169, P=0.005), 2013 cohort (−$18, P=0.78), and 2014 cohort ($88, P=0.14), respectively (P<0.001 for global test of within-cohort differences between physician-group and hospital-integrated ACOs). Spending reductions that were associated with MSSP participation grew consistently with more years of participation among physician-group ACOs but not among hospital-integrated ACOs (Figure 1, and Table S4 in the Supplementary Appendix).
Differential Changes in Categories of Spending and Utilization in 2015
In an analysis that pooled entry cohorts of physician-group ACOs, differential reductions in spending on acute inpatient care, post-acute care, home health care initiated on an outpatient basis, and outpatient care in hospital-owned settings contributed to differential reductions in total spending, with partially offsetting increases in spending on office-based outpatient care (Table 2; cohort-specific estimates are provided in Table S4 in the Supplementary Appendix). In addition, the number of visits to emergency departments differentially declined, whereas there were no significant differential changes in the number of primary care visits, the number of hospitalizations for ambulatory care–sensitive conditions, or 30-day readmission rates.
Among hospital-integrated ACOs, there were no significant differential changes in any spending category, and there were small differential increases in the number of readmissions and decreases in the number of emergency department visits on average across entry cohorts (Table 2). Cohort-specific analyses revealed no consistent pattern of differential changes across entry cohorts (Table S4 in the Supplementary Appendix).
Net Savings to Medicare in 2015
Accounting for shared-savings bonus payments, we determined that the differential spending reductions in the entry cohorts of physician-group ACOs from 2012 through 2014 constituted a net savings to Medicare of $256.4 million in 2015. For hospital-integrated ACOs, bonus payments more than offset aggregate spending reductions (Table 3).
The results of falsification tests and sensitivity analyses supported the main study findings (Tables S5, S6, and S7 and Figs. S1 through S5 in the Supplementary Appendix) but qualified the interpretation of two findings. First, adjustments for HCC scores and chronic conditions derived from diagnoses that had been recorded earlier resulted in a smaller differential reduction in spending in the 2013 cohort of physician-group ACOs ($188 per beneficiary, P=0.03). Second, falsification tests suggested that a differential spending reduction in hospital-owned outpatient settings and a differential increase in the independent office setting would be expected for physician groups in the absence of MSSP participation (because hospital–physician consolidation was held at low levels among physician-group ACOs by definition) but would not have a significant net effect on total outpatient spending or total spending (Tables S5 and S6 in the Supplementary Appendix).
In this study, we found that participation in the MSSP by physician groups was associated with savings for Medicare that grew over the study period, whereas hospital-integrated ACOs did not produce savings (on average) during the same period. Among participating physician groups, spending reductions grew during a 3-year period among organizations that entered the program in 2012 and 2013, and a significant reduction emerged during the second year for 2014 entrants.10 Our estimate of the aggregate reduction in fee-for-service spending accrued by these ACOs in 2015 ($583.4 million) was 39% greater than the corresponding reduction reported by CMS ($419.3 million), which was calculated with the use of ACO benchmarks as counterfactual estimates. Our estimate of net savings to Medicare from these ACOs is nearly 2.8 times as great ($256.4 vs. $92.3 million).
MSSP participation by physician groups was associated with differential reductions in spending on sources of fee-for-service revenue for other providers — including inpatient and outpatient hospital care, post-acute care, and outpatient home health care — but not with differential reductions in spending on office-based outpatient care. Our findings are consistent with the stronger incentives for physician groups in the MSSP to lower such spending and with other evidence challenging the notion that large-scale consolidation of providers supports success under new payment models.22-27
Our results also suggest that shared-savings contracts that do not impose a downside risk of financial losses for spending above benchmarks — which may appeal to smaller organizations without sufficient reserves to withstand potential losses — may be effective in lowering Medicare spending. Thus, current policy that was established by the Medicare Affordability and CHIP (Children’s Health Insurance Program) Reauthorization Act of 2015 to encourage participation in alternative payment models may be too restrictive, since the policy limits bonus payments to providers participating in models that impose downside risk. Moreover, our findings suggest distinctive benefits from MSSP participation that could erode if policy changes, such as requiring ACOs to assume downside risk after fewer years of participation, cause ACOs to leave the program.
Our study has several limitations. First, differences between voluntary MSSP participants and nonparticipating providers may have led to differential spending changes in the absence of the MSSP. However, spending trends in the period before program entry did not meaningfully differ between ACOs and the control group, and sensitivity analyses did not support alternative explanations. We would not expect that organizations knew before making participation decisions how their future spending changes would differ from other changes in their region or that they would lower spending without incentives to do so. Differences in exposure to hospital–physician consolidation between the control group and physician-group ACOs probably explained some of the apparent shift in outpatient care from hospital-owned to independent-office settings associated with MSSP participation. However, the results of falsification tests suggested that differences in exposure to consolidation would not have led to differential reductions in total spending among physician-group ACOs.
Second, some of the savings that were estimated in the 2013 entry cohort of ACOs may have been due to changes in coding practices. However, MSSP incentives to record more diagnoses were modest, and it is unclear why such incentives would elicit a response in only one of the three entry cohorts. Alternatively, we may have underestimated savings in the 2013 cohort if the ACOs served more patients whose health declined in the post-entry period — for example, because of the engagement of such patients in care management programs.
Third, we focused on one measurable aspect of organizational structure that has important implications for ACO incentives, but many other factors may have contributed to differences in savings.20 In addition, our estimates of average effects may have obscured savings achieved by individual ACOs. Nevertheless, a resampling analysis suggested that financial independence from hospitals was a prominent predictor of savings.
Fourth, we lacked data on the costs to ACOs of efforts to lower spending or improve quality and thus could not assess savings from an ACO or a societal perspective. Fifth, we were unable to assess the effects of the MSSP on many aspects of quality of care because claims-based measures are limited and challenging to interpret.8 Finally, our results probably underestimate savings to Medicare because they do not account for spillover effects of ACO efforts on nonattributed patients or the effects of lower fee-for-service Medicare spending on payments to Medicare Advantage plans.28
In conclusion, after 3 years of the MSSP, participation in shared-savings contracts by physician groups was associated with savings for Medicare that grew over the study period, whereas hospital-integrated ACOs did not produce savings (on average) during the same period. Evidence from quasi-experimental evaluations may serve as a better basis for guiding Medicare payment policy than comparisons with targets that are used to set incentives.
From the Department of Health Care Policy, Harvard Medical School (J.M.M., L.A.H., B.E.L., P.H., M.E.C.), the Division of General Internal Medicine and Primary Care, Department of Medicine, Brigham and Women’s Hospital and Harvard Medical School (J.M.M.), and the Division of General Internal Medicine and Primary Care, Department of Medicine, Beth Israel Deaconess Medical Center (B.E.L.) — all in Boston.
Click here for the full list of references.
This Special Article originally appeared in The New England Journal of Medicine.