In the Medicare Shared Savings Program (MSSP), accountable care organizations (ACOs) have financial incentives to lower spending and improve quality. We used quasi-experimental methods to assess the early performance of MSSP ACOs.
Using Medicare claims from 2009 through 2013 and a difference-in-differences design, we compared changes in spending and in performance on quality measures from before the start of ACO contracts to after the start of the contracts between beneficiaries served by the 220 ACOs entering the MSSP in mid-2012 (2012 ACO cohort) or January 2013 (2013 ACO cohort) and those served by non-ACO providers (control group), with adjustment for geographic area and beneficiary characteristics. We analyzed the 2012 and 2013 ACO cohorts separately because entry time could reflect the capacity of an ACO to achieve savings. We compared ACO savings according to organizational structure, baseline spending, and concurrent ACO contracting with commercial insurers.
Adjusted Medicare spending and spending trends were similar in the ACO cohorts and the control group during the precontract period. In 2013, the differential change (i.e., the between-group difference in the change from the precontract period) in total adjusted annual spending was −$144 per beneficiary in the 2012 ACO cohort as compared with the control group (P=0.02), consistent with a 1.4% savings, but only −$3 per beneficiary in the 2013 ACO cohort as compared with the control group (P=0.96). Estimated savings were consistently greater in independent primary care groups than in hospital-integrated groups among 2012 and 2013 MSSP entrants (P=0.005 for interaction). MSSP contracts were associated with improved performance on some quality measures and unchanged performance on others.
The first full year of MSSP contracts was associated with early reductions in Medicare spending among 2012 entrants but not among 2013 entrants. Savings were greater in independent primary care groups than in hospital-integrated groups.
In the Medicare Shared Savings Program (MSSP) — the largest of the Medicare accountable care organization (ACO) programs — participating provider organizations share in savings with Medicare if they keep spending for an attributed population of fee-for-service beneficiaries sufficiently below a financial benchmark. Greater shared-savings bonuses are awarded to ACOs with higher performance on a set of quality measures. Unlike ACOs in the Medicare Pioneer program, very few ACOs in the MSSP face penalties for spending in excess of benchmarks because such downside risk is not currently required.1
The financial benchmark for an ACO is set by the Centers for Medicare and Medicaid Services (CMS) at the average level of Medicare spending for patients served by the ACO during a baseline period before the start of the contract and is updated for each contract year on the basis of national Medicare spending growth since the baseline period. On the basis of comparisons of ACO spending with these benchmarks, CMS has reported savings on average across all ACOs in the MSSP (not including shared-savings bonuses paid) and greater savings achieved by ACOs entering the program in 2012 than by those entering in 2013.2,3
Savings based on these actuarial calculations, however, may differ from actual spending reductions. For example, the substantial geographic variation in Medicare spending growth calls into question the validity of savings estimated by comparing spending in an ACO with a benchmark derived from a national rate of spending growth.4,5 If an ACO is located in an area with high spending growth, its savings could be underestimated. Providers participating in ACO programs and different types of ACOs are not randomly distributed across geographic areas.6-8 Therefore, the extent to which ACOs in the MSSP have lowered spending overall remains unclear, as do differences in savings achieved by ACOs with different traits, times of program entry, or incentives.
Using Medicare claims and CMS definitions of the 220 ACOs entering the MSSP in 2012 or 2013, we compared changes in spending and performance on claims-based quality measures for beneficiaries served by ACOs from before to after the start of ACO contracts with concurrent changes for beneficiaries served by non-ACO providers in ACO service areas.
Study Data and Population
We analyzed Medicare claims and enrollment data from 2008 through 2013 for a random 20% sample of fee-for-service beneficiaries. For each study year from 2009 through 2013, we included beneficiaries who were continuously enrolled in Parts A and B of fee-for-service Medicare in that year (during the time they were alive, in the case of beneficiaries who subsequently died) and in the previous year (to assess preexisting conditions). Using the CMS ACO Provider-level Research Identifiable File, which defines each ACO as a collection of provider taxpayer identification numbers and CMS Certification Numbers (for safety-net providers),9 we attributed each beneficiary in each study year to the ACO or non-ACO taxpayer identification number accounting for the most allowed charges for qualifying outpatient evaluation and management services delivered to the beneficiary by a primary care physician during the year (see the Supplementary Appendix, available with the full text of this article at NEJM.org). Beneficiaries attributed to non-ACO providers constituted the control group. We excluded beneficiaries with no qualifying services.
We followed the MSSP rules for attributing beneficiaries10 but limited qualifying services to those provided by primary care physicians, because many ACOs include no or few specialty practices. This restriction improved balance in precontract spending levels between the ACO group and the control group. In a sensitivity analysis, we modified ACO definitions to account for major changes in the taxpayer identification numbers used by participating providers. (For more details on attribution of beneficiaries and modification of ACO definitions, see the Supplementary Appendix.)
Spending and Quality Measures
The primary outcome of our study was total annual Parts A and B Medicare spending per beneficiary, as defined by ACO program specifications.11 In secondary analyses, we analyzed spending according to the type of service and care setting.
As in a previous study,5 we assessed several claims-based measures of quality of care included in ACO contracts12: hospitalizations for two ambulatory care–sensitive conditions (ACSCs; conditions for which appropriate ambulatory care could potentially reduce the need for inpatient care) — congestive heart failure and chronic obstructive pulmonary disease or asthma13,14; all-cause 30-day readmissions among hospitalized beneficiaries; and screening mammography. We also assessed hospitalizations for ACSCs related to cardiovascular disease and diabetes14 and three preventive services for diabetes, because many contract measures that cannot be assessed with claims focus on these conditions. We specified hospitalizations and readmissions as annual counts and assessed the receipt of preventive services on an annual basis. Finally, we used previously developed methods to assess the use of 31 low-value services (Table S1 in the Supplementary Appendix), which were defined as services that provide minimal average benefit in clinical settings that can be identified from claims with reasonable specificity.15,16 We analyzed the annual count of low-value services received by each beneficiary.
We prespecified several subgroups of ACOs for comparison. First, we categorized ACOs as financially integrated with hospitals (vertically integrated organizations), independent multispecialty physician groups, or independent primary care physician groups, using CMS descriptions, information on the websites of participating organizations, and physician specialty information from claims. Second, we categorized each ACO according to whether risk-adjusted Medicare spending at baseline for the control group in its service area was above or below the median among ACO service areas. Third, we categorized each ACO according to whether risk-adjusted baseline spending for its attributed beneficiaries was above or below risk-adjusted spending for the control group in its service area. To mitigate bias from regression to the mean, we used claims from 2008 (before the study period) to assess baseline spending for these measures. (For more details on the categorization of ACOs, see the Supplementary Appendix.)
Fourth, we used the Decision Resources Group proprietary ACO database17 to identify MSSP ACOs with at least one ACO risk contract with a commercial insurer starting in 2012 or 2013. We considered payment arrangements involving global budgets with one- or two-sided risk sharing, global payments, or capitation to be ACO contracts. When comparing ACOs that entered commercial ACO contracts in 2012 or 2013 with other ACOs, we excluded 12 ACOs with commercial contracts that started before 2012 because these may have had spillover effects that caused lower Medicare spending before the start of Medicare ACO contracts.18
From Medicare Master Beneficiary Summary Files, we assessed the age, sex, race or ethnic group,19,20 and Medicaid coverage of the beneficiaries, as well as whether disability was the original reason for Medicare eligibility and whether beneficiaries had end-stage renal disease. From the Chronic Conditions Data Warehouse (CCW), which draws from diagnoses since 1999 to describe beneficiaries’ accumulated disease burden,21 we assessed whether beneficiaries had any of 27 conditions in the CCW by the start of each study year. From diagnoses in the preceding year of claims, we also calculated a Hierarchical Condition Category risk score for each beneficiary in each study year.22 We determined whether beneficiaries were long-term nursing home residents using a validated claims-based algorithm.23 Finally, from U.S. Census data,24 we assessed area-level sociodemographic characteristics.
We used linear regression and a difference-indifferences approach to estimate changes in spending or quality in the ACO group from the precontract to postcontract period that differed from concurrent changes in local control groups (see the Supplementary Appendix). Our models included the characteristics listed in Table 1 (except with age specified categorically and CCW conditions as indicators and counts) and fixed effects for each ACO to adjust for precontract differences between ACOs and the control group and for any changes in the distribution of ACO-attributed beneficiaries across ACOs. Models also included fixed effects for each hospital referral region (HRR) in each year to compare each beneficiary attributed to an ACO with beneficiaries in the control group living in the same area and to adjust for HRR-specific changes in spending or quality occurring in the control group. Thus, unlike comparisons with financial benchmarks set by CMS, our analysis estimated savings or losses by comparing spending in the postcontract period for ACO-attributed patients with spending that would be expected in the absence of ACO contracts, with the use of local changes in a similar population to establish that counterfactual scenario. We compared our estimates of savings with CMS estimates, using publicly available data on ACO spending and benchmarks to describe the latter (see the Supplementary Appendix).3
We used robust variance estimators to account for clustering within ACOs (for the ACO group) or HRRs (for the control group).25 In sensitivity analyses, we analyzed count variables and spending using generalized linear models with a log link and proportional-to-mean variance function, and we analyzed binary quality measures using logistic regression.
As prespecified in our study protocol, we treated 2013 as the postcontract period for both the 2012 and 2013 cohorts of MSSP ACOs because the 2012 cohort did not enter the MSSP until April 2012 (24% of ACOs) or July 2012 (76%). Our models treated 2009 through 2011 as the precontract period for 2012 MSSP entrants (excluding the 2012 transition year) and 2009 through 2012 as the precontract period for 2013 MSSP entrants. Per the study protocol, we estimated differential changes in total spending and quality separately for each cohort of MSSP ACOs and analyzed savings according to type of service (for descriptive purposes) only if total savings in a cohort were significant (two-sided P<0.05).
We analyzed the 2012 and 2013 cohorts separately because early participants may have differed from later participants in their responsiveness to new incentives and because CMS reported savings differ between the cohorts.2,3 To compare savings between ACO subgroups of interest, we added interaction terms to models to determine whether savings differed according to ACO characteristics. To provide context for interpreting MSSP results, we also updated analyses of the 32 ACOs that entered the Pioneer model in 2012. (For more details on multiple testing in subgroup comparisons and on analyses of the Pioneer ACO model, see the Supplementary Appendix.)5
We conducted several analyses to gauge potential sources of bias. First, we compared trends in total spending between the ACO group and the control group during the precontract period. A key assumption of our difference-in-differences comparisons is that changes from the precontract to postcontract period would have been similar in the ACO group and the control group in the absence of the ACO programs. Similar trends in the precontract period would support this assumption. Precontract trends were similar for all comparisons except for one subgroup comparison, which we note and consider to be uninterpretable.
Second, we tested whether patients’ sociodemographic and clinical characteristics differentially changed from the precontract to postcontract period in the ACO group relative to the control group. Third, we used propensity-score methods to balance observed patient characteristics between the ACO group and the control group within each HRR and study year (see the Supplementary Appendix).26 Fourth, we conducted falsification tests treating 2010 or 2011 as a postcontract year.
After adjustment for HRR, differences in beneficiaries’ clinical and sociodemographic characteristics between the ACO group and the control group were mostly small, and differential changes from the precontract period to the postcontract period were consistently minimal (Table 1 for the 2012 MSSP cohort, and Table S2 in the Supplementary Appendix for the 2013 cohort).
Changes in Spending and Quality
In the precontract period, adjusted total annual Medicare spending per beneficiary was similar in the ACO group and the control group ($133 per beneficiary higher in the 2012 ACO cohort [2012 MSSP entrants] than in the control group, P=0.22; $2 per beneficiary higher in the 2013 ACO cohort [2013 MSSP entrants] than in the control group; P=0.97). Spending trends in the precontract period were also similar ($19 per beneficiary slower per year in the 2012 ACO cohort than in the control group, P=0.50; $11 per beneficiary slower per year in the 2013 ACO cohort than in the control group, P=0.54).
The adjusted differential change (i.e., the adjusted between-group difference in the change from the precontract period) in mean total annual Medicare spending in 2013 was −$144 per beneficiary in the 2012 cohort of MSSP ACOs as compared with the control group (P=0.02), representing a 1.4% estimated savings relative to an expected mean of $9,942 in 2013 for the ACO group. In contrast, the differential change in total spending in 2013 was only −$3 per beneficiary among ACOs entering the MSSP in 2013 as compared with the control group (P=0.96). Contributing to the estimated savings in the 2012 cohort were a 1.4% reduction in spending on inpatient care, a 2.1% reduction in spending on hospital outpatient care that was mostly offset by a 1.5% increase in spending on office-based outpatient care, a 6.1% reduction in spending on postacute care at skilled nursing facilities, and a 2.7% reduction in spending on home health care (Table 2).
Performance on quality measures differentially improved in the ACO cohorts as compared with the control group for a few measures and did not differentially change for other measures, and there was no differential change in the use of low-value services in either ACO cohort (Table 2, and Table S3 in the Supplementary Appendix). Use of generalized linear models did not alter our conclusions, falsification tests revealed no significant differential changes in spending in 2010 or 2011, and results were robust with respect to other sensitivity analyses (Table S4 in the Supplementary Appendix).
ACO Subgroup Comparisons
Estimated savings among independent primary care groups were consistently seen in the 2012 and 2013 cohorts of ACOs (Fig. 1) and were significantly greater than savings among vertically integrated organizations (difference between ACO subgroups pooled across cohorts, −$371 per beneficiary; P=0.005). Post hoc analyses suggested that this difference in total savings was driven largely by greater differential reductions in spending on inpatient care and hospital outpatient care among independent primary care groups. In contrast, CMS estimates of savings did not differ substantially according to ACO type, apparently understating subgroup differences as a result of the actuarial methods used. (For more details, see the Supplementary Appendix, including Tables S5 and S6.)
Estimated savings were significantly greater for ACOs in the 2012 MSSP cohort that had baseline spending above local averages than for those that had baseline spending below local averages (difference, −$285 per beneficiary; P=0.02). Savings in the 2012 cohort followed a similar pattern for ACOs in high-spending versus low-spending areas (Fig. 1), but the difference was not significant (P=0.09) and was uninterpretable owing to differences in precontract trends. In the 2013 cohort, savings were not evident in any subgroup defined by baseline spending.
Commercial ACO Contracting
In the 2012 MSSP cohort, estimated savings in 2013 did not differ significantly between ACOs concurrently entering commercial ACO contracts and ACOs without such contracts (Fig. 1). In the 2013 cohort, differential changes in spending for the 11 ACOs that entered commercial ACO contracts in 2012 or 2013 indicated significant savings in 2013, differing significantly from differential changes in spending among other ACOs in the cohort (subgroup difference, −$611; P=0.001).
The first full year of participation in the MSSP was associated with early savings among ACOs that entered the program in 2012 but not among those that entered in 2013. Savings in the 2012 cohort were on a par with savings estimated for Pioneer ACOs (see the Supplementary Appendix), suggesting that one-sided contracts without downside risk (shared savings only) also may elicit effective efforts to reduce health care utilization. Owing to the one-sided nature of almost all MSSP contracts, however, the aggregate $238 million spending reduction suggested by our estimates for the 2012 MSSP cohort did not result in net savings to Medicare, because Medicare paid $244 million in bonuses without recouping losses from ACOs that had spending above benchmarks.3
Moreover, our results suggest that early gains observed among early MSSP participants may not generalize to later participants. We examined only 12 to 18 months of exposure to MSSP incentives for most ACOs in our study, and savings may grow over time.27 Nevertheless, if meaningful savings from the expanding MSSP materialize, our findings suggest that they may be slow to develop, because participants with less advanced systems for managing care may enter later and require more time to improve care efficiency. Regulatory changes that strengthen incentives for ACOs to lower spending — incentives that have been very weak thus far28-30 — may accelerate savings and will be important to implement and evaluate.31
Although our estimate of aggregate savings in the MSSP is similar to the $234 million in savings determined by CMS from comparisons with benchmarks, we found a different pattern of savings according to ACO structure (see the Supplementary Appendix). Whereas CMS data suggest that savings were generally similar for different types of ACOs, we estimated substantially greater savings for independent primary care groups than for groups integrated with hospitals when comparing spending changes in ACOs with local concurrent changes. There are both theoretical considerations and previous observational studies that would support the pattern of savings that we observed.32-35 In particular, independent physician groups have stronger incentives to lower inpatient and hospital outpatient spending than groups integrated with hospitals because their shared-savings bonuses are not offset by forgone profits from reductions in hospital care.36 Our findings suggest that financial integration between physicians and hospitals, which may increase commercial health care prices,37,38 is not necessary for ACO success. Early signals from the MSSP, however, may not predict the long-term efficiencies from various organizational structures under new payment models.39
Savings in the 2012 MSSP cohort were greater for ACOs that had baseline spending above local averages than for those with below-average spending. This finding is consistent with those of previous studies of Pioneer ACOs5,16 and suggests that providers with more opportunities to cut spending can do so more easily. This finding also confirms the importance of encouraging participation by high-spending organizations as CMS considers transitioning benchmarks for ACOs to average fee-for-service spending in their region.30,31,40 Also consistent with findings from the Pioneer model5 is that the decomposition of savings according to type of service in the 2012 cohort suggested a substitution of lower-priced outpatient care in independent office settings for higher-priced hospital outpatient care and a prominent reduction in spending for postacute care, which is known to vary widely across markets and is thought to be a major source of wasteful care.41 Unlike the Pioneer model, however, the MSSP was not associated with significant reductions in the use of low-value services.
Our study had several limitations. First, because the ACO programs are voluntary, participating providers may differ from other providers in ways related to subsequent spending changes. The similar precontract spending levels and trends between ACOs and non-ACO providers, however, show that ACOs were not already constraining spending. Organizations already planning to limit spending may have selectively participated, but implementing such plans under fee-for-service payment would have run counter to their financial self-interest. ACOs probably were more capable of managing care effectively in response to new incentives than nonparticipating providers, but latent differences in the capacity for change would not invalidate inferences that changes in practice were due to changes in incentives. Second, our estimates of savings did not consider the costs to ACOs of efforts to lower spending or improve quality. Third, because statistical tests of subgroup differences were not adjusted for multiple comparisons in our main analysis (see the Supplementary Appendix), they overstate somewhat the confidence with which we could reject the null hypothesis of no difference. Nevertheless, several findings were observed consistently across the Pioneer model and MSSP or across cohorts of MSSP ACOs.
In conclusion, our findings extend evidence of small but meaningful reductions in spending, with unchanged or improved quality of care, early in the Medicare ACO programs and suggest that progress toward net savings to Medicare or society may be slow. Continued quasi-experimental evaluations will be important for identifying successful subgroups of ACOs that have developed strategies that could be disseminated and for assessing progress from regulatory efforts to strengthen incentives and encourage participation.
The content of this article is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health or the Laura and John Arnold Foundation. Supported by grants from the National Institute on Aging of the National Institutes of Health (P01 AG032952 and F30 AG044106-01A1) and from the Laura and John Arnold Foundation. Disclosure forms provided by the authors are available with the full text of this article at NEJM.org. We thank Pasha Hamed, M.A., for statistical programming support and Matthew J. Hayward, B.S., and Jesse B. Dalton, M.A., for research assistance.
From the Department of Health Care Policy, Harvard Medical School ( J.M.M., L.A.H., M.E.C., B.E.L., A.L.S.), the Division of General Internal Medicine and Primary Care, Department of Medicine, Brigham and Women’s Hospital and Harvard Medical School ( J.M.M.), and the Division of General Internal Medicine and Primary Care, Department of Medicine, Beth Israel Deaconess Medical Center (B.E.L.) — all in Boston.
For the full list of references, click here.
This special article originally appeared in The New England Journal of Medicine as “Early Performance of Accountable Care Organizations in Medicare.”