The 4-year, multipayer Comprehensive Primary Care Initiative was started in October 2012 to determine whether several forms of support would produce changes in care delivery that would improve the quality and reduce the costs of care at 497 primary care practices in seven regions across the United States. Support included the provision of care-management fees, the opportunity to earn shared savings, and the provision of data feedback and learning support.
We tracked changes in the delivery of care by practices participating in the initiative and used difference-in-differences regressions to compare changes over the first 2 years of the initiative in Medicare expenditures, health care utilization, claims-based measures of quality, and patient experience for Medicare fee-for-service beneficiaries attributed to initiative practices and a group of matched comparison practices.
During the first 2 years, initiative practices received a median of $115,000 per clinician in care-management fees. The practices reported improvements in approaches to the delivery of primary care in areas such as management of the care of high-risk patients and enhanced access to care. Changes in average monthly Medicare expenditures per beneficiary did not differ significantly between initiative and comparison practices when care-management fees were not taken into account (–$11; 95% confidence interval [CI], –$23 to $1; P=0.07; negative values indicate less growth in spending at initiative practices) or when these fees were taken into account ($7; 95% CI, –$5 to $19; P=0.27). The only significant differences in other measures were a 3% reduction in primary care visits for initiative practices relative to comparison practices (P<0.001) and changes in two of the six domains of patient experience — discussion of decisions regarding medication with patients and the provision of support for patients taking care of their own health — both of which showed a small improvement in initiative practices relative to comparison practices (P=0.006 and P<0.001, respectively).
Midway through this 4-year intervention, practices participating in the initiative have reported progress in transforming the delivery of primary care. However, at this point these practices have not yet shown savings in expenditures for Medicare Parts A and B after accounting for care-management fees, nor have they shown an appreciable improvement in the quality of care or patient experience. (Funded by the Department of Health and Human Services, Centers for Medicare and Medicaid Services; ClinicalTrials.gov number, NCT02320591.)
Fee-for-service payments give providers the incentive to favor volume over value in the delivery of health care and can produce fragmented care that often lacks coordination, is not patient-centered, and is not proactive in population health management.1–3 Although efforts to improve the delivery of care through changes in primary care (e.g., the use of patient-centered medical homes [PCMHs]) have expanded rapidly in recent years,4,5 early evidence of their effect on the quality and cost of health care is mixed.6
In October 2012, the Centers for Medicare and Medicaid Services (CMS), in collaboration with 39 private and public payers, launched the Comprehensive Primary Care Initiative. The initiative was intended to test a new approach to the payment and delivery of primary care for 4 years in seven regions across the United States, with the goal of improving quality and reducing costs.7 The participating primary care practices were required to make changes in care delivery that would build their capability in five functional areas:
- Access to and continuity of care,
- Planned care for preventive and chronic needs,
- Risk-stratified care management,
- Engagement of patients and their caregivers, and
- Coordination of care with patients’ other care providers.
The initiative did not require practices to have or obtain external recognition as PCMHs.8 The initiative supports the efforts of these practices by offering enhanced payment, data feedback, and learning support.7
The initiative presents an opportunity to evaluate a new multipayer model of payment and primary care delivery in a large and diverse set of practices. In this study, we assess the effects of the initiative on Medicare expenditures, the use of services, selected measures of the quality of care, and patient experience during the first 2 years of the initiative.
The CMS selected seven regions — including four states (Arkansas, Colorado, New Jersey, and Oregon) and three metropolitan areas (Cincinnati–Dayton [Ohio and Kentucky], Hudson Valley–Capital District [New York], and Tulsa, Oklahoma) — on the basis of the extent of payer interest in the initiative and geographic diversity.9–11 Multipayer participation helps facilitate practice transformation by aligning incentives.12 Within the selected regions, CMS chose 502 practices (defined according to physical address) from 978 applicants, using, in large part, a score that weighted meaningful use of electronic health records (EHRs) heavily and did not include expenditures or measures of quality (with “meaningful use” referring to the use of EHR technology to improve the quality of health care and to meet other objectives specified by CMS incentive programs). Scores were not associated with a practice’s expenditures per Medicare beneficiary at baseline or at follow-up. Most of the practices included in the initiative had substantial room to improve care delivery when the initiative began.13
Enhanced payment to initiative practices by CMS and most of the 39 other participating payers was in the form of care-management fees that were not based on visits but were paid on a per-beneficiary per-month basis (in addition to traditional fee-for-service payments) for patients attributed to practices to support and maintain the delivery of enhanced primary care services. (Details of the intervention are provided in Section 1 in the Supplementary Appendix, available with the full text of this article at NEJM.org.) Medicare fee-for-service beneficiaries were attributed on a quarterly basis to practices that delivered the plurality of their primary care visits during a 2-year look-back period (Section 2 in the Supplementary Appendix). For each attributed Medicare beneficiary, CMS paid risk-based, care-management fees that ranged from $8 to $40 per beneficiary per month in the first 2 years of the initiative. The fee level was based on the patient’s hierarchical condition category (HCC, a measure of risk for subsequent expenditures) at the time a beneficiary was first attributed to an initiative practice.14 Other payers (including Medicare Advantage plans, Medicaid managed care, commercial insurers, and CMS [on behalf of Medicaid fee-for-service agencies in some regions]) paid lower fees, in part reflecting the lower average acuity level of their patients.
Annually, beginning in year 2, practices were eligible to share in any Medicare fee-for-service savings resulting from reduced total expenditures, including care-management fees. Many non-Medicare payers also offered practices the opportunity to share in savings. Approaches to calculating shared savings varied across payers.
The approach of Medicare involves calculating savings at the regional level. If a region achieves savings, a practice in that region is eligible to share savings only if it first obtains a minimum number of quality points on the basis of its performance across a set of claims-based measures of quality, electronic measures of clinical quality, and measures of patient experience. The first shared savings distributions occurred in 2015 and were based on savings from 2014 (see Section 1 in the Supplementary Appendix).
Practices also received practice-level feedback reports with quarterly data on Medicare fee-for-service expenditures and service use at the regional, practice, and patient level and received annual data on patient experience and practices’ approaches to care delivery. CMS also funded activities that supported clinician learning, including webinars, in-person meetings, and individualized practice coaching.
We analyzed the 497 practices that were still participating at the end of the first quarter of the initiative (5 practices dropped out after assessing the terms and conditions of participation). We used propensity score matching to select 7 groups of comparison practices — 1 for each region. We selected up to 5 comparison practices per initiative practice to ensure that there were similar characteristics across patients (e.g., age, sex, chronic conditions, and prior expenditures and use of services), practices (e.g., meaningful use of EHRs and number of clinicians), and markets (e.g., mean county income) (Section 3 in the Supplementary Appendix).15 There were a total of 908 comparison practices.
We drew 30% of comparison practices from those that applied to the initiative but were not selected and 70% of comparison practices from those in areas that were near initiative regions and had similar demographic and market factors. Applicants that were not selected to participate in the initiative provided a strong set of potential comparison practices because they expressed the same motivation to participate in the initiative (motivation cannot be observed for practices in external markets), were in the same markets as the initiative practices, and did not differ significantly from initiative practices in baseline risk-adjusted Medicare expenditures or service use (Section 4 in the Supplementary Appendix). Because there were too few unselected applicant practices to ensure close matches for all initiative practices on the matching criteria, we also included comparison practices from nearby markets.
The New England Institutional Review Board (IRB) granted the initiative an IRB exemption on the basis of the federal common rule (section 45 CFR 46.101[b]), because the purpose of the study was to evaluate a public benefit program. The CMS program team designed and administered the execution of the model for the initiative. The manuscript was approved for submission through a standard CMS communications clearance process.
Outcomes and Data
Measures of practice transformation were developed with the use of 37 items from a care-delivery module in a survey of practices that were self-scored on a scale of 1 to 12 points, with higher scores reflecting better approaches to the delivery of primary care. Data were collected from all initiative practices in two survey rounds fielded on the Internet in months 1 through 3 of the initiative and again in months 19 through 22. Twenty-five items were taken from the PCMH assessment instrument (PCMH-A, version 1.3).16 The other items were taken from other surveys or created for the evaluation of the initiative. Members of the evaluation team also visited 21 diverse initiative practices across the seven regions to gather detailed information on program implementation.
We used Medicare claims files (research-identifiable files) from the Virtual Research Data Center. We assessed the effect of the initiative on our primary outcome measures — annualized expenditures in Medicare Parts A and B per beneficiary per month without accounting for care-management fees (gross expenditures) and with accounting for care-management fees (net expenditures). These expenditures did not include beneficiary payments or capitated payments from Medicare for prescription drugs.
To explore the reasons for any changes in expenditures, we also examined utilization measures as secondary outcomes. These outcomes included the annualized number of hospitalizations and outpatient emergency department visits (i.e., visits that did not lead to an admission), numbers of visits to specialists and primary care clinicians, unplanned readmissions within 30 days after a hospital discharge, and hospitalizations for ambulatory-care–sensitive conditions (conditions for which appropriate ambulatory care can prevent or reduce the need for admission to the hospital). Other secondary measures included measures of quality of care (whether patients with diabetes underwent testing for glycated hemoglobin, lipid, and urinary protein levels and had an eye examination, as well as summary measures of whether patients received all or none of these tests, and whether patients with ischemic vascular disease underwent testing for lipid levels) for all beneficiaries and for beneficiaries in the top HCC quartile; continuity of care (determined on the basis of the proportion of primary care office visits at the attributed practice) (see Section 5 in the Supplementary Appendix for a definition of claims-based outcomes); and measures of patient experience.
Outcomes for patient experience were drawn from two rounds of a patient survey distributed by regular mail 8 to 12 months and 21 to 24 months after the initiative began. We sampled a cross section of Medicare fee-for-service beneficiaries who had been attributed to the practice and had visited the practice in the preceding year. More than 25,000 beneficiaries attributed to initiative practices and nearly 9000 beneficiaries attributed to comparison practices responded in each round (we oversampled initiative practices to support practice-level feedback). The survey included six domains of the Consumer Assessment of Healthcare Providers and Systems Clinician and Group Patient-Centered Medical Home Survey (Section 6 in the Supplementary Appendix).17 Response rates for patients from initiative and comparison practices were 45% and 46%, respectively, in the first round and 48% and 47% in the second round.
The sample for our claims-based analysis included 432,080 Medicare beneficiaries attributed to initiative practices and 890,110 beneficiaries attributed to comparison practices during any quarter of the first 2 years of the initiative (October 1, 2012, through September 30, 2014). Our analyses were based on a difference-in-differences framework. For most analyses, each beneficiary contributed up to one observation to the regression analysis during the baseline period (October 1, 2011, through September 30, 2012), one during the first year of the initiative (October 1, 2012, through September 30, 2013), and one during the second year of the initiative (October 1, 2013, through September 30, 2014); however, for the analysis of continuity of care, each beneficiary contributed only two observations, one for a 2-year baseline period (October 1, 2010, through September 30, 2012) and one for a 2-year intervention period (October 1, 2012, through September 30, 2014).
Beneficiaries new to Medicare after the initiative began were included in the analysis but did not contribute a baseline observation. We assigned beneficiary-level weights equal to the product of the share of the year for which the beneficiary was covered by the Medicare fee-for-service program, and a weight ensuring that initiative and comparison practices were balanced (Section 3 in the Supplementary Appendix). Our intention-to-treat approach continued to include in the sample beneficiaries who had died or were no longer attributed to their original practice (because they had begun to obtain the plurality of their primary care visits from a different practice). This approach also continued to include beneficiaries if the practice to which they had been attributed had closed (4 practices), had withdrawn from the initiative (12 practices), had been removed from the initiative (4 practices), had merged (2 practices became 1 practice), or had split (3 practices became 6 distinct practices) (Section 7 in the Supplementary Appendix).
We estimated linear regressions for measures of patient experience and Medicare expenditures with and without care-management fees; zero-inflated negative binomial models for overall hospitalizations, hospitalizations for ambulatory care–sensitive conditions, and emergency department visits; negative binomial models for the number of primary care and specialist visits; and logistic models for readmissions and claims-based quality-of-care measures. The regressions controlled for beneficiary characteristics before the initiative began — demographics (age, race and ethnic group, and sex), region, original reason for Medicare eligibility, Medicaid dual eligibility status, and HCC score — and the baseline characteristics of the beneficiary’s attributed practice, thus netting out remaining observable, preexisting differences between initiative beneficiaries and comparison beneficiaries that were not accounted for by the matching of practices’ mean patient characteristics. Regressions for patient experience also controlled for patients’ prior use of services and self-reported educational level and were weighted to adjust for a potential bias toward nonresponse. All standard errors accounted for the clustering of patient outcomes within practices (Section 3 in the Supplementary Appendix). We performed all statistical analyses with Stata software, version 13.
For two-tailed tests at the 5% significance level, our analysis had 80% power to detect differences in annualized Medicare expenditures that were at least 2% higher or lower than the mean for comparison practices ($16 per beneficiary per month, which is less than the average care-management fee). We did not adjust P values for multiple comparisons but did attempt to avoid type I errors by focusing on summary measures when possible and by using a conservative significance level of 0.01 for all measures of secondary outcomes.
The median total care-management fees from all payers combined over the first 2 years of the initiative were approximately $389,000 per practice, or about 15% of annualized practice revenue, which translates to a median amount of $115,000 per clinician or a mean of $131,000 per clinician. This amount varied according to practice and region depending on the number of participating payers, the number of patients attributed to practices by each participating payer, and each payer’s payment amount.
Changes in Primary Care Delivery
The responses of the practices to the modified PCMH-A survey suggested considerable improvement overall since the start of the initiative (from an average of 6.5 at baseline to an average of 8.8 after 2 years on the basis of a 12-point scale used to assess approaches to the delivery of primary care, with higher numbers indicating better approaches), particularly with regard to risk-stratified care management and access to care, for which the averages increased from 4.6 to 9.7 and from 7.0 to 9.6, respectively (Section 8 in the Supplementary Appendix).
The practice survey and site visits indicated that efforts to undertake transformation were often challenging. Common challenges included refining workflows and procedures for the purpose of implementing, documenting, and reporting initiative requirements, trying to incorporate new staff roles (such as that of care manager) into the primary care team, and trying to overcome the incompatibility of EHRs when attempting to communicate with other providers. Initiative practices began to stratify patients according to risk systematically and hired or repurposed staff to help manage the care of high-risk patients, particularly with respect to providing patient education, monitoring the care of patients with chronic conditions, and providing follow-up after discharge from the hospital or emergency department. To improve patients’ access to care, practices worked on increasing patients’ use of patient portals, decreasing wait times for appointments, increasing telephone access to the practice, and increasing after-hours access to clinicians by means of e-mail, telephone, or in-person visits.
Selected comparison practices and initiative practices had similar characteristics, and the expenditures for and use of services by their attributed Medicare fee-for-service beneficiaries were similar at baseline. (Table 1, and Section 3 in the Supplementary Appendix). The trajectory of Medicare expenditures was similar in the two groups in the 2 years before the initiative began (Fig. 1); regression-adjusted quarterly expenditures increased over time in both groups as beneficiaries became older or died.18 Nearly 4% of both initiative and comparison beneficiaries died during each of the first 2 years of the initiative (P=0.34 and 0.72, respectively).
Effects on Expenditures and Service Use
During these first 2 years, difference-in-differences estimates showed no significant differences (at the 5% level) in the growth of expenditures, without or with the inclusion of care management fees. Without fees, average expenditures in the initiative practices increased $11 less (95% CI, −$23 to $1) per beneficiary per month than those in the comparison practices (difference, 1.3%; P=0.07) (Table 2). With fees, average expenditures in the initiative practices increased $7 more (95% CI, −$5 to $19) per beneficiary per month than those in the comparison practices (difference, 0.9%; P=0.27). A one-sided equivalence test did not support the conclusion that reductions in expenditures without fees equaled or exceeded the fees CMS paid (P=0.87).
Results for overall expenditures were generally consistent across variations in model specifications, the length of time before the initiative included in the baseline, the composition of the comparison practices, and the composition of the sample (i.e., whether we followed beneficiaries attributed in any quarter versus only those attributed in the first quarter), and they did not vary systematically across key subgroups of practices. The estimated effects on Medicare expenditures were larger in magnitude but similar in percentage for high-risk beneficiaries. Effects on expenditures varied across initiative regions. Relative to the comparison group, initiative practices had significant reductions in expenditures when fees were not included in two regions — New Jersey and Tulsa (P=0.005 and 0.026, respectively) — and significant increases in net expenditures when fees were included in Cincinnati–Dayton (P=0.006) (Section 4 in the Supplementary Appendix).
Relative to comparison practices, the number of hospitalizations did not change significantly for initiative practices over the 2-year period (P=0.13) (Table 2), but growth in the number of visits to primary care physicians was 3% less for initiative practices than for comparison practices (P<0.001). There were no significant differences in other outcomes for utilization.
Effects on Quality of Care and Patient Experience
Generally, the initiative did not have significant effects on the processes used as measures of the quality of care for the full sample (Table 3). However, for high-risk patients (the top HCC quartile) with diabetes, the increase in the likelihood of receiving all four recommended tests for diabetes was significantly greater among patients in initiative practices than among patients in comparison practices, with a difference of approximately 3 percentage points in each followup year (P=0.001 in year 1 and P=0.01 in year 2). (See Section 4 in the Supplementary Appendix for detailed results on secondary outcomes.)
Patient ratings of care indicated small improvements with regard to the support providers offered to help patients take care of their own health (3.8 percentage points, P<0.001) and to discuss with patients decisions related to medication (3.2 percentage points, P=0.006). These changes were driven by small improvements (<2 percentage points) in initiative practices and small declines in comparison practices (Table 4). There were no significant effects on other composite measures: ability of patients to obtain timely appointments, care, and information; how well providers communicate with patients; provider’s knowledge of care patient received from other providers; and overall rating of providers by patients.
This evaluation of the large, multipayer initiative after its initial 2 years contributes to our understanding of new approaches to the payment for and delivery of primary care. Prior studies of diverse interventions that focused on the transformation of primary care have been limited and have yielded mixed results.19–33 Most published studies either examined pilot interventions in single markets19–28 with small numbers of practices21–27 and one or a few payers19,21–26,28 or did not examine expenditures.19,24–25,27,29–32 Five studies were conducted in multiple markets and included large numbers of practices or clinics,29-33 but three of these were executed in unusual settings and involved only one payer,29,32–33 and one has not yet examined outcomes.30 In contrast, our study involved a substantial investment from CMS and others through multipayer collaboration, included a large number of practices in diverse regions, and did not require PCMH recognition but did require practices to meet specific requirements across various aspects of care delivery.
Our results suggest that initiative practices are transforming care delivery. However, midway through the intervention, relative to comparison practices, the initiative has not yet generated savings in Medicare Part A and B expenditures that are sufficient to cover care-management fees. The 3% reduction in primary care visits, albeit a small contributor to total expenditures, suggests that nonbillable calls, e-mails, and interactions related to care management, supported by initiative fees, may have supplanted or reduced the need for office visits. We did observe statistically significant but small improvements in two of six domains of patient experience but no appreciable improvements in the quality-of-care measures.
There are a few possible reasons why these findings were not more favorable. First, practices may need more time to fully implement changes in care delivery that translate to improved outcomes.34,35 In addition, since many practices were not necessarily attuned to the details of shared savings, more time may be required for the incentive of shared savings to influence care. It is also possible that primary care practices need stronger value-based incentives, accompanied by consistent incentives for other providers who care for the same patients. In addition, improvements in care that occurred in comparison practices owing to influences such as the growth of accountable care organizations, the increase in penalties for high readmission rates, and other efforts to transform primary care may have made it more difficult for initiative practices to generate savings or broader improvements in quality relative to the comparison practices. Finally, it is possible that practices will reduce expenditures enough to offset a lower fee; CMS will reduce its average fee to $15 per beneficiary per month in the last 2 years of the initiative, reducing not only the gross savings required to reach cost neutrality but also the resources available to achieve those savings.
This study has several limitations. First, practice participation in the initiative is voluntary, and our analysis is limited to their attributed Medicare fee-for-service beneficiaries. Second, because patient experience was not measured before the initiative began, there may have been preexisting differential trends between initiative and comparison practices. Finally, although comparison practices were well matched to initiative practices on the basis of observed characteristics, there could have been differences in unobserved characteristics that influence outcomes.
Analysis of the final 2 years of the initiative will determine the ultimate effect of this approach. As CMS increasingly pays for health care through alternative payment models that reward quality and value, the initiative may help inform future policies guiding models for primary care delivery in the United States.36
The views expressed in this article are solely those of the authors and do not necessarily represent the policy or views of the Centers for Medicare and Medicaid Services (CMS).
Supported by the Department of Health and Human Services, CMS, under contract HHSM-500-2010-00026I/HHSM-500-T0006.
We thank the participating practices and payers and the CMS Comprehensive Primary Care Implementation Team and its contractors for their continual cooperation and willingness to share information and data with the evaluation team; the Research and Rapid Cycle Evaluation Group of CMS for feedback on earlier versions of the manuscript and input into the overall research design; countless staff (too many to name here) from our larger team at Mathematica Policy Research and Group Health Research Institute for their contributions to the evaluation; and Grace Anglin, Richard Chapman, Lara Converse, Tyler Fisher, Mark Flick, John Holland, Jeffrey Holt, Neetu Jain, Zachary James, Nikkilyn Morrison, Brenda Natzke, Connie Qian, Rachel Shapiro, and Xiaofan Sun; Drs. Karen Bogen, Nancy Clusen, Jesse C. Crosson, Nancy Duda, Mariel Finucane, Kristin Geonnotti, Rosalind Keith, Dmitriy Poznyak, Lori Timmins, and Aparajita Zutshi, all from Mathematica Policy Research; and Dr. Robert Reid from Group Health Research Institute.
From Mathematica Policy Research, Chicago (S.B.D.), Princeton, NJ (A.G., D.N.P., F.B.Y., K.S., R.B.), and Washington DC (E.F.T., A.S.O.); and the Centers for Medicare and Medicaid Services, Baltimore (T.J.D., P.H.C., R.R., M.J.P., L.S.). Address reprint requests to Ms. Dale at Mathematica Policy Research, 111 E. Upper Wacker Dr., Chicago, IL 60601, or at firstname.lastname@example.org.
For the full list of references, click here.
This special article originally appeared in The New England Journal of Medicine.