A fascinating feature of this mandatory hip and knee bundled payment is that the government randomized the local areas in which the new model was applied. That randomization, in turn, provides the basis for the new research.

Amy Finkelstein of MIT, Yunan Ji of Harvard, Neale Mahoney of the University of Chicago, Jonathan Skinner of Dartmouth College studied the differences between the hospitals included in the new payment model and those randomly excluded from it. They found some encouraging results, including that the hospitals participating in the new model discharged a significantly smaller share of their hip and knee surgery patients to skilled nursing facilities and other institutional post-acute settings. Previous research has shown substantial variation in the cost and quality of these facilities, suggesting that many may not provide sufficient benefit for the dollars spent on them, and a key objective of the bundled payment model is to reduce the use of inefficient post-acute care. Many hospitals respond to the new payment incentives as expected, by trying to avoid discharges to places like skilled nursing facilities.

The more surprising, and disappointing, results involved the total cost of care. Medicare spending per episode was a bit over $22,400 in the participating hospitals. At those randomly not participating, spending was slightly higher, at $22,900 — but the roughly $500 difference was not statistically significant. And there were no significant differences in observed quality measures, either. Since the whole point of these new models is to improve quality and reduce spending, achieving neither is disheartening.

Before we throw up our hands and give up, though, a few crucial caveats are necessary. Most importantly, the authors only studied the first year of the program. Hospitals, though, may take time to learn how to operate differently under a new model. Experience with Accountable Care Organizations suggests exactly that: Cost savings grow over time as providers learn how to work under the new rules.

In any case, the financial incentives in the first year of the bundled payment program were much weaker than in subsequent years. For example, in the first year, hospitals could gain at most 5 percent of the target price set for the bundle if their costs came in lower than the target, and they faced no penalty if their costs were higher than the target. Ultimately, however, hospitals will be able to gain or lose as much as 20 percent of the target price. The researchers only had the data for the first year, but the results may change materially in subsequent years as experience builds and the financial incentives become more potent (that same conclusion holds for an early evaluation of a voluntary version of bundled payments).

So what lessons can we can learn from this research?

One is the benefit of randomization. Other evidence had suggested much larger savings from bundled payments, but one can never be sure that those don't reflect correlations that aren't due to the program itself. Randomization of the areas in which the new model was implemented (along with the stellar research team that did the evaluation) allows us to have greater confidence that the results are causal. The big differences in results highlight the crucial importance of randomization.

A related lesson is the benefit of continuous learning as we experiment with new payment models. Redesigning payment models is necessarily a messy and imperfect process, and a core component of improving them over time is continuing to learn and make adjustments to them. At the same time, we have to be careful not to overreact to early results, like the research about the first year of the mandatory bundled program, if there is reason to believe the findings may evolve as providers gain experience.

Finally, we should consider whether more of these alternative payment models should be mandatory, because the voluntary versions typically involve excessively weak incentives to induce participation and because we learn less from them (randomization, for example, is dramatically more challenging in a voluntary program). Former Secretary of Health and Human Services Tom Price cut back mandatory bundles (and no, it couldn't have been because he anticipated these preliminary results, which may not turn out to be definitive in any case). His replacement, Secretary Alex Azar, on the other hand, is more open to mandatory programs.

The bottom line is that although the new research is cautionary, I remain supportive of mandatory bundled payments. I hope Secretary Azar introduces more of them, but we will need to continue to monitor their results carefully. Let me be clear: If the ultimate results from the hip and knee bundle are no different than those from the first year, the program should be discontinued – but we are far away from any such conclusion. I suspect that the data from subsequent years will show stronger results.

This column does not necessarily reflect the opinion of the editorial board or Bloomberg LP and its owners.

Peter R. Orszag is a Bloomberg Opinion columnist. He is a vice chairman of investment banking at Lazard. He was director of the Office of Management and Budget from 2009 to 2010, and director of the Congressional Budget Office from 2007 to 2008.

Copyright 2018 Bloomberg. All rights reserved. This material may not be published, broadcast, rewritten, or redistributed.