Co-authored by Michael Greenstone (University of Chicago) and Catherine Wolfram (UC Berkeley) Faculty Directors of The E2e Project.
As featured in the New York Times For Government That Works, Call In the Auditors
The urgency of climate change demands solutions that work. And so, a clear-eyed assessment of how well existing policies and programs are performing is critical, no matter the results. Using rigorous evidence to inform policy design and implementation is the only way to ensure that we effectively confront climate change. The need for objective evidence is particularly important in the case of energy efficiency investments because they play a central role in virtually every single climate change mitigation plan.
Recently, we conducted a first-of-its-kind, randomized controlled trial of the nation’s largest residential energy efficiency program, the federal Weatherization Assistance Program (WAP). The trial consisted of a sample of more than 30,000 WAP-eligible households in the state of Michigan. Our research revealed that investments in residential energy efficiency upgrades among weatherized households in our study cost about double what these households will save on their energy bills (efficiency retrofits cost about $4,600 per household and estimated energy savings are only $2,400). You can read more about our results here.
These findings contradicted commonly held beliefs about the benefits of energy efficiency investments and generated a heated debate. Our previous blog post responded to some of the key concerns.
The Department of Energy (DOE) has since released 36 documents (here and here), totaling 4,446 pages, evaluating WAP. The boldly stated conclusion of the evaluation is that the benefits of the investments are four times the costs.
We have spent many hours poring over these opaque documents. Our judgment is that many of the DOE’s conclusions are based on dubious assumptions, invalid extrapolations, the invention of a new formula to measure benefits that does not produce meaningful results, and no effort to evaluate statistical significance. Using the DOE’s findings, we show below that costs exceed energy savings by a significant margin. We also document major problems with the valuation of non-energy benefits, which comprise the vast majority of estimated program benefits.
Overall, the poor quality of the DOE’s analysis fails to provide a credible basis for the conclusion that the benefits of energy efficiency investments conducted under WAP substantially outweigh the costs. This blog summarizes our assessment of the DOE’s analysis for a general audience. We provide a more technical discussion, using the important example of benefits relating to thermal stress, here.
I. Energy Efficiency Investment Costs Exceed Energy Savings by a Significant Margin.
The typical approach to assessing energy efficiency investments is to test whether the energy savings exceed upfront costs. Using standard economic assumptions, our study and the DOE study agree that costs incurred under the Weatherization Assistance Program significantly exceed energy savings. Here we focus on the ARRA period, but the basic conclusions from outside of ARRA are similar (we explain this here).
What does the DOE find? They collect energy consumption data from approximately 16,000 households. They then compare energy consumption at weatherized households before and after weatherization with energy consumption in a comparison group. Using this basic approach, the DOE finds that the average annual savings for households is $223 per year during the ARRA period. This is the primary result on energy savings.
For most, if not all, households, the decision to pursue weatherization assistance is purposeful. Households may choose to invest the effort of applying for weatherization following – or in anticipation of – a change in energy consumption or their ability to pay energy bills. In other words, there could be important differences in energy consumption patterns across groups of households receiving weatherization in different time periods. The report provides little evidence that the comparison group provides a credible estimate of the energy consumption patterns that would have been observed among weatherized households had they not received weatherization retrofits. The consequence is that we cannot know whether the measured differences in energy consumption are due to the energy efficiency investments or the pre-existing differences between weatherized households and the comparison group.
In contrast, our study is based on a randomized controlled trial (RCT) precisely because we wanted to improve the quality of evidence on the returns to energy efficiency investments. The power of RCTs is that they deliver the true effects, obviating concerns about differences in the types of households that get weatherized and the timing of those decisions. Indeed, this is precisely why society has relied on RCTs to test the efficacy of new drugs for decades and is increasingly using them to evaluate social policy.
Setting aside concerns about the estimation strategy, it is straightforward to convert the estimate of average annual energy savings reported in the evaluation, $223 per year, into a present value of energy savings over the lifetimes of