Co-authored by Michael Greenstone (University of Chicago) and Catherine Wolfram (UC Berkeley) Faculty Directors of The E2e Project.
As featured in the New York Times For Government That Works, Call In the Auditors
The urgency of climate change demands solutions that work. And so, a clear-eyed assessment of how well existing policies and programs are performing is critical, no matter the results. Using rigorous evidence to inform policy design and implementation is the only way to ensure that we effectively confront climate change. The need for objective evidence is particularly important in the case of energy efficiency investments because they play a central role in virtually every single climate change mitigation plan.
Recently, we conducted a first-of-its-kind, randomized controlled trial of the nation’s largest residential energy efficiency program, the federal Weatherization Assistance Program (WAP). The trial consisted of a sample of more than 30,000 WAP-eligible households in the state of Michigan. Our research revealed that investments in residential energy efficiency upgrades among weatherized households in our study cost about double what these households will save on their energy bills (efficiency retrofits cost about $4,600 per household and estimated energy savings are only $2,400). You can read more about our results here.
These findings contradicted commonly held beliefs about the benefits of energy efficiency investments and generated a heated debate. Our previous blog post responded to some of the key concerns.
The Department of Energy (DOE) has since released 36 documents (here and here), totaling 4,446 pages, evaluating WAP. The boldly stated conclusion of the evaluation is that the benefits of the investments are four times the costs.
We have spent many hours poring over these opaque documents. Our judgment is that many of the DOE’s conclusions are based on dubious assumptions, invalid extrapolations, the invention of a new formula to measure benefits that does not produce meaningful results, and no effort to evaluate statistical significance. Using the DOE’s findings, we show below that costs exceed energy savings by a significant margin. We also document major problems with the valuation of non-energy benefits, which comprise the vast majority of estimated program benefits.
Overall, the poor quality of the DOE’s analysis fails to provide a credible basis for the conclusion that the benefits of energy efficiency investments conducted under WAP substantially outweigh the costs. This blog summarizes our assessment of the DOE’s analysis for a general audience. We provide a more technical discussion, using the important example of benefits relating to thermal stress, here.
I. Energy Efficiency Investment Costs Exceed Energy Savings by a Significant Margin.
The typical approach to assessing energy efficiency investments is to test whether the energy savings exceed upfront costs. Using standard economic assumptions, our study and the DOE study agree that costs incurred under the Weatherization Assistance Program significantly exceed energy savings. Here we focus on the ARRA period, but the basic conclusions from outside of ARRA are similar (we explain this here).
What does the DOE find? They collect energy consumption data from approximately 16,000 households. They then compare energy consumption at weatherized households before and after weatherization with energy consumption in a comparison group. Using this basic approach, the DOE finds that the average annual savings for households is $223 per year during the ARRA period. This is the primary result on energy savings.
For most, if not all, households, the decision to pursue weatherization assistance is purposeful. Households may choose to invest the effort of applying for weatherization following – or in anticipation of – a change in energy consumption or their ability to pay energy bills. In other words, there could be important differences in energy consumption patterns across groups of households receiving weatherization in different time periods. The report provides little evidence that the comparison group provides a credible estimate of the energy consumption patterns that would have been observed among weatherized households had they not received weatherization retrofits. The consequence is that we cannot know whether the measured differences in energy consumption are due to the energy efficiency investments or the pre-existing differences between weatherized households and the comparison group.
In contrast, our study is based on a randomized controlled trial (RCT) precisely because we wanted to improve the quality of evidence on the returns to energy efficiency investments. The power of RCTs is that they deliver the true effects, obviating concerns about differences in the types of households that get weatherized and the timing of those decisions. Indeed, this is precisely why society has relied on RCTs to test the efficacy of new drugs for decades and is increasingly using them to evaluate social policy.
Setting aside concerns about the estimation strategy, it is straightforward to convert the estimate of average annual energy savings reported in the evaluation, $223 per year, into a present value of energy savings over the lifetimes of the measures installed by making several assumptions. The following table describes the present value of lifetime savings under different assumptions about the discount rate and the lifespans of the measures. Notably, the table follows the standard assumption in the economics literature and assumes energy prices will remain constant, after adjustment for inflation. It also generously assumes that there is no depreciation in the savings created by the measures.
|Net Present Value of Energy Savings from DOE Study of ARRA Period|
|Time Horizon||Discount Rate|
|3 Percent||6 Percent||10 Percent|
|Note: The table reports the net present value of energy savings (in dollars) implied by the DOE’s household-level energy savings estimates ($223) using a range of discount rates and assumed time horizons. Reductions in energy bills associated with the estimates are assumed to accrue over the life of the measure.|
These “present value” energy savings can be compared against the average cost per weatherized household. The DOE reports a wide variety of costs, and it is difficult to judge which should be compared to the savings. The most straightforward is the average total expenditure per weatherized household: $6,812 during the ARRA period. The DOE reports incurring costs per household of $5,926. When comparing household benefits and costs, the DOE evaluation uses a much lower $3,745 per household, although it is unclear what cost components are omitted to arrive at this low number.
Using a 6 percent discount rate that seems a reasonable approximation of households’ borrowing costs, a 16-year time horizon (the average projected lifespan of efficiency measures in our data), and assuming no depreciation in annual energy savings (likely generous), the DOE estimate of annual energy savings implies a present value of $2,254 per household. This is less than half of the DOE expenditures per household. It also falls significantly below the lowest reported costs of $3,777.
To compute the average discounted value of energy savings, the DOE uses a longer time horizon, a lower discount rate, and assumes that energy prices will increase in real terms over time – all assumptions that make the savings look larger. These assumptions imply a higher present value of $3,190 per household. This higher value notwithstanding, WAP program costs still exceed estimated savings by a significant margin.
The conclusion that the program is remarkably cost effective cannot be justified on energy savings alone. The DOE’s claims rely heavily on the possibility of substantial non-energy benefits. We explore the credibility of these estimated benefits next.
II. Unreliable Approaches to Exploring Whether there are Non-Energy Benefits.
The DOE evaluation reports a program-wide benefit to cost ratio of 4 to 1. To arrive at this conclusion, the report points to a long list of “non-energy” benefits. They include health-related benefits such as reductions in thermal stress and asthma (valued at $6,870 per household) and improved productivity at home and at work due to improved sleep (valued at $3,142 per household), safety-related benefits such as reduced fire risk and reduced carbon monoxide levels (valued at $985 per household), and other benefits such as reduced reliance on food and energy assistance programs (valued at $1,641 per household). The rationale for relating energy efficiency investments to some of these outcomes (e.g., improved productivity due to better sleep) is not well articulated, but nevertheless we examined the basis for the claims.
A close look at these claimed benefits uncovers significant problems with the methods used and assumptions invoked. We highlight three general concerns here.
- Non-energy outcomes are not directly measured. And the path from imputed impacts to benefits estimates is generally dubious.
In contrast to energy consumption, which is measured directly using household energy bills, most of the non-energy benefit estimates are based on survey responses from program participants. Connecting the dots between survey responses and non-energy benefits involves many steps. The resulting benefit values are extraordinarily sensitive to many key assumptions, several of which are not well explained or substantiated.
The significant benefits attributed to reduced hospitalization rates after weatherization provide one example. DOE surveyors asked a select set of households the following question: “In the past 12 months, has anyone in the household needed medical attention because your home was too cold (hot)?” Using a formula they invented that has never appeared in any textbook (more on this below), the DOE deduced from the survey responses that weatherization assistance reduced the share of households needing medical attention by 1.4 percent.
The path from the response to this survey question to monetized benefits is long and winding. The nearly $5,000 per household of mortality reduction benefits are p