CHAPTER 4 The CAIR RIA: Advocacy Dressed Up as Policy Analysis – Reforming Regulatory Impact Analysis


Advocacy Dressed Up as Policy Analysis


Critics allege that regulatory impact analyses (RIAS) serve as little more than a fig leaf to hide the contributions of controversial participants in the rulemaking process and provide an illicit entry point for the White House to tinker with agency decisions when an RIA indicates that the agency's proposal is too costly. Others argue that RIAs provide an unaccountable forum for conservative-leaning economists to hijack or at least delay agency policies by requiring analyses that use inherently under protective economic methods and assumptions. The Clean Air Interstate Rule (CAIR) RIA now provides strong evidence for a third, somewhat overlapping source of concern: namely, that RIAS may serve primarily as a mechanism for promoting agency decisions rather than scrutinizing them.

While the role of the RIA as a post hoc rationalization document is surprisingly ignored in the large and growing literature on cost–benefit analysis, it is safe to say that at least in the case of CAIR, this role is indisputable. The CAIR RIA is remarkable for providing almost no information about alternative policies, while at the same time touting the wisdom of the U.S. Environmental Protection Agency's (EPA) preferred program. This key move undoubtedly saved EPA added headaches that would have resulted from a more candid and comprehensive policy analysis, particularly given the unique vulnerability of the CAIR to legal challenge and political opposition. Indeed, by employing economically conservative assumptions, high discount rates, and quantification choices, EPA crafted an RIA peculiarly designed to protect its rule from devastating criticism, at least with respect to the CAIR’s aggregate economic impact.1

From the standpoint of bureaucratic rationality, in fact, it is quite sensible for agencies to use RIAs as propaganda documents rather than self-critical policy analyses. For “significant rules,” which are the subset of rules required to undergo an RIA, the agency faces a very high probability of being sued as well as having its rules criticized by the media, Congress, and even the White House. Moreover, the RIA is generally prepared at the very end of a rulemaking, after a policy decision has solidified. Under these circumstances, few agency attorneys or appointed officials would allow the agency to undertake an honest, searching public analysis of the costs and benefits of various alternatives relative to the agency's rule.

In this chapter, I argue—based on the CAIR case study—that the real challenge to improving RIAs may lie less in perfecting the methodology, and more in overcoming the multiple, entrenched institutional forces that discourage agencies from engaging in public self-evaluation. I advance this argument in four sections. In the first two, I evaluate the CAIR RIA from two different vantage points—first, as an instrument to help insulate the agency from inevitable legal and political attack, and second as a policy instrument. After concluding that the RIA does a superb job at the former but a poor job at the latter, I consider in the third section whether the two goals are mutually exclusive. If they are, then of course that complicates the project for those who would like RIAs to serve as important vehicles for open-minded policy analysis. Although I suspect that the RIA process is too badly broken on an institutional level to be salvaged, I conclude in the final section with some tentative recommendations for reform that may counteract at least some of the pressures for agencies to transform their policy analyses into litigation support documents.

Before proceeding, there are a few caveats about how the CAIR case study might extrapolate to other rulemakings. First, because the CAIR was extremely contentious and was inevitably going to lead to litigation, it may illustrate the worst case rather than the mean in terms of defensive RIAs. Without a threat of significant litigation, an agency might feel freer to conduct honest policy analysis rather than engage in post hoc rationalization crafted to support its rule in the courts. Second, and perhaps most unique to the CAIR, because of the high level of controversy surrounding it, EPA may have had few real options. Legislative attempts to address the problem had largely failed, putting EPA on the hot seat for coming up with an immediate solution. When the alternatives have been largely exhausted and little time remains for meaningful analysis, the benefits of rigorous policy analysis tend to decline, and an agency's failure to engage in candid policy analysis is much more justified institutionally and analytically.

The RIA as Evidence of Bureaucratic Rationality

Based almost exclusively on adverse health effects, particularly mortality from particulates, the CAIR rulemaking is justified by a benefit-to-cost ratio of at least 25 to 1 (EPA 2005c, 25166). Even more encouraging, EPA suggests that in the end the benefits will probably be considerably larger, although how much larger EPA cannot say (EPA 2005a, 1-9). EPA also concedes that the costs to utilities are likely to be overstated, making the proposal even more justified in terms of a benefit-to-cost comparison than its executive summary suggests (EPA 2005a, 7-19–7-21).

The RIA supports this upbeat conclusion with an analysis that is 240 pages in length with a separate, 180-page appendix. The analysis contains dozens of interlocking assumptions, some of which are understandable only to a small group of social scientists or natural scientists. The RIA includes more than four dozen tables and figures that provide the raw numbers to support EPA’s quantification and monetization calculations. It also provides the results of two independent studies EPA conducted to quantify the uncertainties surrounding its health benefits quantifications.

From the standpoint of bureaucratic rationality, the RIA is impeccable. Under Executive Order 12866, EPA is required to provide the Office of Management and Budget (OMB) with a cost–benefit analysis for all significant rulemakings. EPA does this—perhaps with the assistance of OMB itself— in ways carefully crafted to support the viability of a highly controversial rule in both the media and the courts. It also puts its strongest opponents on the defensive by adopting most of their preferred methods and assumptions to support its conclusion that the CAIR is an indisputable social bargain, costing $1 for every $25 in returns.

Before describing the specific ways in which the RIA is bureaucratically brilliant, it is important first to underscore the contentious environment from which the CAIR emerged. As the first chapter explains, a long history preceded the CAIR and influenced its development. Congressional and presidential efforts to design new legislation to address the problem of nonattainment of the National Ambient Air Quality Standards (NAAQS) and other, related problems from utilities had failed or appeared likely to fail. The only viable option left amidst the rubble was EPA’s own proposal, the CAIR, as well as two partner rules, the Clean Air Mercury Rule and the Clean Air Visibility Rule. Few would view any of these rules as the “best” solution; each constituted an uneasy compromise among competing interests. But these rules did offer one virtue lacking in the legislative proposals—the prospect of implementation.

The CAIR not only was an uneasy compromise solution to the problem, it was also a somewhat heroic regulatory intervention because EPA’s explicit legislative authority to address interstate pollution problems under the Clean Air Act (CAA) was quite constrained. The CAIR was not of the ordinary type of rulemaking that responds to a discrete, deadline-driven command by Congress. Instead, in the CAIR, EPA was attempting to address—on its own initiative and without a congressional requirement that it do so—the extensive nonattainment problems in more than 28 states with regard to fine particulates and ozone (EPA 2004, 4580–81). In these states, at least part of the continuing nonattainment problems is the result of interstate transport of pollutants. For example, EPA observes that “the ozone levels floating into Maryland... actually exceed the new 8-hour ozone standard before any Maryland emissions are added” (McGuffey and Sheehan 2005, 67). For such interstate pollution problems, a national or regional approach is warranted, and EPA was the ideal agency to develop such a large-scale strategy.

Because the states have the ultimate authority under the CAA to determine how to attain the NAAQS, however, EPA needed to propose its regional solution in a firm but noncoercive way. EPA’s primary vehicle was its somewhat obscure State Implementation Plan (SIP) Call authority, which allows it to require certain states to make emissions reductions when they are significantly affecting attainment in a downwind state (42 U.S.C. § 7410(a)(2)(D)). In fact, the CAIR SIP Call requires states to cut emissions of sulfur dioxide (SO2) and nitrogen oxides (NOx) roughly in half (EPA 2005c, 25172–73).

One option EPA offers states—the most obvious and probably the least expensive—is the adoption of its cap-and-trade market for electric utilities, which automatically satisfies its reduction requirements. Electric generating units (EGUS) are an especially good target in this regard because their emissions are very high and arguably excessive given the capabilities of existing control technologies (Ibid.).They are also already trading in a national SO2 market, so some of the basic features of the cap-and-trade program are in place and time-tested. If states choose not to join these regional markets, they must then create their own state cap-and-trade market or otherwise determine how best to meet the reduction requirements by EPA’s deadlines and must adjust their SIPS accordingly (Ibid., 25167).

EPA’s rule is legally precarious, however. To make the case for a SIP Call, EPA must establish, among other things, that each state subject to the Call (in this case, 28 states) is “contribut[ing] significantly” to the problematic pollutants (ozone precursors and fine particulates) in downwind states (42 U.S.C. § 7410(a)(2)(D)), and that EPA’s SIP Call solution utilizes “highly cost-effective” controls to mitigate these interstate impacts (EPA 1998, 57363). This first showing—that there is a connection between the “significant” contributions of an upwind state to a downwind state—is largely outside of the reach of a cost–benefit analysis. Indeed, it was this feature of the CAIR that led to its reversal by the DC Circuit (State of North Carolina v. US EPA 2008a).

The CAIR RIA focuses its firepower instead on supporting the latter requirement—namely, that EPA’s interstate trading program will result in “highly cost-effective” controls for problem pollutants—and accomplishes this by showcasing the CAIR’s large benefit-to-cost ratio. Yet in touting these economic virtues of its program, EPA also shrewdly crafted its RIA in a way that dodges and even attempts to conceal fundamental assumptions inherent in the CAIR which leaves it vulnerable to attack. For example, EPA studiously avoided conceding the possibility that trading could lead to hot spots or areas that did not improve with respect to emissions (EPA 2005a, 8-21–8-22). Such evidence would have only resurfaced in the briefs filed by those challenging the CAIR. By instead circling its analytical wagons around the CAIR and side-stepping a meaningful alternatives comparison, EPA avoided making itself unnecessarily vulnerable to a number of attacks that lay just on the horizon (State of North Carolina v. US EPA 2008a; cf. Motor Vehicle Manufacturers Ass'n v. State Farm 1983).

EPA’s strategic use of the RIA as an advocacy document is evident not only from the RIA’s bottom line, but also from a number of individual, methodological decisions that are discussed in more detail below.

Consider only one policy alternative and make sure it produces a whopping benefit-to-cost ratio in support of the rule.

Perhaps most dazzling from the standpoint of blatant agency advocacy, the RIA only considers EPA’s proposed option against the status quo. A robust and insightful policy analysis, as discussed below, would of course consider at least a handful of alternatives in relation to one another. But in the CAIR RIA, by examining only one policy option, EPA positions its final rule as a legal and political no-brainer. The resulting benefit-to-cost ratio of as much as 40 to 1 makes it difficult for the administration to reject the rule as bad public policy and complicates the ability of regulated parties, who shoulder the costs of the CAIR, to complain publicly about the rule. Even better, by presenting a regulatory option that produces such a high benefit-to-cost ratio, EPA helps buttress its legal case that its proposed reductions are “highly cost-effective.”2

Not only does EPA set up a comparison deliberately designed to make its option look fabulous from a political and legal standpoint, but by providing such a limited glimpse of the policy alternatives, EPA reduces the risk of being hung up in litigation about the viability of close competitor approaches. Acknowledging and, worse yet, providing rigorous documentation of alternative approaches would only provide fodder for its many critics and exacerbate its vulnerability in the litigation that was inevitably to follow. In fact, in light of such an overwhelmingly strong benefit-to-cost justification, industry opponents will be quick to do the math and realize that EPA could have justified additional reductions and still supported its rule with a very favorable benefit-to-cost ratio, but declined to do so, presumably in part to protect the interests of industry (Graham 2007, 183; Graham 2008, 473–74; Shore and Patton 2004, 4). Utilities, which are the parties most affected economically by the rule, may thus find EPA’s analysis of alternatives to be woefully incomplete, but only in ways that accrue to their benefit.

Thus, although EPA’s one-option RIA violates the rules of robust policy analysis, it satisfies the general counsel's aim of limiting litigation risks. By supporting its rule with extensive, flattering analysis and studiously avoiding conceding and documenting other equally appealing alternative approaches, EPA transforms this bureaucratic speed bump into a rulemaking asset.

Use opponent-friendly assumptions to minimize the rule's vulnerability to attack as not “highly cost-effective.”

To shore up the vulnerable features of its favorable benefit-to-cost calculation, EPA uses relatively conservative (utility-friendly) assumptions in monetizing the social benefits of the CAIR. Although the utilities take issue with some of the scientific assumptions about the relationship between sulfur dioxide (SO2), nitrogen oxides (NOx), and health effects, these same opponents to the CAIR are noticeably silent about the monetization-related decisions EPA makes for the social benefits (EPA 2005b, 58–90, 128–29, 922–26). This silence occurs for good reason: EPA’s methods uniformly and sometimes heavily tilt in favor of the utilities’ narrow interests. For example, EPA acknowledges a long list of very substantial social benefits that will result from decreased emissions of SO2 and NOx at the levels specified in the CAIR, but only a subset—perhaps fewer than half of these benefits—are quantified. The rest are only referenced qualitatively as “+ B” on the benefits side of the monetized benefit-to-cost ratio used to justify the rule (EPA 2005a, 1-10). If these benefits could be quantified and monetized, however, the true ratio of benefits to costs would be greater, perhaps as much as double (EPA 2005b, 232-66, 929-30).

EPA’s actual monetization of the health benefits it does quantify similarly uses assumptions that tilt in industry's favor. EPA relies on standard willingness-to-pay (WTP) estimates for death and chronic bronchitis, but for most of the remaining health harms, EPA explicitly underestimates their value, relying only on the costs of treatment and lost wages to calculate monetary values (EPA 2005a, 4-14). Finally, EPA conservatively monetizes this subset of social benefits that it is able to quantify and discounts the resulting benefits at both the 3 percent and 7 percent discount rates for 2010 and 2015, thus appeasing even the most vigorous proponents of discounting (Ibid. 4-52).

EPA is also conservative in calculating the costs of the CAIR to utilities, making assumptions that generally overestimate compliance costs (Ibid. 7-19). In so doing, EPA acknowledges that past experience with markets and industry compliance reveals that its compliance cost estimates are likely to be overstated, perhaps significantly (Ibid. 7-19 to 7-21). EPA also assumes that no innovations or unexpected cost reductions in pollution control technology will occur, even though one could credibly make those adjustments if one were interested in the most accurate (mid range) estimate of compliance costs (Ibid. 1-12).

The result is an explicitly inflated estimate of compliance costs associated with the CAIR set against an explicit undervaluation of social benefits. Those who tout economic analysis as an important regulatory tool and view the rule from the perspective of hostile utilities are forced—by their own arguments and philosophies—to be mollified by EPA’s conservative approach to evaluating the costs from the CAIR.

Reinforce the most vulnerable features of the decision with rigorous uncertainty analyses.

Health benefits were an important part of justifying the rule, constituting more than 90 percent of the monetized benefits, but they were scientifically difficult to establish. Determining first how SO2 and NOx reductions from utilities translate into reduced ambient levels of particulates and ozone, and then predicting how these reductions will lead to fewer deaths and health harms requires a number of fragile and uncertain scientific assumptions. On this score, however, adopting utility-friendly assumptions about the relevant science would diverge too far from mainstream scientific views (some utilities appear to contend that neither of these links is supported by the available science) and might force EPA to contradict some of its earlier scientific conclusions (EPA 2005b, 58–90, 128–29, 922–26).

Therefore, to make its rule less vulnerable to this anticipated line of attack against the causal linkages it felt compelled to make, EPA used both Monte Carlo probabilistic analyses and expert elicitation to identify the extent of uncertainty surrounding its quantification of this limited set of health benefits (EPA, 4-77 to 4-83). These uncertainty analyses were performed only on the quantified and ultimately monetized health–benefits, thus capturing only a slice of the total social benefits at issue in the CAIR (Ibid., 1-6).

Fortunately for EPA, both of these uncertainty analyses confirm that its quantifications are well within the middle of the range of only moderate uncertainty (Ibid. 1-6 to 1-8). Moreover, the clever use of both expert elicitation and cutting-edge Monte Carlo analyses provides a redundant measure of the residual uncertainties that effectively shifts the burden to the challengers to identify concrete problems with EPA’s scientific analysis. Although Keohane points out in Chapter 3 that EPA’s uncertainty analyses could have been still better, EPA’s elaborate use of uncertainty analysis to protect its rulemaking soft spot is nevertheless a telling indication of its strategic use of the RIA process.

Don't sweat the stuff that won't be used against you.

At least two executive orders, governing children's health and environmental justice, are arguably triggered by the CAIR, but they are skillfully interpreted by EPA to be inapplicable (Executive Orders 13045 and 12898). EPA’s move is skillful because these requirements are not of concern to the industry challengers and addressing them could begin to unravel EPA’s preferred option. For example, the possibility of hot spots arising from the hoarding of allowances by some utilities seems to raise, at least in the abstract, the possibility of both environmental justice problems and undue health impacts on subsets of children (Ackerman and Heinzerling 2003, 142-45; Drury et al. 1999). EPA concludes, however, that because its required emissions reductions will result in a net improvement in public health, there is simply nothing to analyze with respect to these two executive orders: everybody will be better off (EPA 2005a, 8-21–8-22).

In summary, the CAIR RIA reflects EPA’s excellent strategic skills: EPA followed the maxim “know thine enemies” and built its RIA to please them. It thus seems ironic that despite EPA’s determination to develop an analysis that supported the wisdom of the CAIR, its rule was ultimately dragged through the court system and reversed by the DC Circuit (State of North Carolina v. US EPA 2008a). Does this suggest that using the RIA as an advocacy document failed or was unwise? Definitely not. Such a conclusion is far too simplistic and presumes a one-issue character for a highly complex and multi-faceted regulatory proposal. In invalidating the CAIR, the court found a number of legal errors related to EPA’s reliance on an interstate trading scheme to solve nonattainment problems in individual, downwind states, errors that cumulatively made the program “fundamentally flawed” in the court's view (State of North Carolina v. US EPA 2008a, 58-59). Yet none of these statutory lapses were even remotely connected to whether the CAIR would require controls that were “highly cost-effective,” the central argument advanced by the RIA.

Indeed, the United States’ argument for rehearing and the court's subsequent decision to allow the CAIR to proceed while EPA revises its rule may be due in part to the fine advocacy work embodied in the RIA (DOJ and EPA 2008; State of North Carolina v. US EPA 2008b). The court ultimately decided that vacating CAIR was not in the best interests of the country given the tremendous health and related benefits associated with the program's promised (and highly cost-effective) reductions in criteria pollutants (State of North Carolina v. US EPA 2008b). By contrast, if the litigation had turned on a finding that the compliance costs were not justified by societal benefits or that the controls were not “highly cost-effective” (issues that the RIA arguably took out of contention), then vacating the CAIR may have been legally necessary since those prejudiced by it would be the utilities and others bearing the brunt of the expense of the program.

The CAIR RIA as Policy Analysis

The CAIR RIA’s value in advancing policy analysis is much less impressive. Indeed, the “A” or “A+” that EPA earned in the previous, strategic bureaucratic category drops to an “F” in the policy analysis category. And, consistent with the potentially mutually exclusive nature of the litigation support and policy analysis goals, virtually all of the assets or positive features of the RIA from the standpoint of bureaucratic rationality are mirrored by weaknesses in terms of policy analysis.

The best gauge of the RIA’s success as a policy instrument is to assess whether it meets the general objectives set for it by cost–benefit analysts. According to adherents of cost–benefit analysis, or CBA, the advantages of CBA (and cost effectiveness analysis, or CEA) include:

transparency and the resulting potential for engendering accountability;

the provision of a framework for consistent data collection and identification of gaps and uncertainty in knowledge;

the development of metrics for both the beneficial and adverse consequences of alternative regulatory approaches, allowing those alternatives to be compared to one another (CEA); and

with the use of a monetary metric, the ability to aggregate dissimilar effects (such as those on health, visibility, and crops) into one measure of net benefits.

Compared against this list, the CAIR RIA is a major disappointment. Indeed, it provides almost a textbook example of how not to do cost–benefit analysis. At least a few of the most egregious problems are outlined below.

Considering Only One Alternative

The biggest strength of the RIA in terms of bureaucratic survival is also its primary weakness as a matter of policy analysis. From this standpoint, it is essentially disqualifying to consider only one alternative and, in fact, a one-alternative approach to policy analysis is flatly rejected in the RIA’s analog, the environmental impact statement (EIS) requirement of the National Environmental Policy Act (NEPA) (42 U.S.C. § 4332(C)(iii)) statement; in the OMB’s RIA guidance (OMB 2003 16); in the mainstream policy analysis literature (Keeney 1996); and in the adherents’ objectives for cost–benefit analysis quoted above.

Moreover, EPA simply has no excuse for its decision to consider only one alternative. It would not have been that difficult to design a few easy-to-analyze alternatives in its RIA. For example, EPA could have considered alternative caps and deadlines, as some of the commenters (including Environmental Defense Fund [EDF]) did in their comments (Shore and Patton 2004, 10–19; EPA 2005b, 232–66, 385–94), as well as considering other sources, in addition to electric generating utilities, as potential participants in the interstate market, leading to even more ambitious reduction targets (Shore and Patton 2004, 20-24; EPA 2005b, 152-76). Ideally, EPA could have also considered alternative strategies to its model market that did not involve pollutant trading, such as strict emissions reduction requirements for certain sectors of industry and for transportation.

Indeed, subsequent reports on the CAIR reveal that this alternatives analysis was undertaken and vigorously debated within the executive branch; yet there is scarcely a trace of these critical choices in the RIA itself. For example, the administrator of the Office of Information and Regulatory Affairs (OIRA) at OMB during the CAIR RIA, Dr. John Graham, reports that EPA and OIRA advocated a more stringent reduction target (90 percent) for sulfur emissions, but the White House rejected their proposal and prevailed on the CAIR (Graham 2007, 183; Graham 2008, 469-70). Yet this 90 percent reduction target, the preferred alternative for both EPA and OIRA, is not mentioned, much less analyzed, in the RIA.

Similarly, OIRA advocated that additional sources, like industrial boilers, be included in the rule to accomplish more reductions, and OIRA even went so far as to meet with some of these sources to discuss the proposal (Graham 2008, 473). The RIA, however, makes no mention of the possibility of including these sources, nor does it consider how their inclusion would affect the costs and benefits of the rule. There was also considerable internal executive branch analysis about the extent to which the CAIR would reduce mercury emissions, with internal estimates that CAIR would “reduce mercury emissions to 34 tons by 2020” (Graham 2007, 184). Again, mercury reductions are listed only cursorily in the RIA and these reductions are never quantified or analyzed (EPA 2005a, 1-9).

The distinct possibility that EPA was deliberately holding back on sharing its extensive analysis of alternatives in the RIA is further evidenced by EPA’s publication of a very elaborate alternatives analysis of the CAIR as part of a legislative briefing only seven months after the RIA was finalized (EPA 2005d). This EPA analysis compares the CAIR against five separate legislative proposals and analyzes alternate emissions reductions of NOx and SO2; alternate attainment rates for fine particulates and ozone; and the effects of the six alternatives on coal production, greenhouse gas emissions, mercury emissions, and renewable generating capacity.

An honest consideration of alternative approaches in the RIA might have also helped EPA develop a rule with a much closer fit to efficiency, if that was the ultimate goal. EDF conducted an analysis on marginal costs and benefits that revealed that EPA may not have selected the most efficient alternative (EDF 2004, 7-8), a problem that also concerned OIRA and EPA staff (Graham 2007, 183; Graham 2008, 473-74). Intuitively, in fact, with a benefit-to-cost ratio of about 25 to 1, there should be some room for further reductions in emissions to reach an efficient policy endpoint (Shore and Patton 2004, 4). Given the CAA’s strong legislative commitment to public health protection, moreover, it is conceivable that Congress (and the public) would expect an outcome in which the quantified marginal benefits at least equal the quantified marginal costs, particularly if the benefits side is the one that is only partially quantified.

The disconnect between EPA’s approach and that suggested by rigorous policy analysis again provides support for the observation that the RIA is intended only to support the final CAIR rule and insulate it from opposition from the courts, the White House, other agencies, and Congress. The RIA makes no pretense of seriously analyzing EPA’s approach for efficiency—or any other social goal, for that matter. EPA concedes as much in its conclusory response to commenters who criticized it for its one-alternative approach:

The EPA conducted extensive analyses to determine highly cost-effective control levels, and the optimal criteria for significant contribution determinations. The CAIR will result in significant air quality improvements, reductions in the unhealthy levels of PM2.5 [fine particulate matter] and for many areas of the CAIR region, and is highly beneficial to society (EPA 2005b, 929).

Potentially Huge Unquantified Benefits

EPA admits that the long list of unquantified social benefits could overwhelm the benefits it does quantify and then monetize, but it still ignores them in conducting its net cost–benefit calculation (EPA 2005a, 1-9, 4-22, 4-24). Ecological benefits are perhaps the most significant in this lengthy list of unquantified benefits (Ibid., 1-10). Fine particulates, ozone, nitrates, and sulfates introduce stressors that can lead to a variety of known and untraced adverse consequences for ecosystem health and productivity (Ibid., 4-70–4-73). In cases where these pollutants are reduced substantially— more than halved in the case of the CAIR—agricultural and ecological benefits could be substantial because the affected ecosystems support great expanses of crops, forests, water supplies, and fisheries. A number of those who submitted comments, in fact, underscored the significance of these unquantified ecological benefits in their comments on the CAIR (EPA 2005b, 918-22, 926-27).

Ultimately, however, because of the methodological difficulties that afflict measurements of these ecological benefits, EPA determined that they could not be quantified reliably. (EPA was also unable to quantify all health benefits, such as premature mortality from short-term ozone exposures [EPA 2005a, 4-26].) Thus, although EPA acknowledged that “[t]he net effect of excluding benefit and disbenefit categories from the estimate of total benefits depends on the relative magnitude of the effects” that remain unknown but are potentially significant, the agency perceived it had no choice in this deterministic monetization exercise other than to simply assign this huge set of unknowns a placeholder value of “+B.” Consistent with the nature of the cost–benefit assignment, moreover, these unquantified variables inevitably dropped out of the monetized cost–benefit comparisons (Ibid., 4-22).

From the standpoint of inevitable litigation, EPA’s methodological approach—bracketing and then effectively ignoring these large and uncertain categories of unquantified social benefits—puts a heavy thumb on the side of industry in the RIA analysis that should only help, or at least not hurt in subsequent opposition by that sector. Indeed, by virtue of its approach of considering only one option against the status quo, these added unquantified benefits are effectively irrelevant in any event. Once EPA has justified its proposal as clearly better on the basis of the benefit-to-cost ratio, the fact that even more benefits would accrue than were initially imagined is gravy.

With regard to policy analysis, however, EPA’s decision to ignore a large portion of the benefits because they cannot be quantified is untenable. To identify the most efficient policy option, as noted above, several alternatives need to be considered; this, in turn, means putting all of the benefits on the table to identify the appropriate point at which the marginal benefits and marginal costs are in equipoise. If a large, or potentially large, portion of the benefits cannot be quantified and are removed from consideration, then the analysis is badly incomplete and will not yield the correct outcome. In fact, when a good portion of the benefits are unquantifiable, prominent economists maintain that cost–benefit analysis is no longer appropriate (Morgenstern and Landy 1997, 455, 465, 472, 476).

The goals of policy analysis are also slighted by EPA’s decision to simply mark the unquantified benefits with a placeholder (+B) and ignore them, without proposing how or whether to reduce these uncertainties in the future. If EPA had been serious about policy analysis, it would have recommended ways to collect information on these large sources of uncertainty. The opportunity to underscore important areas of future research and data collection would have been a major benefit of conducting such an analysis. This future information collection could involve, for example, linking measures of agricultural or ecosystem productivity to changes in the concentrations of ozone and fine particulates over time.

Excessive Quantification of the Remaining Subset of Benefits

It follows from the previous weakness that if EPA cannot even be sure it has quantified the bulk of the benefits, subsequent monetization of the remaining quantified benefits becomes practically useless. If (x+y) = social benefits, and y is unknown but is potentially large and perhaps even greater than x, then excessive efforts at monetization of x is not going to move the ball forward in finding the efficient balance point where marginal benefits meet marginal costs. This is not meant to suggest that the appropriate remedy is for EPA to simply put more resources into quantification of y (or in EPA’s terminology, +B), however. EPA persuasively made a case that the ecological benefits were so difficult to predict, both qualitatively and quantitatively, that any estimation would amount to an unverifiable guess. The appropriate response to these quantitative problems is to acknowledge them and abort efforts to arrive at aggregate, monetized costs and benefits.

Indeed, to nevertheless persist with incomplete quantification in such circumstances is not only analytically corrupt, but also, undercuts statutory commands in the Clean Air Act that EPA err on the side of protecting the public health and welfare (42 U.S.C. § 7409(b)). As Doug Kysar points out in Chapter 10, an inflexible commitment to monetization in cost–benefit analysis, particularly when quantification cannot be accomplished, causes the innumerable scientific unknowns arising with respect to health and environmental harms to be zeroed out and chalked up against public health and environmental protection, despite the fact that in most cases the authorizing statute (and the public) has adopted the opposite value choice.

Nevertheless, in its RIA, EPA engages in this “fifth-significant-digit” sort of analysis for only a section of the health benefits and then provides an excruciatingly detailed monetization of that subset of benefits to compare against the industry costs. Except for the promotional benefits achieved by boasting of a 25 to 1 ratio for the CAIR, it is difficult to find any analytical value in this added monetization exercise. In fact, a good argument could be made that monetizing half of the benefits (or some unknown portion) only makes the analysis that much more misleading and confused.

EPA’s obsession with the precise quantification of a subset of benefits begins by mounting a long succession of scientific hurdles to produce a final, error-prone quantitative estimate of a portion of the health benefits. In fact, EPA acknowledges that this final estimate of a subset of health benefits is scientifically precarious:

[M]any inputs were used to derive the final estimate of [a subset of health] benefits, including emission inventories, air quality models (with their associated parameters and inputs), epidemiological health effect estimates, estimates of values (both from WTP [willingness-to-pay] and COI [cost-of-illness] studies), population estimates, income estimates, and estimates of the future state of the world (i.e., regulations, technology, and human behavior). Each of these inputs may be uncertain and, depending on its role in the benefits analysis, may have a disproportionately large impact on final estimates of total [quantified health] benefits. For example, emissions estimates are used in the first stage of the analysis. As such, any uncertainty in emissions estimates will be propagated through the entire analysis. When compounded with uncertainty in later stages, small uncertainties in emission levels can lead to large impacts on total [quantified] benefits (EPA 2005a, 4-19).

In describing these sources of error, however, EPA’s analysis is highly technical, almost always opaque, and at times contradictory. For example, although the agency seems to concede significant sources of error in its predictions of ambient concentrations (Ibid., 3-1, 3-5, 3-11, 3-20) and further concedes a number of contestable assumptions in predicting mortality from particulates (Ibid., 4-11), EPA still boasts in the executive summary of relatively precise estimates of a range of health effects, which contain no error bars (Ibid., Chapters 1 and 9). The agency also lapses into passages intended to illuminate these uncertainties, which are effectively indecipherable:

The procedures for determining the RFFS (relative reduction factors) are similar to those in EPA’s draft guidance for modeling the PM2.5 standard (EPA, 2000). This guidance has undergone extensive peer review and is anticipated to be finalized this year. The guidance recommends that model predictions be used in a relative sense to estimate changes expected to occur in each major PM2.5 species. The procedure for calculating future-year PM2.5 design values is called the “Speciated Modeled Attainment Test (SMAT). EPA used this procedure to estimate the ambient impacts of the CAIR NPR [Notice of Proposed Rulemaking] emissions controls. The SMAT procedures for the No Further Remediation (NFR) have been revised. Full documentation of the revised SMAT methodology is contained in the Air Quality Modeling TSD [Technical Support Document] (Ibid., 3-12).

Although these impenetrable passages may alienate casual readers, they are unlikely to pose a barrier to resourceful utilities. And to defend against these more vigorous stakeholders’ criticisms, EPA then takes its obsessive quantification of a subset of benefits one step further by commissioning not one, but two extravagant uncertainty analyses to shore up its estimates of a subset of health benefits. Monte Carlo probabilistic analyses and a separate expert elicitation ultimately confirm that EPA’s quantification of a subset of health benefits is in the ballpark, at least within one order of magnitude (Ibid., 4-77–4-83). Yet although these analyses help protect EPA’s CAIR from criticism that it is too costly in light of the benefits, from the standpoint of policy analysis they do little to nothing. Understanding the range of uncertainty surrounding x does little to clarify the uncertainty surrounding the larger set of social benefits (x+y) or how to factor these social benefits (x+y) into an evaluation of competing policy options (Ibid., 5-17).

Even assuming that it turns out (presumably through the agency's omniscience) that EPA’s quantification of x is exactly right and that y = 0, EPA’s next quantification step—its monetization of x—consistently underestimates the dollar value of these remaining benefits, perhaps dramatically. Because EPA lacks WTP values for many of the health harms,3 the agency values most illnesses tied to air pollution events, such as emergency room visits, heart attacks, and pneumonia, based only on generic estimates of lost wages and medical care costs (Ibid., 4-52 to 4-53). An emergency room visit for asthma is thus valued at a flat rate of $286 (Ibid., 4-53). Intangible pain and suffering, inconvenience, nonwage opportunity cost, and loss of consortium associated with these harms are all valued at $0 (Ibid., 4-14). As a result, a number of EPA’s monetized calculations are not only inconsistent with one another, but are socially offensive. For example, according to EPA’s RIA, a heart attack is worth more than twice as much if it occurs in a person aged 55-65 as compared with a child or young adult (aged 0 to 24) or a senior (over 65) (Ibid., 4-62). This is because youth and the elderly are generally not employed and there are no wage-related opportunity costs; only the medical costs of treatment. Equally disconcerting is EPA’s monetization of the costs for a child who misses one day of school because of air pollution-caused illness. In this case, the only monetized costs arise for those children whose caretakers work. If one parent is unemployed, the value assigned to a missed day of school is $0 (Ibid., 4-65).

In a rigorous policy analysis (and putting aside the x+y problem, which arguably makes such efforts to monetize x a nonissue in the first place), EPA presumably would have noted or even highlighted these inconsistencies in its valuation of different health harms and tried to redress them. Yet in the RIA, these disparities are noted only in passing and treated as largely irrelevant to the exercise, which they of course are if one assumes that this is a bureaucratic survival document prepared to protect the CAIR from opposition by the utilities.

To add insult to injury, EPA then proceeds to discount this subset of monetized health harms at two separate levels without offering any nondiscounted calculation or noting the legitimate disagreements about discounting nonmarket goods, especially in the intergenerational context (Heinzerling 1999a, 1999b, 1999c, Revesz 1999). The resulting discounted health benefits are presented in the executive summary at 3 percent and 7 percent discount rates with three significant digits and no error bars (EPA 2005a, 1-2).

Behavior of Market Participants

EPA makes note of the fact that cap-and-trade markets may not function exactly as planned because of unpredictabilities in market behavior, but it does not explain or explore what these sources of unpredictability might be or how they could affect its proposal. Instead, EPA flatly assumes in its analysis, without elaboration, “that all States in the CAIR region fully participate in the cap and trade programs” (EPA 2005a, 1-12).

A rigorous policy analysis would dedicate at least several pages or even a chapter to the viability of this assumption, particularly because a cap-and-trade approach is only one of several competing approaches available to EPA to recommend to states in the SIP Call and because unpredictabilities in trading behavior could affect the states’ achievement of the reduction targets. In fact, as EPA well knows, one problem with its cap-and-trade proposal from the standpoint of meeting the health targets is the now-familiar “hot spot” problem, which occurs if a utility purchases large amounts of allowances and produces excessive emissions, thus creating dangerous concentrations of pollutants in localized areas.

Because the CAIR permits banking allowances, including the banking of pre-2010 SO2 allowances, the hoarding of SO2 allowances may also allow for high emissions over time in ways that may not have been adequately explored by EPA (EPA 2005a, 7-4). Such concerns about the effectiveness of the cap-and-trade market present tremendous legal and political vulnerabilities for the CAIR, however. As a result, EPA did not even acknowledge these potential sources of market slippage in its RIA, much less analyze their implications.

Costs to Utilities

EPA’s estimates of the utilities’ compliance costs are based on overly pessimistic projections that do not incorporate the insights gained from retrospective studies of compliance costs in general and SO2 markets in particular (EPA 2005a, 7-19). The agency in fact notes that its compliance cost estimates in the past were overestimated by as much as 80 percent and concedes that all of the errors in its RIA tend in the overestimation direction (EPA 2005a, 7-19; McGarity and Ruttenberg 2002). It further acknowledges that changes in scrubber technology and in demand for electricity are both projected in ways that may cause the compliance cost estimates to be too high, perhaps significantly so (Ibid. 7-19–7-21). Yet EPA never explains why it makes sense to err in the direction of overestimating compliance costs, particularly when it appears that the overestimates may be more than double the expected costs based on the literature.

The Bigger Picture

Utilities contribute not only ozone precursors and fine particulates, but also mercury and greenhouse gases to the airshed; neither of the latter two important pollutants is analyzed in the RIA.4 Unlike some of EPA’s other decisions, this limitation in the scope of the RIA is more defensible: one must find a stopping point, and limiting the analysis to the two NAAQS-related pollutants seems reasonable.

On the other hand, from the standpoint of policy analysis, at least some mention of the important relationships among utility-generated pollutants is warranted. For example, in the RIA, EPA predicts that coal-fired plants will become more important in the future as a source of energy production (EPA 2005a, 1-11), despite growing recognition of their disproportionately high contribution of greenhouse gas pollutants as well as mercury. Might the multiple pollutants produced by utilities caution against cost–benefit solutions that are tipped so heavily in favor of the continued low-cost operation of coal-fired power plants? Do technological solutions exist that might not be much more costly but could begin to reduce the mercury or carbon dioxide (CO2) emissions simultaneously with the control of SO2 and NOx emissions? Would it even be in the utilities’ long-term interest to address these pollutants all at once in a predictable way?

In fact, some of the legislation that EPA compared with the CAIR in an economics briefing addressed these issues and required more renewables and considerably less dependency on coal (EPA 2005d, 15). This type of more unified cost–benefit analysis explaining how the proposal affects all four of these significant pollutants—or explaining how the other two pollutants intersect with the proposal—would seem to be far preferable in an RIA from the standpoint of useful policy analysis (EPA 2005b, 106–27, 422–40). A more comprehensive analysis would also help readers assess how well the president was meeting his pledge to reduce all four pollutants under the Clean Air Act (see Graham 2008, 470).


EPA shrugs off concerns about equitable disparities arising from its proposal by reassuring readers that everybody's health will improve as a result of the CAIR reductions (EPA 2005c, 8-21–22). However, this simplistic assurance begs the question of whether the improvements will be equitable. Will poor people in some areas pay a 2 percent increase in their electricity bills and receive in return very negligible health benefits, or even a reduction in health benefits, because they live in predicted hot spots surrounding some of the dirtiest participating utilities? At the same time, perhaps a number of wealthy communities might pay a similar 2 percent increase in their energy bills, but experience dramatically better air quality—maybe as much as a 10 percent improvement—because they are located in less utility-polluted areas.

The agency also fails to outline other demographic characteristics of those who are most likely to benefit from the reductions. Presumably the biggest beneficiaries are the elderly, children, and African Americans, who are particularly susceptible to asthma, but EPA never suggests that this is the case (Ibid. 4-50). It instead concedes only that the demographic characteristics are important and that it used “projections based on economic forecasting models developed by Woods and Pole, Inc.,” without further elaboration of what these projections revealed (Ibid. 4-24).


Among the touted virtues of cost–benefit analyses are the added transparency and political accountability they provide for submerged agency policy choices as well as their value in positioning the agency's preferred policy against credible alternatives. As the preceding discussions make clear, this noble goal of transparency has also been forsaken in EPA’s RIA. The numerous blind spots and limitations of the agency's analysis—for example, considering only a single alternative— makes the resulting analysis quite opaque because it is completely unclear why EPA made many of the decisions it did. Moreover, even if its analysis were more complete, the 240-pages of technocratic explanations are generally accessible only to experts. Arguably, in fact, even these experts will be unable to evaluate the RIA without assistance from interdisciplinary expert teams because the RIA uses complex models from several disciplines, including the natural sciences, statistics, and economics.

One could argue that the crisp executive summary in the RIA nevertheless makes an important advance in communicating the policy bottom line of the CAIR in an understandable sound bite. In fact, the RIA does make it clear that EPA’s CAIR is, in terms of benefits versus costs, better than doing nothing if one takes all the uncertainties in favor of the affected industry (Ibid, 1-1). But that is about all the RIA does say; it offers no insights about whether EPA’s CAIR is the best policy among alternatives. Thus, to the extent that readers are left feeling as though EPA has struck the right balance in its rule after reading this short summary, they have been badly misled.

The Goal of the Analysis

The RIA is obviously framed around Executive Order 12866’s requirement that agencies conduct cost–benefit analyses on influential rulemakings. However, in a high-quality policy analysis, this singular goal would be supplemented with analyses keyed to other policy objectives more relevant to the regulatory task at hand.

According to some policy analysts, in fact, the most important feature of policy analysis is the goal it selects to evaluate policy alternatives (Keeney 1996, viii, 1, 22). Professors Shapiro and Schroeder in fact persuasively document the legal irrelevancy of cost–benefit endpoints for most environmental mandates (Shapiro and Schroeder 2008, 436). If policy analysis should be done, they argue, it should be based, at least in part, on the goals of the statute (Ibid. 471).

Thus, if EPA’s mandate for SIP Calls actually did require it to identify “highly cost effective” controls in upwind states, then its analysis should focus on evaluating alternatives based on this criterion. Social benefits in such an analysis would arguably be irrelevant; instead the evaluation would consider only the capabilities of competing pollution control technologies and programs over a broad range of total emitters.

Yet although an extensive cost-effectiveness analysis is included in the preamble of the final rule (EPA 2005c, 25195-229), no cost-effectiveness comparison is provided in the RIA despite the centrality of this assessment to the statutory mandate and to economic analysis. EPA may simply have been out of time and unable to include it in the RIA. Or perhaps EPA decided that adding a chapter in the RIA on cost-efficiency and identifying the knee of the curve would only reveal the existence of equally cost-effective alternatives and distract from its otherwise clear and compelling message that the CAIR was justified by a benefit-to-cost ratio of at least 25 to 1.

At an even broader level, to the extent that efficiency—and Kaldor Hicks efficiency, to be precise—is the benchmark against which rulemakings are measured, acknowledging the stark limitations of this goal, particularly with regard to ensuring equitable outcomes, is essential (Ackerman and Heinzerling 2003; Markovits 2008; Sagoff 1981). Indeed, even if equity can still be qualitatively factored into the analysis, the limitations of efficiency as an end goal are too important not to at least acknowledge in the analysis.

Can EPA Navigate Its Rules through the Executive Branch and the Courts and Conduct Rigorous Policy Analysis at the Same Time?

Rigorous policy analysis and bureaucratic survival are not only two very different goals, but they may work at cross purposes. Before developing reforms for regulatory impact analyses, the first order of business is to consider whether the current agency incentive structure creates a viable environment for meaningful policy analysis under any circumstances. If, even with the best tools and guidelines, agencies will be strongly inclined to transform RIAS into advocacy documents, then reforms that focus exclusively on fine-tuning the agency's methods for conducting cost–benefit analyses will miss the target by a considerable margin.

Consideration not only of the CAIR RIA case study, but also of the literature more generally reveals multiple reinforcing reasons, both in theory and experience, to expect that a large number of RIAS will be prepared as self-serving, post hoc rationales rather than open, honest policy analyses. The best analog to the RIA requirement is the National Environmental Policy Act (NEPA), which requires agencies to analyze alternatives to their proposed projects (42 U.S.C. § 4332(c)(iii)). Almost 40 years of experience with NEPA reveals that, although its analytical requirements may help eliminate some of the very worst projects, much of NEPA’s promise of probing policy analysis and agency transparency has given way to agencies that now “act as if the detailed statement called for in the statute is an end in itself, rather than a tool to enhance and improve decision-making,” and turn the environmental impact statement (EIS) into a “litigation-proof” document that does not adequately raise or consider alternatives (CEQ 1997).

Yet in comparison to RIAS, NEPA analyses should be more complete and probing. Agencies conducting them may at least be sued for arbitrary fact-finding or the inadequate consideration of alternatives (Calvert Cliffs v. Atomic Energy Commission 1971).

When agencies conduct an RIA, however, the absence of judicial review means that OMB is the lone analytical police, at least in ensuring the RIA meets minimal standards of rigor and transparency. In some cases, obviously the CAIR RIA being one of them, this permits agencies to prepare RIAS that would not survive judicial review under NEPA because of fundamental analytical flaws, such as failing to consider more than one alternative in the analysis (OMB 2003, 16). This is not to suggest that judicial review should be required for RIAs, for the net benefit of judicial review under NEPA itself is highly contestable and uncertain, but only that experience with NEPA reveals that agency-conducted policy analyses are often not done in rigorous or honest ways.

A similar phenomenon has been observed in agency responses to the Administrative Procedure Act's (APA) notice and comment process. Several scholars have argued that, because notice and comment periods often allow interest groups to have a field day, agencies work behind the scenes to perfect their rules before they are published as proposed rulemakings (West 2004). The result is a great deal of backroom policymaking, including potentially extensive, unrecorded meetings with interest groups that fall outside the protections of the APA because they are done before the proposed rule is published (Ibid.). These APA-generated pressures, which include the prospect of judicial review, may be so great that in some cases agencies may avoid informal rulemaking altogether, particularly when they can make policy another way, through guidances or adjudications for example (Mashaw and Harfst 1987; Pierce 1991).

The possibility that RIAs might follow this same pattern of nontransparent, defensive policy-making observed under NEPA and the APA seems plausible, and the CAIR RIA only serves to reinforce this possibility in practice. In the CAIR, the RIA was published at about the same time that EPA issued its final rule. The RIA was thus finalized 2 years after the proposed rulemaking and 7 to 10 years into the larger policy exercise. The possibility that the RIA was going to provide an honest exposition of alternatives at this stage of the rulemaking seems not only naïve but fantastical, at least from a legal perspective. Under current circumstances, then, bad policy analysis may make for better policy outcomes, all things considered (see Graham 2008, 182–83; Graham 2008, 469–74).

Yet the effect of these institutional forces on the RIA process may be even more perverse. Beyond providing agencies with rational incentives to transform their economic analyses into advocacy documents, institutional pressures may actually create a recurring substantive bias in favor of regulated parties and against environmental and health protection in these analyses. This arises from two overlapping features of the hostile institutional environment in which RIAs are produced. First, empirical research reveals that environmental and health rules tend to be challenged throughout their life cycle much more heavily by regulated parties than those representing the diffuse public who benefit from the protections (Yackee and Yackee 2006). These empirical observations are supported by theory; in the often arcane and highly technical area of environmental and public health regulation, regulated parties have the greatest stake in the rule-making outcomes and also have the most resources to participate. Pluralistic processes in such a setting will be imbalanced just by virtue of the resource and stake equation (Gormley 1986; Komesar 1995).5

In such a lopsided, adversarial environment, agencies are more likely to adopt assumptions that routinely tip in the direction of industry or other powerful stakeholders that enjoy disproportionate influence in the political process. In the case of the CAIR, for example, OIRA demanded that EPA provide “less optimistic” alternative estimates of the benefits of sulfur reduction in its RIA. While it is possible that the conservative tilt of these resulting assumptions may have had nothing to do with efforts to appease industry and related skeptics, the “less optimistic” assumptions led to benefit estimates that were ten times lower than EPA’s original estimates and triggered strenuous objections from some EPA staff (Graham 2007, 183; Graham 2008, 471).

The conservative tilt of the economic analyses is further exacerbated by a second feature: the litigation-driven inclination of agencies to quantify as much as possible in their RIAS to reach a flattering cost–benefit justification. Strong incentives for quantification help to insulate a rule from judicial and political controversy, but for environmental and public health rulemakings, this quantification bias will also cause agencies to bracket and effectively ignore unquantifiable features (as EPA did in the CAIR), almost all of which are most likely to arise on the social benefit side of the equation (Ackerman and Heinzerling 2003, 108-10). Again, the environmental and health benefits will tend to be undervalued in these litigation-motivated quantifications.

It does ultimately appear that RIAs on balance tend, as an institutional matter, to expose rule-makings to challenge by the most aggressive and resourceful stakeholders. If in response to this pressure agencies tend to tip their assumptions in a direction that protects them as much possible from these same future litigants, then the result is not simply an unhelpful policy analysis, but one that may be routinely biased against the public interest.

RIA Reform

In light of these perverse institutional forces, the most obvious and perhaps the only viable corrective is to devise ways to separate the RIA process as much as possible from judicial review and related political pressures on the agency. Ideally, agencies would be rewarded for conducting searching policy analyses or, at the very least, not penalized for self-critical and transparent consideration of alternatives.

If this type of safe analytical space cannot be created within the existing administrative structure, it may be counterproductive to recommend that agencies engage in analyses that only serve to make their already fragile regulatory decisions even more vulnerable to litigation and tipped to favor the most litigious parties. A tool intended for honest policy analysis will be transformed into yet another legal lever that the wealthiest stakeholders can use to pin the agency's rule to their preferred policy outcomes (Schmidt 2002).

This reform section begins with several tentative recommendations for agencies to engage in honest policy analysis without dooming their rulemakings in the process. I remain dubious that these reforms would be sufficient to coax the agency out into the analytical sunshine, however. The second part of the reform section then offers several tentative recommendations for substantive reform of RIA methodology that should accompany these procedural adjustments. Again, these recommendations are only a start and must be supplemented with other, creative reforms if RIAS are ultimately to become useful instruments for encouraging and assessing good policy.

Creating a Safe Administrative Space for Probing Policy Analysis

The first and most vital step for RIA reform is to correct the institutional disincentives—those emerging both from judicial review and the political process—that discourage agencies from conducting honest, searching policy analysis. With regard to the perverse incentives created by judicial review, one reform possibility is to reward an agency that conducts a rigorous RIA with super-deference (that is, a clearly erroneous standard) for its substantive policy choices. For example, in the case of the CAIR, if EPA had conducted an RIA that followed rigorous policy analysis guidelines or survived review by an expert advisory board, the court would attach a strong presumption in favor of the policy choices made in the rulemaking, well beyond the arbitrary standard. Only in cases where the agency does not conduct a rigorous RIA (as evaluated against a respected and established benchmark) would the agency's fact-finding and ultimate proposal receive a harder look with regard to the underlying assumptions and the record as a whole.

This judicial review safety zone would not only provide some reward to agencies that engage in a rigorous policy analysis, but also would protect them from having that analysis used against them. The courts have periodically taken a “hard look” at agency actions and have remanded rules when an agency offers an inadequate explanation for its selection of a particular alternative (among a larger set). This appears even more likely when the agency's own record suggests equally compelling alternatives (Motor Vehicle Manufacturers Ass'n v. State Farm 1983).

Under the proposal here, the agency would be freed of this type of judicial oversight if it provides a high-quality analysis of alternatives. In contrast to current practices of judicial review, then, the adequacy of the agency's actual explanation (or reasoned analysis) for selecting one option over others would receive great deference and would not be subjected to a hard look or even arbitrary and capricious review. Because explaining one choice over others involves primary political considerations, it is the area least amenable to judicial oversight in any event (Chevron v. NRDC 1984). Agency accountability under this recommendation is instead provided by the agency's rigorous and accessible evaluation of the available alternatives, including alternatives that on paper might appear more attractive to some stakeholders than others.

To enjoy the reward of superdeference, however, the agencies would have to first establish that they followed a set of respected guidelines for conducting policy analysis. Guidelines for these policy analyses could be issued by the National Academy of Sciences or some other neutral and respected expert body. The items discussed in the next section provide a start on the substance for such guidelines.

Even with this legal safe harbor, however, the political costs of coming out in the open with controversial policy choices may still lead agencies to shy away from conducting a truly searching policy analysis. As a result, agencies must also receive greater political insulation, which could be accomplished in part by requiring that the RIA or policy analysis process occur at a much earlier point in the rulemaking. An early RIA process would obviously focus the agency on alternatives analysis at a point in its rulemaking when it is still open to such options. Because the agency's RIA would be more preliminary, moreover, it would not need to provide rigorous cost–benefit justifications for each alternative, but could instead provide only crude approximations that are mostly qualitative. Even if the agency does attempt to quantify the alternatives, the time between the RIA and the final rule—the publication of a proposed rule, notice and comment, and final rule—will provide an opportunity for refinement and thus help distance the analysis from the final decision.

If it is also true that agencies are already conducting some of their policy analyses behind closed doors at an early stage in the process, then requiring an RIA at an earlier point may offer a more open and transparent forum for conducting these analyses. In the case of the CAIR, such an early RIA may have emboldened EPA to share its deliberations on alternative reduction targets, additional sources, and additional pollutants in a way that was more insulated from subsequent, strategic litigation decisions. Obviously, however, in situations where the White House or other political officials strike a deal at the outset regardless of a cost–benefit analysis, earlier opportunities for an RIA are unlikely to lead to more honest analyses. In situations where the deal has not been struck until later in the process, however, requiring an early RIA might give the agency a running start in analyzing alternatives in an open-minded way. Such a public analysis could also inhibit subsequent dealmaking, or at least limit the credible options available to political negotiators.

Substantive Adjustments to the RIA Process

Once agencies have the freedom to conduct more searching policy analyses, they may be able to develop creative approaches that not only meet, but exceed most analysts’ wildest expectations. In this “let 1,000 analyses bloom” world of RIAS, the need for formulaic guidelines and rigid itemized lists of characteristics might become unnecessary and inadvertently chill creative and probing analysis.

Assuming, however, that some substantive direction is still needed—at the very least to identify when an agency's RIA should be entitled to superdeference—there are at least three substantive features of cost–benefit analysis (present at least in the CAIR RIA) that would benefit from reform. They are as follows:

Limit unreliable quantifications and eliminate monetizations of nonmarket goods to make the analysis more comprehensive and technically accurate.

Reform begins with an explicit rule that places a premium on presenting an analysis that provides a comprehensive snapshot of all costs and benefits and alternatives, leaving no significant costs or benefits off the table in the final decision.6 Quantified benefits should not dominate when no information suggests that they, in fact, compose the most significant benefits. Charts such as those found in the CAIR RIA in the executive summary that add a “+B” to account for uncertain benefits would thus be strongly discouraged because they omit from the final analysis a potentially significant portion of the benefits.

In this setting of great uncertainty regarding both the quantification of social harm and its monetization, efforts to put only that subset of benefits that can be quantified into monetized units will result in an incomplete and potentially skewed factual basis for decisionmakers and could lead them down the wrong path. The powerful and familiar metric of dollars, for example, tends to anchor thinking in ways that are biased against remaining uncertainties. Partial monetization is also likely to fall prey to the “availability” heuristic: simple, but badly incomplete information— taking the form of cost–benefit price tags—overshadows the uncertainties and related complexities of the underlying tradeoffs (cf. Tversky and Kahneman 1974). Indeed, even if it is true that the incomplete, monetized benefit-to-cost ratio of the CAIR was crucial in changing skeptics’ minds (Graham 2008, 472), the CAIR RIA reveals that in closer cases (which could be the lion's share of rules), unsound policies could result from the natural tendency of decisionmakers to focus primarily only on those benefits and costs that have been monetized.

Even if the quantification of all significant types of social harms could be accomplished reliably, however, monetization of the social harms remains highly problematic in a policy analysis setting because the monetization assumptions are inescapably value-laden, and at the same time are multilayered and difficult to coordinate with or explain to policymakers. As a result, the multiple assumptions used by EPA economic analysts, however well-meaning, can diverge significantly from what a policymaker might believe is appropriate. In the case of the CAIR, for example, EPA provides no indication in the RIA that an estimate of $0 for lost school days or a cost-of-illness approach for most health harms (which gives lower valuations to youth and the elderly) comports with the policy judgments held by decisionmakers or the public at large or is otherwise the best monetized choice among the large range of alternative valuations. Although including WTP for other health estimates at least provides some accounting of public valuations for these intangible costs, the available WTP studies, as well as their extrapolation to a particular question, are similarly value-laden and contestable, with a wide range of equally credible estimates.

At a more conceptual level, the legitimacy of using the market to value nonmarket goods has been vigorously challenged by a number of political scientists, ethicists, lawyers, and even some economists (Shabman and Stephenson 2000). Because the goods are not bought and sold on the market, identifying market-analog prices is highly artificial and also ignores the civic reasons that motivate citizens to protect the public goods or the general public welfare in the first place (Sagoff 1981; Shabman and Stephenson 2000). Moreover, using market analogs (such as revealed preferences or WTP) to isolate market prices for nonmarket goods is based on a number of rosy assumptions about consumer behavior and consumer valuations that are empirically tenuous at best and arguably refuted at worst (Shabman and Stephenson 2000, 383-84).

Set against these multiple problems with the monetization of nonmarket goods is the fact that more transparent and informative means can be used to educate policymakers about the comprehensive costs and benefits of policy options that do not require monetization as a prerequisite. Natural units can be used in place of monetized benefits and costs in ways that provide a much less assumption-laden picture of the societal consequences of a policy. Rather than monetize all of the various health benefits, these benefits are simply listed and set against the costs an individual consumer would be charged in their electricity bills. Empirical work in fact suggests that decisionmakers themselves prefer the more direct, natural unit measurements of the implications of policy alternatives. In the hydropower arena, for example, analysts have found that stakeholders actually prefer qualitative, nonmonetized projections as the basis for negotiation and decisionmaking and tend to ignore monetized valuations (Gowan et al 2006).

In the CAIR, EPA could have taken at least two, complementary approaches to its economic analysis following these recommendations. The first and most straightforward approach for assessing whether EPA’s proposed controls were “highly cost effective” against alternatives would have been to prepare a cost-effectiveness analysis that considers only the cost per ton of emissions reductions in alternative scenarios (see Figure 4.1). In this quantitative comparison, the analysts’ main job is simply to quantify alternative compliance costs and their associated pollutant reductions; there is no need for quantification of social benefits, monetization of social benefits, or discounting. To provide a broader range of alternatives, EPA would also ideally consider sources in addition to the electric utilities, as well as other regulatory methods for accomplishing reductions, beyond markets.

Figure 4.1 Marginal Cost Curve of Abatement for SO2 Emissions from EGUs in 2010
(NOx Emissions at 1.5 million tons)

Reprinted from EPA Rule 2005c, 25204.


An argument can be made, particularly in the wake of the CAIR litigation, that EPA’s assignment was broader and included but did not stop with this assurance of cost-effectiveness. For situations when an aggregate analysis of all of the benefits in relation to the costs is deemed necessary, it is still possible to develop mixed quantitative–qualitative cost–benefit analyses that are much improved in technical accuracy, comprehensiveness, and transparency compared with the incomplete monetized assessment in the CAIR RIA executive summary. In this situation, EPA would summarize its rule using an alternatives table that lists the aggregated costs—ideally to the consumer—on one side compared against the significant quantified and unquantified (but not monetized) benefits, presented on the other side. See Table 4.1, which serves as an illustration (many of the numbers are fictitious; others are drawn from analyses done by Shore and Patton 2004, 1415). Because this approach avoids the problems associated with monetization, it is substantially less error-prone and assumption-laden.

Indeed, this table (or something like it) actually provides the equivalent of a WTP matrix specifically tailored to the rulemaking that sidesteps all of the multiple, assumption-laden steps involved in monetizing the nonmarket goods and considers policy options more holistically (Ackerman and Heinzerling 2003, 210-12). Unlike with WTP surveys, moreover, policymakers would take direct responsibility for choosing the preferred option from the alternatives. The table could even be translated by economists and social scientists into surveys that provide more direct information about the general public's preferences for specific types of trade-offs. To provide more direct public input, the table of alternatives could also be submitted to citizen advisory groups or vetted in public hearings or through notice and comment. Even without these focused sources of public input, the increased transparency should create sufficient political and media pressure on policymakers to make the best choice.

Table 4.1 Alternative Scenarios in Natural Units

Cost to consumers and society at large

Benefits or harms averted


5,000 (± 2,500) premature deaths
7,000 (± 3,500) nonfatal heart attacks
4,000 (± 2,000) hospital admissions
7,000 (± 3,500) acute bronchitis, children

3 million fewer tons of SO2 into ecosystems
0.05 million fewer tons of NOx into ecosystems

2% improvement


13,000 (± 6,000) premature deaths
18,000 (± 9,000) nonfatal heart attacks
11,000 (± 5,000) hospital admissions
16,000 (± 8,000) acute bronchitis, children

5.5 million fewer tons of SO2 into ecosystems
1.9 million fewer tons of NOx into ecosystems

8% improvement


19,408 (± 9,500) premature deaths
27,000 (± 13,000) nonfatal heart attacks
16,000 (± 8,000) hospital admissions
24,000 (± 13,000) acute bronchitis, children

6.5 million fewer tons of SO2 into ecosystems
2.2 million fewer tons of NOx into ecosystems

12% improvement


21,000 (± 10,000) premature deaths
29,000 (± 15,000) nonfatal heart attacks
18,000 (± 8,000) hospital admissions
26,000 (± 13,000) acute bronchitis, children

7 million fewer tons of SO2 into ecosystems
2.6 million fewer tons of NOx into ecosystems

14% improvement

Note: Ecosystem benefits from reduced pollutant loading include benefits to commercial forestry, fishing, recreation, and ecosystem function.

Distributional impacts

If they are likely to be different for vulnerable subpopulations like the poor, the elderly, children, or minorities, then the demographics of those experiencing health effects (and benefiting from the reductions) at each option should also be included more fully in the analysis as well as in the summary tables. The distributional features of each alternative could be graphed as a pie chart of winners and losers in demographic categories or might be flagged only when the distribution of gains and losses, even in a localized area, is likely to vary among alternatives. However they are determined, the distributional implications of alternatives are critical to policy analysis and are missing in the standard methodology of aggregating costs and benefits (Graham 2008, 420-22, 516-24).

If these distributional impacts cannot be estimated quantitatively, then qualitative statements should be provided, both in summary form and in an expanded version, which explains how different vulnerable populations might be affected by the alternatives, not only in absolute terms, but also relative to one another. For example, if middle- and upper-class communities will experience the greatest health benefits under a preferred option relative to the poor, then this feature needs to be communicated. Conversely, if the benefits are likely to accrue primarily to asthmatics and sensitive subgroups, such as children and the elderly, then this important feature of the proposal deserves mention.

Adaptive learning

A number of commentators have criticized administrative processes for inadvertently discouraging agencies from collecting information over time, after a policy is in place (Blais and Wagner 2008; Doremus 2001; McGarity 1992). If RIAS are to accomplish their full potential, agencies must identify the types of technical information that would benefit from further study or collection and propose how this research might be accomplished. Unfortunately, and as discussed earlier, the CAIR RIA simply inserts what is known and moves on. Incorporating adaptive learning into the analysis will not only produce better short-term analyses, but also better policies and information in the long term.


The CAIR case study reveals that EPA has become quite adept at using the RIA to reinforce the wisdom of its rulemaking. That the RIA offers nothing to policy analysis is, in fact, precisely the point; in other words, the point is to protect the rulemaking, not to open it up to attack.

Reform of the RIA process must address this institutional reality. RIAS will never be produced under ideal conditions of academic solitude away from the pressures of litigation and political interference. Instead, a safe harbor for meaningful agency policy analysis needs to be created that insulates the agency from at least some of these institutional pressures. Until then, recommendations for methodological refinements of the RIA process are likely to fall on rationally deaf, bureaucratic ears.


1. I bracket the interesting but currently unanswerable question of who the primary proponent of the CAIR actually was. The evidence suggests, in fact, that the ultimate decision to select the CAIR over alternatives may have been an edict from the White House (Graham 2008). Because this project is focused on the analysis contained in the RIA itself, however, I do not attempt to speculate about the CAIR’s origins.

2. In the CAIR, EPA considers the highly cost-effective controls to be those at the “knee of the curve.” (EPA 2005c, 25201). Undoubtedly, controls with a 20 to 1 benefit-to-cost ratio are at or below this knee.

3. There may also be disagreements about EPA’s values of death and chronic bronchitis that at least included some intangible, pain and suffering factor through WTP surveys. For example, EPA assumes life is worth about $5.5 million per life (in 1990$). EPA 2005a, 4-52. Because these estimates at least include an intangible component, they are presumably less contestable on monetization grounds than many of the other harms discussed in the text.

4. The operation of coal-fired power plants also leads to an increase in worker deaths and health harms as a result of coal mining. (EPA 2005a, 4-13) (alluding to worker safety, but not analyzing it).

5. High-salience rulemakings like CAIR and NAAQS-related rules are likely to generate interest from the full range of affected stakeholders, however, and thus these rules may ultimately be exceptions to this general rule of skewed participation by regulated parties (Blais & Wagner 2008).

6. I am assuming that economic analysis and efficiency remain the goal and bracket my own concerns about the wisdom of those objectives.


Clean Air Act, § 109(b), 42 U.S.C. § 7409(b).

Clean Air Act, § 110(a)(2)(D), 42 U.S.C. § 7410(a)(2)(D).

National Environmental Policy Act, § 102(C)(iii), 42 U.S.C. § 4332(C)(iii)

Calvert Cliffs v. U.S. AEC, 449 F.2d 1109 (D.C. Cir. 1971).

Chevron v. NRDC. 1984. 467 U.S. 837 (S.Ct.).

Motor Vehicle Manufacturers Ass'n v. State Farm Mutual Automobile Insurance Co. 1983. 463 U.S. 29 (S. Ct.)

State of North Carolina v. EPA. 2008a. slip opinion, to be published at 531 F.3d 896 (D.C.Cir.).

State of North Carolina v. EPA, 2008b. 2008 WL 5335481 (D.C. Cir.).

Executive Order 12866.

Executive Order 12898.

Executive Order 13045.

Ackerman, Frank and Lisa Heinzerling. 2003. Priceless: On Knowing the Price of Everything and the Value of Nothing. New York, NY: The New Press.

Blais, Lynn and Wendy Wagner. 2008. Emerging Science, Adaptive Regulation, and the Problem of Rulemaking Ruts. Texas Law Review 86:1701.

Council on Environmental Quality (CEQ). 1997. The National Environmental Policy Act: A Study of Its Effectiveness After Twenty-five Years. Washington, DC.

United States Department of Justice and U.S. Environmental Protection Agency. 2008. Brief in support of Petition for Rehearing or Rehearing En Banc for State of North Carolina v. U.S. EPA. September 24. Washington, DC: US DOJ.

Doremus, Holly. 2001. Adaptive Management, the Endangered Species Act, and the Institutional Challenges of “New Age” Environmental Protection. Washburn L.J. 41:50.

Drury, Richard Toshiyuki et al.. 1999. Pollution Trading and Environmental Injustice: Los Angeles’ Failed Experiment in Air Quality Policy. Duke Environmental Law & Policy Forum 9: 231.

Environmental Defense Fund (EDF). 2004. Stop Blowing Smoke in the Heartland, available at

U.S. Environmental Protection Agency (EPA). 1998. Finding of Significant Contribution and Rulemaking for Certain States in the Ozone Transport Assessment Group Region for Purposes of Reducing Regional Transport of Ozone. Federal Register 63:57356.

———. 2004. Proposed Rule to Reduce Interstate Transport of Fine Particulate Matter and Ozone (Interstate Air Quality Rule). Federal Register 69:4566.

———. 2005a. Regulatory Impact Analysis for the Final Clean Air Interstate Rule. EPA-452/R-05-002. Washingon, DC: U.S. EPA, available at

———. 2005b. Corrected Response to Significant Public Comments on the Proposed Clean Air Interstate Rule, Docket Number OAR-2003-0053, March; Corrected April. Washington, DC: EPA

———. 2005c. Final Rule to Reduce Interstate Transport of Fine Particulate Matter and Ozone (Clean Air Interstate Rule); Revisions to Acid Rain Program; Revisions to the NOx SIP Call; Final Rule, Federal Register 70:25162.

———. 2005d. Multi-Pollutant Legislative Analysis: Comparison Briefing. Washington, DC: U.S. EPA

Gormley, Jr., William. 1986. Regulatory Issue Networks in a Federal System. Polity 18:595.

Gowan, Charles, Kurt Stephenson, and Leonard Shabman. 2006. The Role of Ecosystem Valuation in Environmental Decision Making: Hydropower Relicensing and Dam Removal on the Elwha River. Ecological Economics 56:508.

Graham, John D. 2007. The Evolving Role of the U.S. Office of Management and Budget. Review of Environmental Economics and Policy 1: 171.

———;. 2008. Saving Law through Administrative Law and Economics. University of Pennsylvania Law Review 157: 395.

Heinzerling, Lisa. 1999a. Discounting Life. Yale Law Journal. 108:1911.

———. 1999b. Discounting Our Future. Land & Water Law Review 34:39.

———. 1999c. Environmental Law and the Present Future. Georgetown Law Journal 87:2025.

Keeney, Ralph L. 1996. Value-Focused Thinking: A Path to Creative Decision-making. Cambridge, MA: Harvard University Press.

Komesar, Neil. 1995. Imperfect Alternatives: Choosing Institutions in Law. Chicago: University of Chicago Press.

Markovits, Richard. 2008. Truth or Economics: On the Definition, Prediction, and Relevance of Economic Efficiency. New Haven, CT: Yale Press.

Mashaw, Jerry L. and David L. Harfst. 1987. Regulation and Legal Culture: The Case of Motor Vehicle Safety. Yale Journal on Regulation 4:257.

McGarity, Thomas O. 1992. Some Thoughts on “Deossifying” the Rulemaking Process. Duke Law Journal 41:1385.

McGarity, Thomas O. and Ruth Ruttenberg. 2002. Counting the Cost of Health, Safety, and Environmental Regulation, Texas Law Review 80:1997.

McGuffey, Mark and Gary R. Sheehan. 2005. Taking Care of CAIR. Natural Resources and Environment. Summer 2005: 67.

Morgenstern, Richard D. and Marc K. Landy. 1997. Economic Analysis: Benefits, Costs, Implications. In Economic Analyses at EPA: Assessing Regulatory Impact. Edited by Richard D. Morgenstern. Washington, DC: RFF Press.

U.S. Office of Management and Budget (OMB). 2003. Circular A-4. Washingon, DC: OMB.

Pierce, Jr., Richard J. 1991. The Unintended Effects of Judicial Review of Agency Rules: How Federal Courts Have Contributed to the Electricity Crisis of the 1990s. Administrative Law Review 43:7.

Revesz, Richard. 1999. Environmental Regulation, Cost–Benefit Analysis, and the Discounting of Human Lives Colorado Law Review 99:941.

Sagoff, Mark. 1981. At the Shrine of Our Lady of Fatima or Why Political Questions Are Not All Economic. Arizona Law Review 23:1283.

Schmidt, Patrick. 2002. Pursuing Regulatory Relief: Strategic Participation and Litigation in U.S. OSHA Rulemaking. Business and Politics 4:71.

Shabman, Leonard and Kurt Stephenson. 2000. Environmental Valuation and Its Economic Critics. Journal of Water Resources Planning and Management Nov./Dec. 2000: 382.

Shapiro, Sidney A. and Christopher H. Schroeder. 2008. Beyond Cost–Benefit Analysis: A Pragmatic Reorientation. Harvard Environmental Law Review 32:433.

Shore, Michael and Vickie Patton. 2004. Environmental Defense Fund, Comments on Proposed CAIR Rule to U.S. EPA. March 30 (available through for Docket OAR 2003-0053)

Tversky, Amos and Daniel Kahneman. 1974. Judgments under Uncertainty: Heuristics and Biases. Science 185:1124.

West, William F. 2004. Formal Procedures, Informal Processes, Accountability, and Responsiveness in Bureaucratic Policy Making: An Institutional Policy Analysis. Public Administrative Review 64:66.

Yackee, Jason W. and Susan W. Yackee. 2006. A Bias toward Business? Assessing Interest Group Influence on the Bureaucracy Journal of Politics 68:128.