Decision Support Framework for Developing Regional Energy

Jan 8, 2014 - Likewise, the technical complexity associated with the kinds of problems for ... In order to develop and test a decision support framewo...
1 downloads 0 Views 608KB Size
Download PDF
Policy Analysis pubs.acs.org/est

Decision Support Framework for Developing Regional Energy Strategies Douglas L. Bessette,*,†,‡ Joseph Arvai,†,‡,§ and Victoria Campbell-Arvai†,‡ †

Department of Geography, University of Calgary, 2500 University Drive NW, Calgary, Alberta T2N 1N4, Canada Institute for Sustainable Energy, Environment and Economy, University of Calgary, 2500 University Drive NW, Calgary, Alberta T2N 1N4, Canada § Decision Research, 1201 Oak Street, Eugene, Oregon 97401, United States ‡

S Supporting Information *

ABSTRACT: In an effort to reduce “carbon pollution” as well as prepare the U.S. for the impacts of climate change, President Obama’s 2013 Climate Action Plan calls for changes to be made to the nation’s energy system. In addition to focusing on alternative portfolios of different fuels and power-generation technologies, researchers and advisory panels have urged that changes to the nation’s energy system be based on a decisionmaking framework that incorporates stakeholders and accounts for real-world resource, supply, and demand constraints. To date, research and development on such a framework have proven elusive. The research reported here describes the development and test of a potential decision support framework that combines elements from structured decisionmaking (SDM) with portfolio analysis, methods that have been used independently to elicit preferences in complex decision contexts. This hybrid framework aimed to (1) provide necessary background information to users regarding the development of coupled climate-energy strategies; (2) account for users’ values and objectives; (3) allow for the construction of bespoke energy portfolios bounded by real-world supply and demand constraints; and (4) provide a more rigorous basis for addressing trade-offs. Results show that this framework was user-friendly, led to significant increases in users’ knowledge about energy systems and, importantly, led to more internally consistent decisions. For these reasons, this framework may serve as a suitable template for supporting decisions about energy transitions in the United States and abroad.

1. INTRODUCTION

Developing rigorous and stakeholder-based climate-energy strategies will be a complex and challenging undertaking. To cut through this complexity, Arvai et al.2 and the National Research Council7 have argued that energy strategy development should go beyond identifying specific generation and delivery options to also include the development of transparent, inclusive, and scientifically rigorous decision-making frameworks. These frameworks, which may be deployed regionally or nationally, should guide multistakeholder, evidence-based deliberations about energy development and delivery. Having these kinds of frameworks at the ready would add legitimacy to efforts like the President’s CAP, as well as to many regional energy transitions currently underway, while also ensuring that the most common judgmental obstacles preventing defensible decision-making are being addressed. These judgmental obstacles are considerable. For instance, decision makers often fail to fully characterize and bound

In his 2013 Climate Action Plan (CAP), U.S. President Barack Obama stated that the United States stands at a “critical juncture” with respect to climate change and its environmental consequences and economic costs.1 In response, the President put forth a broad-based plan to reduce air pollution, spark business innovation, “grow” new fuels and engineer new sources of energy, and increase the efficiency of cars and appliances. Obama linked these objectives specifically to climate change by calling for a reduction of what he termed “carbon pollution” in America. At its heart, the President’s CAP calls for changes in the nation’s energy system, which relies on a portfolio of different fuels and power-generation technologies, as well as research and development activities targeted at new infrastructure and energy investment. Thus, despite the CAP’s clear emphasis on climate change, it also speaks to the development of a new national energy strategy. This call for linking climate change to a national energy strategy echoes several high profile calls from both researchers2−5 and advisory panels6,7 working domestically and abroad. © 2014 American Chemical Society

Received: Revised: Accepted: Published: 1401

August 15, 2013 December 13, 2013 January 8, 2014 January 8, 2014 dx.doi.org/10.1021/es4036286 | Environ. Sci. Technol. 2014, 48, 1401−1408

Environmental Science & Technology

Policy Analysis

Figure 1. Research design.

to help people make more internally consistent, rational choices.11 Six basic elements comprise an SDM framework: (1) clearly defining the decision problem, along with its bounds and constraints; (2) identifying objectives that will guide the decision-making process, along with the performance measures that will gauge their success; (3) creating logical and creative alternatives that address these objectives; (4) establishing the consequences of each alternative; (5) confronting inevitable trade-offs when selecting among alternatives; and finally, (6) implementing decisions, monitoring outcomes, and adapting to changing conditions.2,11 In addition to applying SDM, researchers have since the early 2000s relied on portfolio analysis, or the construction of unique packages of alternative fuels and power-generation technologies, to elicit beliefs about the interrelated objectives of energy strategy development.19 For example, Palmgren et al.20 asked respondents to rank their willingness-to-pay for different hypothetical no- and low-CO2 emission options, with each option presented blending different fuels and technologies. Fleishman et al.5 expanded upon this approach by asking participants to rank portfolios developed to account for realworld constraints and resources, with each option emitting less CO2 than the status quo. Each approach, SDM and portfolio analysis, represents an important step forward in energy strategy development. However, research suggests that both SDM and portfolio analysis are not without their shortcomings. SDM processes, for example, tend to be time and resource intensive. Likewise, the technical complexity associated with the kinds of problems for which SDM is typically applied, coupled with the complexities inherent in multiattribute decision-making, mean that active facilitation is often required.11 Portfolio analysis, in contrast, is more intuitive (and, hence, typically quicker than SDM). The emphasis here is placed on developing or ranking options that account for real-world constraints. This however comes at the expense of more rigorous methods for evaluating options that explicitly account for trade-offs across objectives. For example, portfolios are typically compared holistically (e.g., via ranking) rather than on an attribute-by-attribute basis. Taking this approach means running the risk of overweighting salient attributes and, hence,

decision problems, either casting them too narrowly, e.g., focusing on a single objective, such as reducing CO2 emissions, or casting the problem too broadly by incorporating too large a set of competing objectives and stakeholders.8 Related, decision makers commonly struggle with decisions involving multiple objectives, often failing to consider or weigh the importance of various attributes that might be significant yet not immediately apparent. In each of these cases, the result is deliberative paralysis,9 either because the alternatives put forth are judged to be too simple (and, hence, partisan) or because the decisionmaking processes are simply too complex and prone to stalemate. Beyond judgmental obstacles, decision makers must also deal with informational challenges. For example, many stakeholders and decision makers are generally unfamiliar with the range of attributes that warrant consideration during strategy development, e.g., the economies of scale associated with large solar farms or the reliability of fuel cells. This lack of thorough understanding, combined with the fact that decision makers tend to anchor on particularly salient alternatives and fail to think creatively about possible solutions,10 can result in individuals relying too heavily on status quo infrastructure or existing high-value investments. As a result of these difficulties, decision makers often fail to adequately specify consequences and confront difficult tradeoffs. Instead of accounting for anticipated gains and losses, individuals often rely on intuitive impressions reflecting their values or priorities.11 This is not to suggest that intuitive responses do not have a role to play alongside more effortful analysis; see Kahneman,12 for a review. However, previous research suggests that effective decision support systems will help decision makers to make more thoughtful trade-offs across objectives and better balance intuitive and analytic modes of reasoning.13,14 Previous work in the related context of environmental and risk management15−17 has been notably successful in addressing these challenges through the application of structured decisionmaking (SDM). SDM relies upon an analytic-deliberative process 18 that makes complex technical concepts more accessible to decision makers by decomposing complex decision problems. SDM also provides a framework intended 1402

dx.doi.org/10.1021/es4036286 | Environ. Sci. Technol. 2014, 48, 1401−1408

Environmental Science & Technology

Policy Analysis

the selection of alternatives that do not adequately account for decision-makers’ full range of concerns.13,17 In this research, we sought to develop and test a decisionsupport framework that would adopt the strengths of SDM and portfolio analysis, specifically one that would provide necessary background information regarding the development of coupled climate-energy strategies, account for decision-makers’ values and objectives, allow participants to develop their own unique energy portfolios that were responsive to real-world opportunities as well as supply and demand constraints, and provide a more rigorous basis for addressing trade-offs. Importantly, we also sought to develop a framework that would be intuitive and user-friendly, was accessible to a wide range of users, and could serve as a template for stakeholder-based deliberations about energy and climate strategy.

that had been deemed suitable by engineers from MSU and Black & Veatch. Twenty different generation options were presented to participants (Table 1; Supporting Information, Table S-2). Table 1. Summary of Generation Options, Energy Portfolio Composition (For Treatments 3 and 4) Represented by the Percentage of Participants Selecting a Given Option, and the Percent Contribution of Each Option toward Total Energy Output generation options

participants’ portfolio (%)

Powerplant Options small modular nuclear reactor 75.0 biofuels + CCS 38.5 natural gas 32.3 coal + CCS 28.1 biofuels + coal (cofired) 25.0 biofuels 22.9 coal 18.8 natural gas + CCS 12.5 natural gas + coal (cofired) 9.4 Decentralized Options distributed natural gas 38.5 solar photovoltaic 27.1 distributed solar 21.9 distributed wind 14.6 fuel cells 12.5 hydroelectric 8.3 microturbine 7.3 Off-Site Options wind farm 24.0 solar farm 22.9 renewable grid purchase 14.6 conventional grid purchase 4.2

2. METHODS 2.1. Context. In order to develop and test a decision support framework in a real-world setting, the context for this research was an initiative at Michigan State University (MSU) aimed at generating a new long-term plan for the university’s power-generation system. Electricity, as well as steam for heating and cooling, is currently generated on the MSU campus by way of a 99-megawatt coal, natural gas, and biofuel cogeneration facility. As part of the university’s ongoing sustainability initiative, administrators initiated a deliberative process aimed at involving stakeholders in decisions about the future of the current power plant. This initiative offered a unique opportunity to develop and test the kind of decisionmaking framework outlined above. 2.2. Online Framework. A 6-part research framework (Figure 1) was constructed reliant upon an online software module developed by Compass Resource Management (Vancouver, BC, Canada) and an interactive energy system model developed by Black & Veatch (a multinational energy consultancy), Compass Resource Management, and MSU’s Chief Energy and Environmental Engineer. The energy system model was tailored specifically to MSU’s campus. Participants engaged with each of the six parts (described below), including the energy system model, or “energy portfolio builder” (Supporting Information, Figure S-1), through an online dashboard. Despite the framework being entirely online, a facilitator (the paper’s first author) was present during all treatments to provide instruction to participants. 2.3. Design. The 6-part decision support framework relied on two independent variables: whether or not participants were given an opportunity to construct their own energy strategies (i.e., portfolios) and the order in which participants undertook holistic ranking and attribute weighting tasks. The result was a 2 × 2 factorial design, yielding four treatment scenarios (Figure 1, T1−T4). Part 1 of the design contained a 7-point, closed-ended, selfrating survey intended to collect demographic information, as well as baseline information pertaining to participants’ knowledge levels about energy systems (Supporting Information, Table S-1). In part 2, all participants reviewed a short online primer, presented as a series of webpages, about energy systems on the MSU campus. This primer provided participants with general information about energy-related concepts such as supply and demand, as well as information about possible efficiency upgrades and the fuel types and generation options

total energy output (%) 20.5 8.9 6.7 6.8 6.2 3.7 4.7 2.0 1.8 9.5 5.2 4.3 2.7 2.5 1.3 1.3 4.3 3.8 2.8 0.8

These options were subdivided on the basis of whether they would be deployed in a traditional centralized power plant, in a decentralized fashion, or off site. The power plant options, coal, natural gas and biofuels, were each available with or without additional carbon capture and storage (CCS) technology. During part 2, participants also read about the relationship between energy generation and the six key objectives elicited from stakeholders living and working on or near the MSU campus prior to the start of this experiment: reducing the costs, capital and operating, associated with energy generation as measured by a tuition increase passed on to MSU students; reducing greenhouse-gas emissions (GHGs) as measured by a percent reduction from the status quo; improving local air quality by reducing particulate emissions as measured by a percent reduction from the status quo; creating employment opportunities as measured by the additional number of full-time equivalent jobs created; minimizing land use impacts associated with energy generation as measured by the acreage required; and pursuing energy strategies that are seen as highly innovative as measured on a constructed Likert scale ranging from 1 (not at all innovative) to 4 (very innovative). At the conclusion of the primer, participants were provided with an overview of the individual decision support tasks that would follow. The presentation of information and tasks across the framework’s four treatments was identical with one exception: participants in Treatments 1 and 2 were provided instructions on how to 1403

dx.doi.org/10.1021/es4036286 | Environ. Sci. Technol. 2014, 48, 1401−1408

Environmental Science & Technology

Policy Analysis

Table 2. Energy Strategies (i.e., Portfolios) Presented to Participantsa energy strategy option objective

attribute

1b

2

3

4

5

6c

6d

cost (↓) GHG emissions (↓) particulate emissions (↓) employment (↑) land use (↓) innovation (↑)

annual premium ($) reduction from current (%) reduction from current (%) full time equivalents acres required 1 (low)−4 (high) scale

0 0 0 0 0 1

88 28 13 5 13 1.6

544 91 84 41 13 2.5

362 64 27 39 54 2

776 100 100 60 18 3

0 30 14 0 0 1

participant-specific

a

In treatments 1 and 2, participants evaluated options 1b through 5, and 6c. In treatments 3 and 4, participants evaluated options 1b through 5, and 6d. Symbols (↓ and ↑) corresponding with each attribute represent the desired directionality as defined by stakeholders’ objectives. bRepresents maintaining the status quo. cAdditional investigator-generated option presented in treatments 1 and 2. dParticipant-generated option presented in treatments 3 and 4.

Table 3. Mean Self-Ratings Elicited from Participants by Treatment treatment 1 self-rating item 1. 2. 3. 4. 5. 6. 7.

satisfaction with decisionsa stress during decision makinga difficulty of decision making tasksa level of accuracy of decisionsa amount of information providedb comfort with input supporting actual energy decisionsa knowledge level (Δx)̅ c

treatment 2

treatment 3

treatment 4

x̅

SE

x̅

SE

x̅

SE

x̅

SE

5.70 3.15 3.35 5.63 3.54 4.89 0.81

0.16 0.23 0.20 0.18 0.16 0.20 0.13

5.80 3.10 3.67 5.65 3.68 5.05 0.96

0.16 0.22 0.29 0.18 0.20 0.21 0.13

5.51 3.28 3.57 4.72 4.23 5.09 1.05

0.18 0.23 0.18 0.21 0.16 0.22 0.16

5.33 3.48 3.56 5.21 4.06 4.92 1.25

0.19 0.22 0.20 0.20 0.16 0.22 0.12

7-point Likert scale where 1 = “none at all”; 4 = “moderate”; 7 = “very”. b7-point Likert scale where 1 = “not enough”; 4 = “just enough”; 7 = “too much”. cValues indicate changes from beginning to end in terms of participants’ responses on 7-point Likert scale where 1 = “very little”; 4 = “average”; 7 = “great deal”. a

and not by undertaking a comparison of the specific generation options or efficiency level that comprised each strategy. The second method, attribute weighting, asked participants to determine the relative importance of the six objectives used in the energy system model to characterize the performance of the available alternatives. Attribute weights were elicited via a swing-weighting procedure,21 and preferences orders were established offline using an additive utility model.11 Again, the individual generation options and efficiency level that comprised each strategy was not displayed. The sequence in which participants evaluated alternatives was counterbalanced to account for potential order effects. Specifically, it is conceivable that assigning weights to attributes prior to holistic ranking might result in rank orders that are better calibrated (because participants were required to explicitly consider all of the attributes as part of an initial weighting task) than rank orders obtained without an explicit consideration of all attributes. This counterbalancing of evaluation modes comprised the experiment’s second main independent variable (i.e., factor 2; see Figure 1) in the 2 × 2 factorial design. Part 5 of the framework asked all participants a series of 7point, closed-ended self-rating questions. Six questions focused on participants’ self-ratings of the decision support treatment to which they were assigned (Table 3, items 1−6). An additional variable measured participants’ change in knowledge level regarding energy systems (Table 3, item 7). This knowledge variable was composed of 13 questions that asked participants, once at the beginning and once at the end of the study, about the specific fuels, methods of power generation, and conservation and efficiency measures used in the study, MSU’s current method of power generation, and the financial

use the portfolio builder, while participants in Treatments 3 and 4 were not. Part 3 of the framework, which comprised one of the two main independent variables (i.e., factor 1; see Figure 1), allowed participants in treatments 3 and 4 to construct their own future energy strategy for MSU’s campus (in treatments 1 and 2, the portfolio building module was deactivated). Participants could construct a portfolio by combining (up to) five power-plant generation options, (up to) three decentralized energy options, and (up to) two off-site options, as well as by selecting one of four efficiency levels. While assembling their portfolios, participants were provided immediate feedback regarding the degree to which energy demand was being met (all participants were required to meet MSU’s future anticipated energy demand), as well as the degree to which their portfolio was, or was not, meeting the six key objectives outlined above. Each user-generated portfolio was logged for later analysis. Next, in part 4, participants were asked to evaluate six different energy strategies. In treatments 1 and 2, participants evaluated an alternative representing the status quo and five additional strategies developed by the investigators reflecting a broad range of energy generation and efficiency options (Supporting Information, Note S-1). In treatments 3 and 4, participants evaluated an alternative representing the status quo, the portfolio they designed themselves during part 3, and four of the investigator-developed strategies. Participants were asked to evaluate these alternatives using two different methods. One method was holistic ranking, whereby participants simply ordered the six available energy strategies from most to least preferred. Participants ranked strategies by examining only how each strategy performed in terms of achieving the six objectives identified above (Table 2) 1404

dx.doi.org/10.1021/es4036286 | Environ. Sci. Technol. 2014, 48, 1401−1408

Environmental Science & Technology

Policy Analysis

Table 3, Item 5). In addition, t tests conducted on the information variable (Table 3, Item 5) showed participants in treatments 3 and 4 reported receiving “just enough information” on average, i.e., their mean response was not significantly different from 4 on the 7-point Likert scale (t = 1.305, df = 94, p > 0.05), whereas participants who did not build portfolios reported receiving significantly less information (t = −3.158, df = 85, p < 0.01). In terms of the research’s second factor, the order in which participants engaged with the holistic ranking and attribute weighting exercises had no significant effects on any of the self-reported responses. A reliability analysis was conducted to combine the thirteen knowledge-level questions about energy systems discussed above into a single dependent variable (Chronbach’s α = 0.87). A between-subject analysis conducted using this new variable showed that participants who used the portfolio builder (treatments 3 and 4) reported learning significantly more about energy development and delivery than did participants who did not use it (F = 4.00, df = 1, p < 0.05; Table 3, item 7). 3.2. Portfolio Composition, Ranks, and Attribute Weights. Across both portfolio-building treatments (3 and 4), nuclear power was the preferred generation option, having been selected by 75% of participants and providing 20.5% of total energy output, as computed by taking the average of all participants’ selections. Distributed natural gas was the next most preferred generation option, selected by 38.5% of participants and comprising 9.5% of energy output. Purchasing electricity from the conventional grid was the least preferred energy option; it was selected by only 4.2% of respondents and accounted for less than 1% of total energy output (Table 1). When asked to rank portfolios containing these fuels and energy types, participants consistently ranked portfolios containing nuclear power highest and portfolios relying on coal and conventional natural gas lowest. Across all treatments, the most expensive, yet best performing, portfolios with respect to GHG and air emission reductions (Table 2, options 3 and 5) were consistently ranked highest, while the least expensive and highest emitting portfolios (Table 2; options 1, 2, and 6) ranked lowest. Option 3, a portfolio that relied primarily on nuclear power, was ranked highest (xr̅ ank = 2.28, SE = 0.10), while option 1, the status quo, ranked lowest (xr̅ ank = 5.07, SE = 0.11). On average, participants’ self-constructed portfolios ranked second highest (xr̅ ank = 2.86, SE = 0.17). Despite the marked contrast between cost, GHG, and air particulate emissions’ effect on participants’ ranks, participants consistently weighted these three performance measures similarly in the swing-weighting task. Reducing GHG emissions received the highest weight in treatments 2 and 3 (xw̅ eight = 0.85, SE = 0.03; xw̅ eight = 0.85, SE = 0.03, respectively), whereas minimizing cost received the highest weight in treatments 1 and 4 (xw̅ eight = 0.82, SE = 0.04; xw̅ eight = 0.84, SE = 0.03, respectively). Reducing particulate air emissions was consistently weighted third (xw̅ eight = 0.78, SE = 0.02). The mean weights of minimizing cost, reducing GHGs, and reducing air particulate emissions were not found to be significantly different (t tests, p > 0.05). Multivariate analysis of variance showed that the order in which the ranking and weighting exercises were presented to the participants did have a significant effect on participants’ attribute weights (λ = 0.91, F = 2.8, p = 0.01); whether or not participants engaged with the portfolio builder however had no significant effect on elicited weights.

and environmental costs associated with each (Supporting Information, Table S-1). The framework concluded with part 6, which provided participants an opportunity to indicate their final preferences for energy strategy alternatives. Of primary interest was participants’ preferences for the strategies they either ranked as most preferred or built themselves and the best-fit strategies determined on the basis of attribute weights. To this end, participants in treatments 1 and 2 were asked to indicate a preference for either their highest ranked strategy (from part 4) or the best-fit strategy implied by their attribute weights (also from part 4). Participants in treatments 3 and 4, in contrast, were asked to indicate a preference for either the strategy they built (in part 3) or the best-fit strategy implied by their attribute weights (from part 4). 2.4. Subjects. In total, 182 randomly selected juniors and seniors took part in this research split across treatments 1 (n = 47), 2 (n = 49), 3 (n = 45), and 4 (n = 41). Rather than relying upon student subjects simply as a convenience sample, this subject pool was selected for two reasons. First, administrators at MSU had identified students as critical stakeholders in decisions about the university’s energy strategy (because they lived and studied in close proximity to MSU’s existing cogeneration facility and because some of the costs associated with significant energy-related infrastructure upgrades could potentially be recovered from the tuition and fees paid by students and their families). Second, MSU students were already being actively consulted about the university’s desire to rethink its energy strategy. In parallel with this study, a broader survey of stakeholders’ opinions using this decision support tool was undertaken. However, this broader sample received a standardized (i.e., absent the experimental design reported here) version of the decision support tool. Thus, their responses will not be reported here. The research took place on the MSU campus. Participants who constructed their own portfolios (T3 and T4) required 45−60 min to complete the framework, whereas participants who did not build portfolios (T1 and T2) required 40−45 min. All participants were paid $30 for their time. 2.5. Data Analysis. Univariate (ANOVA) and multivariate (MANOVA) analyses of variance were used to analyze participants’ self-reported evaluations and learning about energy systems. Descriptive statistics were used to summarize and compare portfolio-building outputs, rank orders and attribute weights, and the participants’ final choice.

3. RESULTS 3.1. Participants’ Self-Reports. Across all treatments, participants reported high levels of satisfaction with their decisions, a high degree of internal consistency (i.e., accuracy), and a high level of comfort with their input informing actual energy strategy development at MSU (see Table 3). Participants also reported low levels of stress and mental difficulty associated with completing the decision-making tasks. A between-subject comparison conducted on mean responses showed a significant effect of the portfolio-building module (MANOVA, λ = 0.86, F = 3.00, p < 0.01). Specifically, participants in treatments 3 and 4 reported their decisions to be significantly less accurate with respect to their values and concerns than did participants who did not use the portfoliobuilding module (p < 0.01; Table 3, Item 4); participants who built portfolios also reported having been provided with more information to thoughtfully make those decisions (p < 0.01; 1405

dx.doi.org/10.1021/es4036286 | Environ. Sci. Technol. 2014, 48, 1401−1408

Environmental Science & Technology

Policy Analysis

Figure 2. Comparison of mean preference orders for energy strategies by treatment (T1−T4) when established by holistic ranks as opposed to implied by attribute weights. Axis labels refer to the relative position of strategies according to holistic ranking or attribute weighting and not to specific strategies. For example, in treatment 1 (T1), the alternative ranked third (denoted by ↓) is in position 1 when preference orders are based on attribute weights. If respondents’ ranks and implied preference orders based on attribute weights were perfectly calibrated, each option would fall on the 45-degree line.

ensure that the objectives used in this research were representative of the target sample and the performance measures were explicitly explained to participants. This emphasis on transforming complex and technical aspects into terms that link directly to stakeholder values was, in our view, key in making the trade-off analysis exercise relevant and the overall experience useful for participants.11 Participants’ self-reports also suggest that SDM and portfolio analysis combined effectively in stakeholder-driven energystrategy development. Individuals on average reported little stress or difficulty in terms of engaging with the framework, and those individuals who constructed portfolios, which on average required more time (up to 15 min) and effort spent confronting trade-offs, reported no greater stress or difficulty than did those participants who merely compared predetermined options. However, importantly, participants who built portfolios felt they had been provided with significantly more and yet also just the right amount of information to make their decisions. In contrast, participants who did not build portfolios reported desiring more information. Despite reporting high levels of satisfaction and low levels of stress, individuals who constructed their own portfolios reported their decisions to be significantly less accurate with regards to their values and concerns than did participants in the non portfolio-building treatments. This is a curious result in that on the basis of prior research13,17 internal consistency was predicted to be highest in conditions where an explicit focus on objectives was combined with the opportunity to develop bespoke energy strategies. The likely explanation for this disparity is that designing bespoke energy strategies encourages counterfactual thinking among participants. The act of building a custom strategy alerts people to the possibility of other strategies that they may also find suitable, and when evaluating their custom alternatives, people are likely reminded of the other alternatives that they could have built. This may cause discomfort and, in turn, a decrease in self-reported internal consistency. Previous research by Wilson and colleagues24,25 supports this explanation. Their findings show that more time spent thinking about the reasons to make (or not make) certain decisions, as well as the act of evaluating options, leads to an increase in the propensity for postdecision dissatisfaction and regret. It is important to point out that there may be differences in terms of the degree to which users of this decision support framework think their decisions are internally consistent (via the self-reports discussed above) and the extent to which their decisions are internally consistent (as measured by behavior). To study this, participants’ rank orders were compared with the preference order implied by attribute weights as determined by an additive utility model (part 4 in our experimental design; see

Using an additive utility model, portfolio rank orders were computed for each participant based on their attribute weights and mean rank orders were computed for each treatment. Figure 2 compares the preference orders for each strategy option across all 4 treatments, both those established by holistic ranking and those implied by attribute weights. When computed using the additive value model, Option 5 (Table 2) was found to be the top-performing alternative in 153 out of a possible 182 comparisons (84%) across all 4 treatments. Option 1, the status quo, was the worst performing option by a similar margin (in 83% of cases across all 4 treatments). During holistic ranking, user-generated portfolios were ranked in the first or second position on 25 (26%) and 22 (23%) occasions (from 96 total observations), respectively. In sharp contrast, participants’ self-generated portfolios were found to be the top-performing alternative via the additive utility model on only 11 (12%) occasions; these portfolios were found to be the lowest performing portfolio via the additive utility model on 17 (19%) occasions. 3.3. Final Preferences. In part 6, which asked participants to select between either (a) the portfolio they ranked highest in treatments 1 and 2 (n = 85) or (b) the portfolio they constructed themselves in treatments 3 and 4 (n = 93) and the best performing portfolio according to the additive utility model, the majority of participants in treatments 1 and 2 (83%) indicated a preference for their top-ranked portfolio over the best-fit (i.e., most internally consistent according to attribute weights) portfolio. Likewise, a majority of participants in treatments 3 and 4 (60%) indicated a preference for their selfgenerated portfolio over the best-fit option. ANOVA revealed that the order in which the ranking and weighting tasks were presented did not have a significant effect on participants’ final choices in any treatments (treatments 1 and 2: F = 0.05, df = 1, p > 0.05); treatments 3 and 4: F = 3.02, df = 1, p > 0.05).

4. DISCUSSION The primary objective of this research was to develop and test an interactive decision support framework that would provide stakeholders and decision makers an opportunity to thoughtfully construct and compare energy strategy options in a realworld setting. Three approaches were used to evaluate the framework: participants’ self-reports, the amount and type of knowledge participants gained, and the overall decision quality as a function of the internal consistency of participants’ decisions. Overall, individuals across all four treatments reported high levels of satisfaction and comfort with their decisions. These kinds of positive reports are relatively common in the literature on decision-aiding13,17,22 and have been linked to the explicit values-focus approach of SDM.23 Significant effort was taken to 1406

dx.doi.org/10.1021/es4036286 | Environ. Sci. Technol. 2014, 48, 1401−1408

Environmental Science & Technology

Policy Analysis

shown that participating in a structured, values-focused approach leads to higher levels of self-reported knowledge than does taking part in a more conventional, unstructured, and technically focused approach to decision-making.17 When people must actively draw connections between their objectives and information presented about the available alternatives, the information takes on more relevance. This is in contrast to conditions where people engage in more conventional, passive decision-making approaches and are able to focus on salient attributes without fully understanding the range of implications associated with a particular choice.2,13,17 In the research reported here, we suspect that the relatively large increases in self-reported knowledge in treatments 3 and 4 could be attributed to the phenomenon outlined above; however, we also believe on the basis of debriefing sessions with some participants that developing alternatives caused people to interact on a deeper level with the information presented. This would suggest that making stakeholders more active members in the decision-making process, as opposed to simply passive recipients of information, is key in future energy strategy development, as well as risk communication.27,28 While testing the decision-making framework described here was central to our research, the specific energy strategies constructed by and the preference orders gathered from the student population were also critical in determining the direction of MSU’s sustainability initiative. It is important to note that at no time were renewable or low-emission energy technologies encouraged or identified as superior to other options (i.e., as part of the framework itself or by the facilitator present during the sessions). In spite of this, low or noemission sources of energy dominated the majority of constructed and selected strategies. Despite the “cost” attribute being the first one identified in the primer, in the portfolio builder, and in the holistic ranking and attribute weighting tasks, and thus a possible anchor, an analysis of the most popular forms of power generation showed that the attribute “emissions reduction” was most important to participants. The top six, and nine of the top 11, energy sources, those sources selected by at least 20% of participants (see Table 1), were no or low-emitting technologies. These technologies vary considerably in cost, as well as in the amount of jobs created and land necessary for adoption, and yet are consistent in their lack of CO2 and air particulate emissions. While consensus is difficult to achieve, one may be said to exist here, as attribute weighting also identified Portfolio 5, the best performing portfolio with regards to emissions reduction (see Table 2), as the best-fit option for 83% of participants. Certainly, this result is not entirely divorced from the fact that the subject pool was composed of university students who likely have different priorities, concerning the development of a regional energy strategy, than does a representative sample of the adult population. This possible dichotomy as well as how effective the framework is when deployed entirely online and without supervision is currently being examined in a follow-up study. Such research is critical in determining the capabilities of this framework when deployed on a wider (e.g., state or national) scale and when in-person facilitation is not possible. While project sponsors and study participants agreed that this decision support framework was innovative and an improvement over existing approaches, determining the extent to which it can quickly and directly acquire reliable information from a large number of stakeholders still needs to be investigated.

Figure 1). Here, our results suggest that both the process of portfolio construction and attribute weighting play important roles in terms of leading to internally consistent choices. Specifically, Figure 2 shows that building portfolios and engaging in weighting first, followed by ranking, leads to better calibrated, i.e., more internally consistent, choices, as determined by the best-fit slope of the line drawn through the coordinates obtained for ranks and weights in each treatment. Decisions would be perfectly calibrated if rank orders and implied preference orders based on weights were identical; i.e., a best-fit slope of 1. In this case, treatment 1, which did not involve portfolio construction and asked participants to rank portfolios before assigning attribute weights, had the lowest best-fit slope (0.49). Treatment 4, which asked participants to build portfolios and assign attributes weights first (before ranking), had the highest slope (0.90). These results, along with the fact that the best-fit slopes of treatments 2 and 3 (0.78 and 0.80, respectively) fall somewhere in between, suggest that the combination of portfolio construction and weighting attributes first leads to highly internally consistent choices. The second method used to evaluate overall decision quality was the choice offered to respondents between two portfolios: either the portfolio they constructed in treatments 3 and 4 or the portfolio they ranked highest in treatments 1 and 2 and the best-fit portfolio determined by the respondents’ attribute weights. Were respondents’ decisions perfectly calibrated, these portfolios would be the same. However, only 18% of individuals (15 out of 85) ranked their best-fit portfolio highest, and only 12% of individuals (11 out of 95) were responsible for constructing their own best-fit portfolio. This final choice was provided to ensure that participants did not rely too heavily, or anchor, on either the portfolio they constructed or the portfolio ranked highest when subsequently presented with a portfolio that better addressed their preferred objectives. Were respondents making internally consistent decisions, all 154 (minus the 26 individuals who actually built their own best-fit portfolio or ranked it highest) would have selected the best-fit option, yet overwhelmingly, participants chose their own constructed (60%) or highest ranked portfolio (83%). A possible explanation for this observation exists in the fact that across all four treatments individuals were not instructed that the second portfolio shown in this task was a “best-fit” portfolio. They were merely provided a choice between the portfolio they constructed or ranked highest and an “other portfolio.” By this point, individuals had invested considerable time and effort in either constructing their own portfolio or comparing portfolios in the holistic ranking exercises. Previous research has demonstrated that such investments often lead to a status-quo bias in peoples’ decision-making or the propensity of individuals to select or stick with a default option.26 Investigating the degree to which people anchor to the portfolios they constructed or investigated previously, specifically by presenting them another option that is clearly identified as their own “best-fit” option, is a critical task and one already underway in a follow-up study. While it is true that participants in treatments 3 and 4 reported lower levels of internal consistency, participants across all treatments reported significant increases in knowledge. Notably, participants in treatments 3 and 4 reported significantly higher levels of knowledge following the study than did those participants who did not use the portfolio builder. Previous research on decision-aiding methods has 1407

dx.doi.org/10.1021/es4036286 | Environ. Sci. Technol. 2014, 48, 1401−1408

Environmental Science & Technology

Policy Analysis

(10) Keeney, R. L. Value focused brainstorming. Decis. Anal. 2012, 9, 303−313. (11) Gregory, R.; et al. Structured Decision Making: A practical guide to environmental management choices; Wiley-Blackwell: West Sussex, UK, 2012. (12) Kahneman, D. Thinking, fast and slow; Farrar, Straus and Giroux: New York, 2011. (13) Arvai, J. L.; Gregory, R. Testing alternative decision approaches for identifying cleanup priorities at contaminated sites. Environ. Sci. Technol. 2003, 37, 1469−1476. (14) Wilson, C.; McDaniels, T.L. Structured decision-making to link climate change and sustainable development. Climate Policy 2007, 7, 353−370. (15) Gregory, R.; Wellman, K. Bringing stakeholder values into environmental policy choices: A community-based estuary case study. Ecol. Economics 2001, 39 (1), 37−52. (16) McDaniels, T. L.; Trousdale, W. Value-focused thinking in a difficult context: Planning tourism for Guimaras, Phillipines. Interfaces 1999, 29, 58−70. (17) Wilson, R. S.; Arvai, J.L. Evaluating the quality of structured environmental management decisions. Environ. Sci. Technol. 2006, 40 (16), 4831−4837. (18) National Research Council. Understanding risk: Informing decisions in a democratic society; National Academy Press: Washington, DC, 1996. (19) Carley, S. Decarbonization of the U.S. electricity sector: Are state energy policy portfolios the solution? Energy Economics 2011, 33 (5), 1004−1023. (20) Palmgren, C.; et al. Initial public perceptions of deep geological and oceanic disposal of carbon dioxide. Environ. Sci. Technol. 2004, 38 (24), 6441−6450. (21) Clemen, R. T. Making Hard Decisions: An Introduction to Decision Analysis; PWS-Kent Publishing Co.: Boston, MA, 2004. (22) Arvai, J. L.; Gregory, R.; McDaniels, T.L. Testing a structured decision approach: Value-focused thinking for deliberative risk communication. Risk Anal. 2001, 21 (6), 1065−1076. (23) Keeney, R. L. Value-focused thinking. A path to creative decision making; Harvard University Press: Cambridge, MA, 1992. (24) Wilson, T. D.; et al. Introspecting about reasons can reduce post-choice satisfaction. Pers. Soc. Psychol. Bull. 1993, 19, 331−339. (25) Wilson, T. D.; Schooler, J.W. Thinking too much: Introspection can reduce the quality of preferences and decisions. J. Pers. Soc. Psychol. 1991, 60 (2), 181−192. (26) Thaler, R. H.; Sunstein, C. R. Nudge: Improving Decisions about Health, Wealth, and Happiness; Penguin Group: New York, 2008. (27) Arvai, J. L., Rivers, L., III, Eds. Effective Risk Communication; Routledge: Oxford, 2013. (28) Fischoff, B. Risk perception and communication unplugged: Twenty years of process. Risk Analysis 1995, 15, 137−145.

In spite of the new questions raised by this research, the results reported here suggest that our hybrid decision-support framework leads to internally consistent choices while also leading to significant increases in learning and knowledge. More importantly, by coupling stakeholder engagement with a defensible, evidence-based decision-support approach, this hybrid framework could serve as a template for meaningf ully engaging a wide range of stakeholders in important decisions about their region’s energy future. Indeed, it is through interlinked regional energy strategies, each one adapted to localized opportunities and constraints, that we believe the vision of a national energy strategy may be achieved.

■

ASSOCIATED CONTENT

* Supporting Information S

Additional figures and tables including the description of each energy generation option as presented to participants, the specific generation options that comprised each investigatorgenerated strategy, the pre- and postexperiment self-rating survey, and a screenshot of the energy system model (i.e., the energy portfolio builder). This information is available free of charge via the Internet at http://pubs.acs.org.

■

AUTHOR INFORMATION

Corresponding Author

*E-mail: [email protected]; phone: 587-894-6695; fax: 403282-6561. Notes

The authors declare no competing financial interest.

■

ACKNOWLEDGMENTS This research was funded by Carbon Management Canada, the Canada School for Energy and Environment (under a Proof of Principal Grant), and the Office of the Vice President for Finance and Operations at Michigan State University. We are grateful to Jennifer Battle, Lynda Boomer, Richard Grogan, Kathy Lindahl, Graham Long, Lauren Olson, Robyn Wilson, and Tim Wilson for their assistance and insights throughout the research process.

■

REFERENCES

(1) Executive Office of the President. The President’s Climate Action Plan; The White House: Washington, DC, 2013. (2) Arvai, J. L.; et al. Decision support for developing energy strategies. Issues Sci. Technol. 2012, 28 (4), 43−52. (3) Nemet, G. F.; Kammen, D. M. U.S. energy research and development: Declining investment, increasing need, and the feasibility of expansion. Energy Policy 2007, 35, 746−755. (4) Dixon, R. K.; et al. U.S. energy convservation and efficiency policies: Challenges and opportunities. Energy Policy 2010, 38, 6398− 6408. (5) Fleishman, L. A.; Bruine de Bruin, W.; Morgan, M. G. Informed public preferences for electricity portfolios with CCS and other lowcarbon technologies. Risk Anal. 2010, 30 (9), 1399−1410. (6) Parkhill, K. A.; et al. Transforming the UK energy system: Public values, attitudes and acceptability - Synthesis report; UKERC: London, 2013. (7) National Research Council. America’s Energy Future: Technology and Transformation; National Academies Press: Washington DC, 2009. (8) Keeney, R. L.; Raiffa, H. Decisions with Multiple Objectives: Preferences and Value Tradeoffs; Cambridge University Press: Cambridge, UK, 1993. (9) Howard, R. A. Decision analysis: Practice and promise. Manage. Sci. 1988, 34 (6), 679−695. 1408

dx.doi.org/10.1021/es4036286 | Environ. Sci. Technol. 2014, 48, 1401−1408