Lee Merkhofer Consulting Priority Systems
Choosing the Wrong Portfolio of Projects

"Whether MODA is right for you depends on how important choosing the right projects is and on whether your organization can accept the discipline required to follow a rigorous prioritization process."

A Project Prioritization Example

The following example illustrates how multi-objective decision analysis (MODA) may be used to construct a project selection decision model and how the model may be applied to obtain a priority listing of projects. The example is a government application. Unlike private sector applications whose descriptions are usually restricted by confidentiality agreements, government applications, including this one, can often be described in detail.

Things to notice:

  1. Influence diagrams can be extremely useful for identifying and understanding the considerations important for making decisions.
  2. Dominated alternatives may be eliminated
  3. Though additive value models are the norm, MODA may demonstrate that a non-additive model is appropriate.
  4. Recognizing that some stakeholders distrust "expert estimates," an analysis with stakeholder participation will be more likely to persuade stakeholders if those stakeholders are advised of the estimates generated by experts but given the option to use their own estimates instead.
  5. Despite initially holding diverse views, participating in an exercise to create a prioritization model often results in the opinions of participants becoming more closely aligned.
  6. Sensitivity analysis often generates insights about the considerations that drive priorities.

Problem Description

Too much traffic

In the late 1990's, the question of how to alleviate severe traffic congestion along Oregon's Highway 99W was gaining urgency. Highway 99W is the principal connection between the Portland metropolitan area and the Oregon seacoast, a recreational destination and the center of the State's commercial fishing industry. Although it is a State-designated "level of importance highway and freight route," highway 99W served as Main Street for wine-tasting destination towns Newberg and Dundee. Crash rates on the highway exceeded the statewide average, and congestion was increasingly regarded as an impediment to the timely provision of police and fire services [1].

Many citizens and some local interest groups were lobbying for additional freeway segments to route traffic around the towns with the greatest congestion. Other groups, including anti-growth and anti-highway interests, were fighting any new highway capacity that would displace existing businesses and agricultural use and potentially turn impacted communities into "bedrooms" for Portland workers.

Newspaper article

The alternatives being promoted ranged from doing nothing to providing increased bus service, building a new rail system, and constructing freeway bypasses. Federal regulations require that any such changes be subject to an Environmental Impact Statement (EIS). Accordingly, the objective for the prioritization effort described here was to identify a small set of preferred alternatives, hopefully with stakeholder support, suitable for assessment according to the EIS process. To this end, the Oregon Department of Transportation (ODOT) funded this application of MODA [2].

Decision Makers

Creating a successful model for prioritizing projects requires engaging decision makers in the model construction process. Real decision makers are the best source of the value inputs the model needs if it is to be able to mimic what real decision makers would choose if they could spend the time and had the ability to analyze choices in a fully rational and unbiased way.

Team structure

Figure 26:   Participating Teams and Roles.

For the purposes of this effort, the decision makers were assumed to be a Project Advisory Committee established by ODOT. The Committee was composed of 25 members selected to represent the affected communities, special interest groups, and citizens. Committee membership encompassed the full range of opinions about which alternatives should be chosen.

In reality, the real decision makers were not Committee members. To ensure involvement from real decision makers, a 10-member Project Oversight and Steering Team (POST) was established. The POST was composed of elected officials from the impacted jurisdictions plus senior managers from ODOT and other transportation-related state and federal agencies with statutory authority to choose alternatives for the EIS, make final decisions, and secure necessary funding. The POST was regularly briefed on the status of the effort. Briefings to the POST were important because the represented organizations would need to be in agreement over the alternatives that should move forward according to the decision making process.

The Analysis

The concept for the analysis was to engage Committee members in a group process to construct a model for prioritizing alternatives for solving the Highway 99W traffic problem. Experience shows that if participants create and then provide the inputs to a model for prioritizing projects they nearly always agree with the priorities produced by the model, provided, of course, that the participants believe the model to be sound.

Duration and Level of Effort

Analyses with public participation always take much longer than would otherwise be the case. This exercise was conducted over a nine-month period, but the prioritization model was constructed and applied in just four meetings of the Committee, with between-meeting analyses provided by a Technical Team composed of ODOT staff and consultants. Much of the time spent between Committee meetings was devoted to estimating health, safety, environmental, and transportation system outcomes, effort that is typically not needed for prioritizing projects conducted by private sector organizations. The budget for the effort was "significantly lower than budgets for comparable previous studies" [2].

Estimating Likelihood of Implementation

One additional team was established to support the effort, an Agency Advisory Committee. The Agency Advisory Committee was composed of federal and state resource agency specialists knowledgeable about regulatory compliance criteria. The main role of the Agency Advisory Committee was to provide judgments regarding the likelihood that proposed solutions could obtain the permits needed to be implemented. Figure 26 above summarizes the roles of the participating teams.

Objectives Hierarchy

At the first Committee meeting, members selected and agreed to a set of MODA-compliant objectives. Figure 27 displays the objectives as an objectives hierarchy. As illustrated, the objectives included satisfying the transportation needs of various categories of travelers; protecting public health, safety, and the environment; protecting community socio-economic quality; and minimizing costs. These eleven lowest-level objectives were accepted by the Committee as defining the types of value expected from a successful solution to the traffic problem (local travelers transportation value, public health and safety value, environmental value, and so forth). One additional objective, "maximize the likelihood of implementation success," was added in recognition that any solution recommended by the Committee would need to garner sufficient support to obtain regulatory approvals and funding, an outcome that was far from being a sure thing.


Objectives hierarchy for selecting transportation system upgrades

Figure 27:   Objectives hierarchy adopted by Project Advisory Committee.


Alternatives

Following the initial Committee meeting, the Technical Team created eight alternatives for evaluation. Each alternative consisted of a package of multi-modal components selected to complement a primary transportation mode (highway, bus, or rail) plus complementary elements including transportation system management, transportation demand management, roadway improvements, and land use policies. The goal for the development of alternatives was to span the range of possibilities while providing a complete package of activities designed to maximize the chance of meeting Oregon's strict transportation and land use regulations.

Newspaper article

The alternatives put forth by the Technical Team were: (1) No Action (the base alternative), (2) Transportation Management featuring a high level of express bus service, (3) Widening the Existing Highway, (4, 5, 6) three Highway Bypass alternatives in different land use corridors, (7) Interurban (heavy) Rail, and (8) Light Rail.

Information was obtained, analyses conducted, and profiles prepared for each alternative, including obtaining rough estimates of transit service levels (e.g., distances between transit stations and park-and-ride lots), cost per mile of construction, equipment costs, and maximum daily passenger throughput.

Influence Diagrams

Concurrent with the definition of alternatives, the Technical Team created influence diagrams to identify the factors that determine the degree to which each objective is achieved. As argued previously, developing influence diagrams is relatively easy and very effective for improving understanding. Figure 28 provides, as an example, the influence diagram constructed for the objective, "satisfy the needs of travelers" (this same diagram applies to all five traveler types identified in the objectives hierarchy).


Influence diagram for travelers' satisfaction

Figure 28:   Influence diagram for travelers' satisfaction.


Influence Diagram "Drivers"

The colors in the diagram show that travelers' satisfaction was considered in three distinct time periods: during normal system operation, during the initial construction period for the alternative, and during disruptive incidents, such as when accidents occur. An asterisk in a bubble indicates that the Technical Team designated the factor to be a "driver;" that is, a factor judged dependent on the selected alternative and particularly important for determining the level of achievement for the objective. As shown by the asterisks in the upper tier of this diagram, performance during normal system operation and during incidents were judged more important with regard to estimating achievement of the objective than performance during the alternative's initial construction period.

The additional asterisks in the influence diagram show that the Technical Team believed that, with regard to normal system operation, the drivers were: (a) the percent of travel demand satisfied, (b) the numbers of the representative origin/destination (o/d) trips taken by mode, and (c) the judged quality of those trips. The drivers for trip quality were (a) the estimated total number of hours spent in transit, and (b) the estimated quality of each of those hours, the latter of which depends on the comfort and stress level of the trips and whether or not the mode of travel frees travelers to do other things, such as reading. Note also that the number of hours spent traveling in hours of peak-period congestion was identified as a driver for estimating trip quality.

Newspaper article

Similar to the travelers influence diagram, the influence diagrams for the other objectives identified factors that should be considered when assessing performance. The drivers for health and safety were mainly factors that influence the degree to which the selected alternative would improve traffic flow, thereby reducing the frequency of congestion-related accidents and allowing ambulances and other safety vehicles to arrive where needed more quickly. Also influencing public health were risks from emissions of toxic air pollutants (e.g, exhaust from cars, buses, and trains) and other exposures. The factors influencing the environment related mainly to the disruption or loss of habitat for valued plant and animal species. The main factors for community economics were loss of land currently used for business and agriculture. The main factors for socio quality were noise and visual aesthetic degradation, community fragmentation from new highway or railroad facilities and induced population growth. The cost factors included both the costs paid by taxpayers and any costs paid by users due to new tolls or fares.

Committee Approval of Alternatives and Influence Diagrams

At the second Committee meeting the influence diagrams and alternatives for evaluation were presented. The Team noted that one of the alternatives, light rail, appeared to be dominated by the heavy rail option. In other words, the benefits of the two alternative rail systems were estimated to be very similar, while the costs of light rail were estimated to be significantly higher. With the agreement of the Committee, the light rail option was eliminated from further consideration.

Committee members approved the remaining seven alternatives and recommended only minor changes to the definitions and profiles for each. With regard to the influence diagrams, only one Committee member had previous experience with the diagrams. However, it was apparent that all of the members quickly learned the meaning of the bubbles and arrows and found the diagrams easy to interpret and helpful {2}.

Mathematical Form of the Aggregation Equation

Determining the mathematical form of an equation for translating performance estimates into a measure of project value is a critical step in the design of any priority system. As noted on the previous page, if performance measures are mutually preferentially independent, the aggregation equation will be additive. The reason for seeking an additive equation, or a simple variation of an additive equation, is that the components of the equation will, in this case, be easy to derive. Performance measures will most likely be preferentially independent if the objectives are fundamental objectives [3].

By inspection, it is apparent that the objectives in the lowest-level of the objectives hierarchy are fundamental objectives, with the possible exception of the travelers' satisfaction objectives. To verify the suitability of an additive equation, the test for preferential independence was explained to Committee members. In order to conduct the test. hypothetical swings in travelers' satisfaction were posed ,and Committee members where asked whether or not the values of those swings depends on the levels of performance assumed for other objectives. In all cases, Committee members expressed the opinion that the increments in value from the swings were, at least approximately, independent of the levels of performance assumed for the remaining objectives. In contrast, the value of a swing in the probability of implementation clearly depends on the value the alternative would produce if implemented. In fact, the value of an increase to the probability of implementation is proportional to the value of the alternative assuming it is implemented. Accordingly, it was concluded that an aggregation equation of the following mathematical form would be appropriate:


V(x1,x2...x11, Prob12) = Prob12 × [w1V1(x1) + w2V2(x2) + ... + w11V11(x11)]


V(x1,x2...x11, Prob12) is the computed value of an alternative having estimated performance levels x1,x2...x11 and a probability of implementation equal to Prob12. (Note that value was not expressed in dollar units, although scaling to dollars was possible. Instead, estimates of the attractiveness of the alternative traffic solutions were expressed in units of relative value.) The Vi are single attribute value functions, and the wi are weights.

Technical Estimates

Following the second Committee meeting, the Technical Team created a data set for supporting the assessments of performance for each alternative. Quantitative or qualitative estimates were generated for each of the drivers in the influence diagrams plus estimates for other factors deemed useful, to the extent that the Team felt that such estimates could be reasonable generated. The table below lists the factors relevant to each objective for which estimates were developed. For example, under one of the bypass alternatives, it was estimated that existing traffic volumes through downtown Newberg would decrease by about 20% and by 40% through downtown Dundee, with even greater reductions in the volume of freight traffic. The estimates were obtained using available travel models; health, safety and environmental impact models; and best professional judgment. Where quantitative estimates could not be generated, qualitative judgments were provided.


Factors estimated by the Technical Team and provided to the Committee

Figure 29:   Factors estimated by the Technical Team to Aid Assessments of Performance.


Estimates of the Degree to Which Transportation Needs Would be Met

To generate performance estimates for each of the proposed alternatives, agreement was first reached on appropriate 20-year growth projections for populations within the study region under the base-case, do nothing alternative. The population scenarios were used to obtain traffic demand forecasts using, as noted above, transportation system models configured to the study region. To assess the adequacy of the transportation system, representative origin-to-destination (o/d) trips were defined that the selected transportation alternative would be expected to serve. The more of these representative trips that were conducted under the alternative (summed across all transportation modes), the more effective the solution was assumed to be at serving transportation demand.

Estimates of Impacts on Land Use

Impacts on land use were estimated by extrapolating current patterns based on qualitative assumptions. New highway or rail facilities were assumed to require kilometer-wide strips of land corridors, while corridors for expanding existing highway and rail facilities were more narrow, conforming to the existing footprints of those facilities. Impacts on resources of various types (industrial land, habitat, wetlands, etc.) under each alternative were estimated by multiplying the number of hectares of the subject resource within the footprint of the alternative's corridor times the percentage of total hectares in the corridor that would be included in the right-of-way for the proposed improvements. GIS applications were used for coverage mapping and calculation of hectares.

Estimates of Induced Population Growth

The potential for new bypass facilities to induce population growth was controversial. No commonly accepted methods exist for predicting induced population growth resulting from improvements to regional transportation systems. To provide a rough estimate, Team members estimated the degree to which the alternatives would increase the feasibility for workers to commute to and from Portland. Specifically, the Team calculated the number of individuals currently living in the subject area whose potential commute times to the Portland metropolitan area is above a threshold deemed too long to make commuting practical. Then, the number of those individuals who, under each alternative, would have their potential commute times reduced to a time below the threshold was computed. This number was assumed to represent the potential for induced population growth.

The Prioritization Process

The usual approach for completing a MODA-based, project selection model is to specify measures for quantifying the degree to which each alternative would achieve each of the model's objectives. Then, scoring scales, models, or related methods are used to obtain the performance estimates for each alternative. To this point of the analysis, the Technical Team had provided estimates for each of the bulleted factors in the table in Figure 29. These factors, as indicated above, are factors believed by the Technical Team to be relevant to assessing the degree to which each objective would be met. Therefore, one approach for completing the remaining step, assuming the usual approach, would be to specify equations for combining each of the factors corresponding to each of the objectives into a measure of performance for that objective. For example, to measure performance relative to the objective "satisfy the travel needs of locals," an equation might be provided for combining (1) the number of weekday and weekend vehicle trips on 99W, (2) the number of hours of weekday and weekend congestion on 99W, (3) the number of hours of congested conditions on all roads in the study area, and (4) a measure of the quality of the time spent traveling. Such equations, if they could be provided, could be viewed as sub-models for quantifying the performance outcomes for the alternatives.

In this application, in contrast to the usual approach, the evaluation process allowed Committee members to form their own opinions about the degree to which each alternative would achieve each objective. In other words, the Technical Team explained their estimates for the factors in the Figure 29 table, but allowed scorers to provide their own judgments about the levels of performance under each alternative. This approach, wherein technical estimates of performance are provided but can be accepted or rejected by those designated to provide "scores," is useful for situations wherein, for whatever reason, scorers might not believe or find convincing the technical estimates that are provided to them.

Committee Approval of the Plan for Evaluating Alternatives

Following the presentation of the Team's estimates at the third Committee meeting, the proposed plan for the comparative evaluation of alternatives was described. Figure 30 summarizes the evaluation process as proposed to the Committee. Committee members would estimate the performance that each alternative would achieve relative to each of the eleven lowest-level objectives using 1-to-10 scoring scales. The probability of successfully implementing each alternative would also be estimated by the Committee members, with suggestions provided by the Agency Advisory Committee. Committee members would provide a weight for each objective and the weights and scores would be multiplied to produce the estimate of the alternative's contingent value assuming it could be implemented. The probability of successful implementation would then be multiplied by the alternative's computed contingent value to produce an expected value for each alternative. The expected values would be used to rank the alternatives.


Method for analyzing alternatives

Figure 30:   Method for Computing the Relative Value of Each Alternative.


Technical Team Weights and Scores

Between the third and final Committee meetings, the Technical Team tested the proposed evaluation process and generated a set of weights and scores representing the thinking of the Technical Team. The test helped to guide planning and the creation of materials (e.g., scoring forms, supporting information) for the process to be used for the Committee. As an aside, I strongly recommend that the project portfolio management (ppm) team test scoring and weighting procedures prior to using them to obtain weights from executives and scores from technical personnel.

Elicitation of Weights from the Committee

At the start of the final Committee meeting, for which an entire day was allocated, Committee members began the process of assigning weights to the eleven lowest-level objectives in the objectives hierarchy. A weight assessment method known as point allocation with an emphasis on swing weighting was used. Each participant allocated 100 points among the objectives to indicate the judged relative value of achieving (or avoiding) specified changes (swings) to the achievement of the objectives. The weight assignment process incorporated elements of the nominal group technique. Weights were assigned in two rounds, with a chart presented between the rounds showing the distribution of individuals' weights. Committee members were also shown the weights assigned by the Technical Team. Participants expressed surprise at the similarity between the round one Committee weights and the weights assigned by the Technical Team. The only difference of note was that Committee members tended to weight avoiding damage to the environment, community economics and socioeconomic quality higher, and satisfaction of travel needs lower, than did the Technical Team. As might be expected, the comparisons of weights provided by different Committee members and between the Committee and the Technical Team generated spirited discussion, especially among the Committee members who assigned the highest and lowest weights to each objective. These discussions prompted some individuals to change their weights in the second round, although most weight changes were modest.

Committee Scoring

Following the assignment of weights, Committee members scored each alternative using 1-to-10 scales as described above. The Committee was advised to interpret a score of "1" to mean "very poor performance, comparable to the worst performance occurring anywhere in the region's transportation system." For example, with regard to environmental performance, a score of "1" was interpreted as meaning a level of environmental damage roughly equal to the worst environment impact occurring or that had occurred anywhere in the region due to transportation facilities. A score of "10" was defined as "excellent performance, comparable to the best performance seen anywhere in the region's transportation system." Intermediate scores, it was explained, should be interpreted as being linear in value. For example, a score of "5" should be interpreted as a level of performance roughly halfway in desirability between scores "1" and "10."

In the case of the cost objective, the "1-to-10" scale was defined in a slightly different way (economic costs were simply normalized to a 1-to-10 scale). The Committee was advised to assume that a score of "1" was a dollar cost equal to the cost of the alternative believed to have the highest cost, and a score of "10" was defined as a cost equal to the cost of the alternative believed to be least expensive. The twelfth objective, the likelihood that the alternative would be implemented, was also generated by Committee members. It was expressed as a probability between 0% and 100%.

As with the weight assessment, scores were obtained in two rounds. Following the initial round of scoring, the variations in scores assigned for each objective and alternative were presented to the group in graphs, and the individuals whose scores differed the most from the group average were asked to explain their reasoning. As was the case with weights, the graphs showing the variations in assigned scores generated spirited discussions which, as revealed by the scores assigned in the second round, resulted in scores moving closer together. Not all score adjustments, however, were made by the members with the most extreme scores. Sometimes, Committee members moved their scores closer to what had been a more extreme level. Such changes occurred when a member with a more extreme score presented a justification for that score that was convincing to the other Committee members.

Using a simple spreadsheet, each individual's weights, scores, and probabilities of implementation were combined to obtain an expected value for each alternative and a prioritization of alternatives for each individual. Also, the group average weights, average scores, and average probabilities were combined to obtain a "Committee view" for the relative value for each alternative and a ranking of alternatives.

Results

Top-ranked alternative

Figure 31:   Top-ranked bypass alternative.


  1. The highest ranked alternative, when the averages of Committee weights, scores, and probabilities of implementation were entered into the model, was a Bypass: an 11-mile, four-lane highway in a corridor to the south of Newberg and Dundee.
  2. The second and third ranked alternatives, again using Committee averages, were the Transportation Management with enhanced express bus service alternative and the Rail alternative. The alternative to Widen the Existing Highway ranked lowest.
  3. When each person's individual weights, scores, and probabilities of implementation were entered into the model, for nearly every participant, these same three alternatives ranked at the top.
  4. The fact that nearly everyone had the same three alternatives ranked highest came as a surprise to some participants, especially those who had initially expressed preferences for other alternatives. Participants were offered the opportunity to change their scores, however, none chose to do so. Evidently, they were comfortable in their judgments and found the logic of the model and results compelling.
  5. Committee scores closely paralleled those developed independently by the Technical Team. This correspondence was interpreted by ODOT as demonstrating that, "Stakeholder advisory committees carefully constituted to incorporate technical, jurisdiction, and interest-group points of view, can be relied upon to develop rational interpretations of complex data sets" [2].
  6. The most significant difference between Committee and Technical Team scores was that Committee members tended to estimate higher probabilities for successful implementation for the Transportation Management with express bus service alternative and for the interurban Rail alternative than did the Technical Team, whose probabilities more closely matched the low estimates provided by the Agency Advisory Committee.
  7. Sensitivity analyses showed that maximizing the likelihood of implementation was by far the most influential objective in the ranking; it was the only objective for which varying estimates across the range defined by the low and high inputs produced a change in the rank order. This result highlighted the importance of obtaining accurate assessments of the likelihood that a preferred alternative could be successfully implemented. Thus, "Though it is tempting to evaluate options based on the assumption that each could be implemented, it is critical to consider any obstacles to implementation that options face" [2].

After deliberation, the Committee chose to recommend that further study be limited to the three options with the highest priority. The Project Oversight Steering Team (POST) agreed and forwarded these alternatives on for Environmental Impact Statements. The Alternatives Analysis Technical Report, published in 1997, documented the prioritization effort and results [4].

Epilogue

Bypass phase 1

In September 2000, in preparation for the EIS, the POST began development of a Purpose and Needs Statement. After deliberation, the POST concluded that the Transportation Management alternative (with its bus, bicycle, pedestrian, and local traffic circulation improvements) could not meet the traffic congestion reduction objectives of the Purpose and Need Statement, and that a bypass was needed. The Tier 1 (Location) EIS was completed and the Record of Decision was issued by FHWA in June 2005. The Tier 2 (Design/Construction) EIS process identified and approved a specific location for the bypass within the Tier 1 corridor [1].

The bypass project will be the largest new state highway construction project in northwest Oregon in nearly 35 years. Its total cost is estimated to be $263 million [5]. The Oregon Jobs and Transportation Act passed in 2009 provided $192 million in funds, sufficient to conduct a first phase of the project, a four mile, two-lane stretch with bridges that began at Highway 219 at the east end of Newberg and is continuing southwest to bypass most of Newberg and Dundee [6]. Phase 1 is estimated to be complete and open to traffic sometime in late 2017 [4].

Is MODA the Right Approach for Your Organization?

Though government applications of MODA, such as the above, differ from typical, private-sector applications, the key steps and challenges are the same. MODA is the approach I use to help organizations establish ppm capability. Independent scientific organizations have designated MODA best-practice [7], and I have listed the reasons I believe MODA is the superior methodology for project prioritization.

However, the MODA process is not for everyone. As illustrated by the example, creating a MODA-based prioritization model is a demanding effort, more so than the simplistic, but much easier to implement, qualitative procedures that, for example, advise you to "consider urgency, importance, and value," and then, "go with your gut feel" [8].

Creating a formal MODA-based model for prioritizing your organization's projects will likely not be easy. It requires defining the specific objectives that apply to your projects and specifying performance measures for quantifying the degree to which those objectives are achieved. These objectives and performance measures can't be defined in an arbitrary way; the definitions must meet technical requirements stipulated by the MODA methodology.

The pages in this part of the paper describe the step-by-step, MODA, model-building and application process. Following the steps requires hard thinking and real effort. The reward is a best-practice prioritization model that can be applied over and over again both to produce a strict priority listing for your current projects and to evaluate and properly insert new projects into the priority order. Whether expending the necessary effort to create and then maintain a quality project prioritization tool makes sense depends on how important selecting the right projects is to your organization, and on whether within your organization there is sufficient appetite to assimilate a rigorous, demanding, logic-based, project prioritization process.

The next page describes the final steps in the process of creating a project selection decision model, creating a consequence model to simulate the outcomes of project decisions.

References

  1. Oregon Department of Transportation, "TIGER III Application, Newberg-Dundee Bypass Phase One," DUNS Number: 809580681, TIGER ID Number: Fricke1580254, 2011.
  2. M. Schwartz, M. W. Merkhofer and R. Upton, "An Innovative Approach to Multiple-Criteria Evaluation of Multimodal Alternatives: Newberg-Dundee Transportation Improvement Project Case Study," Transportation Research Record: Journal of the Transportation Research Board 1617, 139-148, 1998.
  3. R. L. Keeney and D. von Winterfeldt, "Practical Value Models, The Art of Assessing Multi-attribute Utility Functions," Organizational Behavior and Human Performance 19(2), 267-310, 1977.
  4. "Newberg-Dundee Transportation Improvement Project, Bypass Element Location (Tier 1)," Yamhill County: Environmental Impact Statement 2005.
  5. O. Cogan, Dundee: Transportation System Plan, City of Dundee, Oregon, 2003.
  6. Yamhill County, "Yamhill County Notice of Adopted Amendment (2012-08-27)," 2012.
  7. Letter from the NAS to DOE, as referenced in M. W. Merkhofer and R. L. Keeney, "Multiattribute Utility Analysis of Alternative Sites for the Disposal of Nuclear Waste," Decision Analysis 7(2), 1987.
  8. T. Sussex, "How to Prioritize Work When Everything is #1," Liquid Planner Blog, April 2014.