Lee Merkhofer Consulting Priority Systems
Implementing project portfolio management

"The challenge comes when the organization tries to find an equation for combining assessments of project performance into a single measure for computing the value of its projects."

Specifying the Aggregation Equation for Computing Project Value

As noted previously, it is common for organizations to define multiple criteria for evaluating candidate projects. The challenge comes when the organization seeks a way to combine assessments relative to its various criteria into a single metric for quantifying project priority. What organizations need is an equation that shows how assessments of performance can be aggregated into an estimate of the value of that performance. Assuming that projects can be defined to be independent of one another, an organization with the ability to quantify the value of its candidate projects can correctly prioritize them based on the ratio of project value to project cost.

Perhaps the most important contribution from the field known as multi-objective decision analysis (MODA) is a theoretically sound logic for defining project performance measures and combining them into a measure of project value. On this page I describe how to obtain the desired equation for project value. Figure 24 identifies the required steps and shows how they relate to my 12-step process for creating a project selection decision model.


Steps for creating a project selection model

Figure 24:   Steps for defining the equation for aggregating performance estimates.


Multi-Objective Decision Analysis

As described previously, the objectives, Oi, i = 1, ..., N, for the project portfolio are identified. Then, for each objective a performance measure Xi, i = 1, ..., N, is specified (performance measures are also called attributes). Each performance measure measures the degree to which a project enables the corresponding objective to be met [1]. The previous pages describe how to define and structure objectives and how to select performance measures.

If performance measures are properly defined, an equation can be derived for combining the performance measures into a measure of project value [2]. I call the equation for combining performance measures the aggregation equation. The aggregation equation shows how to mathematically combine the assessments of performance relative to the organization's various objectives. According to MODA, the aggregation equation is a value function if there is no uncertainty over the performance achieved by projects (or if uncertainty is ignored). The equation is utility function if uncertainty over project performance is quantified [3]. (To avoid confusion, note that a value function capable of providing an equivalent dollar value of projects must be a measurable value function. A measurable value function provides a measure of value expressed on a cardinal scale [4].)

For Practicality, the Value/Utility Function Must Have a Simple Mathematical Form

Value functions and utility functions for measuring project value may be assessed from organizational decision makers (i.e., senior executives) using a formal, interview-based process [5]. The assessment process requires decision makers to express preferences over hypothetical outcomes for project performance. The problem is that if there are more than just a few performance measures, the number of required preference judgments is too large for the interviewing process to be practical [6]. However, if some specific mathematical form can be assumed for the function (e.g., additive, multiplicative, etc.), eliciting it becomes much easier.

Obtaining a Mathematically Tractable Value Function

The simplest mathematical form that can be assumed for a value function is the additive form [7]:


V(x1,x2...xN) = w1V1(x1) + w2V2(x2) ...+ wN VN(xN)

In this equation, V is the computed value, the Vi are single attribute value functions (also called scaling functions) that express preferences for consequences that differ in terms of only a single attribute, and the wi are weights. As indicated, with the additive form, value is a weighted sum of single attribute value functions. Thus, if a value function has the additive form, it can be obtained (relatively easily) by assessing the single attribute value functions and the wi weights. Furthermore, because a scaling function translates performance as measured on a performance scale to value measured on a preference scale, it is often possible to determine the scaling function without the need to formally assess it from decision makers. For example, if preference is proportional to the performance measure and a logarithmic scale is used to express performance (e.g., the performance scale is such that each scale level represents a performance that is ten-times more desirable than the previous scale level), then the scaling function will be exponential, indicating that each scale level is ten-times more desirable than its predecessor.

In addition to being relatively easy to assess, the additive value function has the desirable property that project value is the sum of distinct types of value. For example, if the value function has the additive form and projects produce financial value, health and safety value, brand image value, and so forth, the value of each project will be the sum of the various types of value that it generates:


ValueTotal = ValueFinancial + ValueHealth & safety ... + ValueBrand image

Aim for Satisfying the Conditions Needed for an Additive Value Function

Nearly all professional applications of MODA assume either a value function with the additive form or some simple extension of the additive form [8]. If the objectives hierarchy has been structure properly, the multi-attribute function will have a simple form. You cannot, though, merely assume a simple form for the aggregation equation—the mathematical form depends on the selected performance measures and the structure of the objectives hierarchy.

In order for the additive form to apply, it is necessary that performance measures satisfy the conditions known as mutual preferential independence and difference independence [4]. If the portfolio objectives are all fundamental objectives rather than means objectives, the required independence conditions can typically be met [2]. However, if means objectives are used or if the selected performance measures quantify only one means for achieving a fundamental objective, the value function will often not be additive. Thus, it is important to verify that an additive form for the value function is appropriate. If the additive form is used in situations where the independence conditions do not apply, computed project priorities will be logically indefensible and almost certainly inaccurate.

Achieving the Requirements for an Additive Value Function

Formal tests for the appropriateness of an additive value function are described on a previous page. In theory, the tests should be conducted using preference judgments provided by the organization's senior decision makers. However, you can conduct a quick version of the tests using your own judgments as a mental experiment [5]:

Choose one of the performance measures and imagine that the level of performance relative to that measure swings from some reasonable undesirable level to some reasonable desirable level. For example, suppose the context is highway transportation projects and public safety is an objective. Other objectives might be reducing commute times and minimizing costs to drivers due, for example, to highway tolls and the wear and tear to autos caused by rough roads. Depending on the portion of the highway transportation system impacted by projects, a poor level of performance for safety might be, say, one fatal traffic accident per year. A good level of performance might be zero fatal accidents per year. The specifics of the swings you define don't matter, however, the range covered by a swing must be roughly consistent with what might be realistically obtainable from projects.

Once you've defined a potential swing for an objective's performance measure, ask yourself whether the value of that swing, the difference in value between the undesirable and desirable levels of the measure, changes depending on the performance levels assumed for other measures. Continuing the example, the question is whether the value (desirability) of avoiding one fatality from accidents changes depending on the commute times and costs experienced by drivers. In other words, would the value attributed to avoiding one fatal accident change appreciably if average comminute times went from say, 12 minutes to 30 minutes? Would the value attributed to avoiding a fatal accident change depending on whether costs to drivers changed from 50 cents to 3 dollars?

Preferential independence

The different levels of performance assumed for these other objectives should, likewise, be consistent with the levels that might actually be achieved through projects. In other words, the fact that you might feel that the value you would attribute to avoiding one fatality might change if highway tolls were increased to the point that drivers could not afford to drive to work is irrelevant if such high tolls would not result under any portfolio projects. The question is whether the value of swings change as performance measures are moved across the range of performance levels that might conceivably occur given the projects that are actually under consideration.

If the value of swings for a given performance measure do not change depending on the levels of performance assumed for other measures, that measure is preferentially independent of the other measures. With regard to the transportation example, the value of avoiding one fatal accident will, for most people, be independent of the performance levels achieved relative to other objectives. This is because safety is a fundamental objective, and the values of swings in performance relative to fundamental objectives are viewed by most people to be preferentially independent of other measures [9].

In order to justify an additive form for the value function, the answer to these sorts of questions needs to be that the performance measures have been defined in such a way that the values of any and all realistic performance level swings do not depend on the levels assumed for other performance measures. If this is the result for each and every performance measure, the measures are mutually preferentially independent and difference independent. In this case, and only in this case, will the additive form be appropriate for the (measurable) value function.

The Most Common Cause of Non-Additivity is the Use of Means Objectives

The most common cause of non-additivity is the use of means objectives in the design of the value model [6]. Using means objectives doesn't guarantee non-additivity, but non-additivity is often the result. As an example, consider concern over salmon populations in Pacific Northwest. The estimated population of salmon in the Puget Sound has declined over 95% since the 1940's [10]. The fundamental objective for a portfolio of projects aimed at reversing the decline might be to "preserve and increase salmon populations." The decline in salmon populations is believed to be mostly due to contamination of the Puget Sound and nearby streams with pesticides. Thus, it is tempting to evaluate proposed projects based on the means objective to "reduce concentrations of pesticides in salmon habitats." However, the relationship between pesticide contaminations and salmon populations is complex. Pesticide exposure doesn't simply kill salmon outright, it can kill the aquatic invertebrates that salmon feed on. Also, pesticides can cause neurological damage that affects the salmon's sense of smell, hampering the ability of the salmon to escape predators and find their way back to spawning grounds. Of even more importance to project prioritization, the toxicity of any given pesticide to salmon can become significantly greater when combined in the "chemical soup" that results when multiple pollutants blend together [11]. Any analysis that simply added the values of the concentration reductions for each pesticide achieved by a project would miss this important non-linear, synergistic effect and likely underestimate the value of projects aimed at better managing pesticide usage. Focusing on means objectives rather the on fundamental objectives creates non-additivities for the equation for determining project value.

Author Ralph Keeney provides another example, this one relevant to prioritizing transportation projects for improving safety [12]. One means objective might be to reduce speeding by drivers. A second means objective might be to minimize driving under the influence of alcohol. Any value model that simply added the two measures would miss the observed tendency that speeding while under the influence of alcohol is generally more dangerous and, therefore, less desirable than the sum of desirability of the consequence of each separately. Once again, the error introduced by using means rather than a fundamental objective is due to non-additivity in the value of achieving means objectives.

Redefine Objectives and Performance Measures to Obtain Mutual Preferential Independence

If tests show that performance measures are not mutually preferentially independent, the portfolio objectives and their performance measures must be redefined until independence is achieved [6]. Typically, there are multiple, distinct sets of objectives and performance measures that can be selected as the basis for designing a value function. Unless the selected set produces mutual preferential independence you will probably not be able to come up with a proper equation for computing project value. Keeney argues that a failure to obtain mutual preferential independence typically means that objectives and performance measures do not represent the best way to structure the decision model [13]. Like Keeney, my experience is that in nearly all real-world situations and for most decision makers it is possible to define objectives and their performance measures so as to satisfy the independence conditions needed to obtain an additive value function. If your definitions of objectives and performance measures don't provide preferential independence, think hard about redefining objectives and their performance measures in such a way that preferential independence is, at least approximately, obtained.

Variations on the Additive Value Function

In practice, value models are typically designed with mostly additive terms, but with some non-additive terms [6]. The non-additive portions of the model are typically based on simple, logical reasoning. For example, if project success requires some uncertain event to occur, then the value function will be the product of the probability of the required event multiplied by the value obtained assuming the event occurs. Pharmaceutical product development projects, for example, are typically evaluated using value functions of this type. The ability to market a drug requires regulatory approval, which depends on the outcomes of trial tests of the drug. In such cases, maximizing the likelihood of project success is an objective, and the project value function multiplies the project's probability of success times it's value if successful. Similarly, whether or not a project provides a particular type of value may depend on the outcome of some uncertainty. In this case, the value function is mostly additive, but includes as one of its terms the contingent value multiplied by the probability that that value component materializes.

Another common justification for including multiplications in the value function is a situation described on the previous page, If the influence diagrams for some objectives indicate a quantity-intensity decomposition, then the mini-value functions representing these types of value will be multiplicative. In summary, as noted above, value models are typically composed of mostly additive terms, but with some multiplications.

Value Functions Often Include Time Series with Discounting

A common situation is one where the different components of project value begin and end at different points in time. Consider, for example, a project to implement a new technology that allows the organization to obtain an advantage over its competitors. The value added by the project might include, for example, incremental increases to future revenues. These revenue increases won't begin, of course, until the project is completed. Once the increments to revenue begin, they may persist for some period of time. However, the revenue increments aren't likely to continue indefinitely. As competitors see the advantages that the new technology provides, these firms will seek to obtain their own technological advances. Thus, there will be a point in time when the advantages obtained by implementing the technology will be eroded. Accounting for the incremental revenue value provided by the proposed project thus requires estimating when revenue increases will begin and for how long they will continue.

In contrast to the above example, consider an organization's decision to conduct a project that is highly desired by some of the organization's stakeholders. For example, an electric distribution utility might be considering a project to replace a portion of its above-ground, telephone-pole wiring with underground cable. Citizens with homes in the impacted area might be very much in favor of the project. One of the utility's objectives might be to enhance and improve its image. Thus, the goodwill that would be generated by the project might be considered to be a component of the project's value. The generated goodwill might be short lived, but it likely won't be delayed until after the project is completed. Instead, the improvement to stakeholder relations is likely occur immediately, as soon as the decision to conduct the project is announced.

Time is money

The point of these examples is that correctly valuing project consequences typically requires estimating when they occur and for how long they will last. As Benjamin Franklin reportedly said, "Time is money." Multi-period benefits are typically modeled as time streams and discounted to obtain present value equivalents. The common practice of discounting and summing the value of the benefit time streams is consistent with utility theory and can be derived from two assumptions: (1) that investors wish to maximize their wealth and (2) that a market exists that will provide a return from investing cash [6].

Determine Whether Uncertainties Will Be Quantified

If accounting for uncertainties over the performance of projects is needed, including cases where there is a risk that the project will be unsuccessful, those uncertainties must be quantified and the value model will need to be a utility function. A utility function has its own, more stringent requirements for additivity.

Whether or not uncertainties affect the prioritization of projects depends on the nature and magnitude of those uncertainties and the degree to which decision makers are risk averse. In my experience, a probabilistic analysis is typically not required if uncertainties are continuous and of small to moderate magnitude. In such cases, project priorities computed with a value function and a deterministic analysis will be the same or nearly the same as those obtained with a corresponding utility function and probabilistic analysis. However, if uncertainties are discrete with potentially serious implications, such as the case where accidents may occur that harm people or produce significant costs for the organization, a probabilistic analysis will often produce dramatically different project values and is, therefore, required to properly prioritize projects.

Obtaining a Mathematically Tractable Utility Function

For situations where uncertainty in project outcomes is significant, two alternative approaches are available for constructing a utility function. One, which has been referred to as the Keeney-Raiffa approach, is to assess a utility function directly over the performance measures by establishing the applicability of the independence condition additive independence. If additive independence can be shown to hold, then the utility function will have the additive form [14]:


U(x1,x2...xN) = w1U1(x1) + w2U2(x2) + ... + wNUN(xN)


Similar to the case for an additive value function, a utility function with the additive form may be quantified relatively easily, in this case by assessing the single-attribute utility functions, Ui(xi) and weights.

The other approach, referred to as the Stanford School (where I was trained), is to create an additive value function and then transform it to an exponential utility function [15]:


U(x1,x2...xN) = 1 - e-V(x1,x2...xN) / R


My recommendation is that you use the Stanford School approach. It does not require any additional independence conditions other than the delta property, which is typically an easy assumption provided that the outcomes of the projects under consideration can't result in large changes to the worth of the organization. The use of an exponential utility function with value in the exponent ensures that the equivalent financial value of every project will be quantified. The approach produces a utility function that includes a risk tolerance parameter that may be used to characterize the organization's willingness to accept risk. Empirical research has shown that the exponential utility function provides a good approximation to the utility functions obtained through more complex and time-consuming assessment methods based on lotteries [16].

Assessment strategy

Recommended Assessment Strategy

In summary, the basic steps and strategy for creating a project value model based on a utility theory are as follows.

  1. Redefine projects as needed to be independent of one another. Combine interdependent projects into larger projects, with "versions" that consist of the various, reasonable combinations of the interdependent project options.
  2. Define fundamental objectives and performance measures with the goal of achieving mutual preferential independence.
  3. Decide whether uncertainties in the outcomes of projects as expressed by the performance measures can be ignored. If so, then a value function will be an adequate means for assessing value. If, on the other hand, the outcomes of projects are highly uncertain, especially if the uncertainties are discrete, then a utility function should be used.
  4. Regardless of whether you'll need a value function or a utility function, check the performance measures for mutual preferential independence. Use mental tests as described above.
  5. If the performance measures are, at least approximately, mutually preferentially independent, an additive value function may be used.
  6. If, on the other hand, the tests fail, then look for an alternative way to define portfolio objectives and performance measures so that the necessary independence conditions for the additive value function apply.
  7. Once independence is obtained, assume an exponential utility function with an additive value function in the exponent. Quantify the function by assessing (1) single-attribute value functions for each performance measure and (2) weights. The only additional assessment required is organizational risk tolerance, which may be specified using methods described in Part 4 of this paper.

The next page provides an example a real-world application of the MODA process to create and apply a prioritization model.

References

  1. R. L. Keeney and H. Raiffa, Decisions with Multiple Objectives, Wiley, New York, 1976.
  2. R. L. Keeney, Value-Focused Thinking: A Path to Creative Decisionmaking, Cambridge, MA. Harvard University Press, 1992.
  3. J. S. Dyer, et. al., "Multiple Criteria Decision Making, Multiattribute Utility Theory," Management Science, 38(5), 1992.
  4. J. S. Dyer and R. K. Sarin, "Measurable Multiattribute Value Functions," Operations Research 27, 810-822, 1978.
  5. R. L. Keeney, "The Art of Assessing Multiattribute Utility Functions," Organizational Behavior and Human Performance 19(2), 267-310, 1972.
  6. R. L. Keeney and D. von Winterfeldt, "Practical Value Models," in Advances in Decision Analysis From Foundations to Applications eds. W. Edwards, R. E. Miles, Jr. and D. von Winterfeldt, Cambridge University Press, New York, 104-128, 2007.
  7. H. Raiffa, "Preferences for Multi-Attributed Alternatives," RM-5868-DOT/RC, The Rand Corporation, Santa Monica, Calif, 1969.
  8. R. L. Keeney, "Decision Analysis," in Handbook of Operations Research, J. J. Moder and S. E. Elmaghraby (eds.), Van Nostrand Reinhold, New York, 1978.
  9. R. L. Keeney and R. S. Gregory, "Selecting Attributes to Measure the Achievement of Objectives," Operations Research 53(1), 1-11, 2005.
  10. S. J. Levin, Since Time Immemorial: The Decline of Columbia River Basin Salmon, 2016.
  11. C. A. Laetz, et al. "Elevated Temperatures Increase the Toxicity of Pesticide Mixtures to Juvenile Coho Salmon." Aquatic Toxicology, 146,38-44, 2014.
  12. R. L. Keeney, "Value-Focused Brainstorming," Decision Analysis, 9(4), 303-313, 2012.
  13. R. L. Keeney, "Common Mistakes in Making Value Trade-Offs," Operations Research 50(6, 935-945, December 2002.
  14. R. A. Pollak, "Additive von Neumann-Morgenstern Utility Functions," Econometrica, 35. 485-494, 1967.
  15. J. E. Matheson R. A. Howard,“An Introduction to Decision Analysis.” The Principles and Applications of Decision Analysis vol. I, R. A. Howard and J. E. Matheson (eds.), Strategic Decisions Group: Menlo Park, California, 17-55, 1968.
  16. F. H. Barron, D. von Winterfeldt, and G. W. Fischer, G.W., "Empirical and Theoretical Relationships Between Value and Utility Functions," Acta Psychologica, 56(1-3), 233-244, 1984.h