Lee Merkhofer Consulting Priority Systems
Implementing project portfolio management

"To find metrics for measuring project value construct a project value model."

Finding Metrics for Valuing Projects

The primary function of project portfolio management (ppm) is to manage the project portfolio so as to generate maximum value [1, 2]. The organization's senior executives determine for whom value should be created. Consistent with the organization's mission, some organizations focus on generating value for shareholders; others seek value for customers, citizens, employees and/or other organizational stakeholders. Regardless, in order to maximize the value generated by the project portfolio, organizations need the ability to estimate the value of projects.

Project Value

The value of a project is not an intrinsic property of the project. Rather, project value is a reflection of the usefulness of the project to the organization that is condering doing it. In addition to the specifications of what the project will do, project value depends on the organization's strategy, the needs and opportunities that it is facing, the resources that it has available to it, and the preferences of its senior decision makers, including their willingness to accept risk.

Value Means Worth

value means worth

Despite the fact that so many organization-specific factors determine project value, a single, dollar value can be ascribed to every project at every organization. Project value, as I've defined it, is what obtaining the project consequences is worth to the organization—it is, the amount of money, at present, that is perceived by the organization's senior decision makers as being of equal worth to the prospect of obtaining the project's, possibly uncertain, future consequences.

Organizations Seek to Maximize Project Value

Although most organizations understand the concept of project value, many lack the ability to assign dollar values to projects. However, even without a formal model for quantifying project value, organizational decision makers, those individuals who make the final choices about which projects to conduct, often do so based on an intuitive feeling that some projects are worth more than others [3]. If the value estimated to be contributed by a candidate project is perceived to be sufficiently large, the project is selected. If not, it is rejected. Unfortunately, due to the difficulty of forecasting project consequences, the multiple considerations relevant to determining the value of those consequences, and the limits of human intuitive reasoning, it is easy for good projects to be overlooked and poor projects to be chosen, especially when risk is involved.

Clearly, it would be useful to organizations to have the ability to quantify the value of projects. Some organizations are, in fact, experimenting with formal methods for valuing and prioritizing projects. So, how can an organization measure the value of its projects? Or, to state the question in the way it is more typically posed: "What are the metrics for measuring project value?"

Bottom Up vs. Top Down Modeling

Bottom up or top down

Most organizations have difficulty answering this question. The natural tendency is to measure what's easy to measure, not necessarily what's most useful or important. Organizations typically follow a bottom-up approach. Managers start by defining what appear to be useful metrics, but then can't come up with the aggregation equation for combining those metrics into a proper measure of project value. They end up using scoring models ungrounded in any theory of value and/or vague and dubious concepts such as "balance" and "strategic alignment." The resulting numbers assigned to projects have little if anything to do with the actual value of those projects. Unless there is a logical way to combine selected metrics into a measure of project value, those metrics will not be of much help for identifying value-maximizing project portfolios.

So, how, then, do you determine the metrics and associated equations for computing the value of projects?

Value Models

The answer is: "Reverse the process, use a top-down approach and create a project value model" [4].

Consistent with my definition of project value, I define a project value model to be a model for computing the value of a project. A value model embodies the values and preferences that the organization wishes to apply for the purpose of comparing, prioritizing and selecting projects, judgments that are appropriately provided by the organization's most senior executives. Like other kinds of mathematical models, a value model has inputs and outputs, consists of variables and equations, and can be implemented in software. The inputs to a project value model describe the characteristics and estimated consequences of conducting a candidate project. The output of the model is an estimate of what obtaining the project's consequences is worth to the organization.

An Example Value Model

There are many different types of value models. Net present value (NPV), for example, is a value model that accounts for one type of value, financial value. The inputs to NPV are the cash flows resulting from an investment, incremental cash that would be received and/or paid out over time as a consequence of the decision to conduct the project. The preference judgment needed for the model is the discount rate. The output of the NPV value model is the present value of those cash flows, an amount estimated to be of equivalent worth to the project.

NPV is a start, but we need value models capable of accounting for other types of project value in addition to financial value. That's because financial return isn't the only objective relevant to project decisions.

Multiple objectives complicate decision making because the objectives often conflict with one another, requiring the decision maker to accept tradeoffs. For example, when choosing to purchase a product, you might care about cost and quality. Higher quality often means a higher price.

So, how do you create a multi-criteria model for measuring project value?

Theory for Measuring Value

There is, in fact, a theory for measuring value. The theory is called utility theory, but it also goes by the more presumptuous title, The Theory of Rational Decision Making. The concepts were first outlined more than a half century ago by mathematicians Oscar Morgenstern and John von Neumann in their book, "Theory of Games and Economic Behavior" [5].

Morgenstern and von Neumann were interested in how one ought to choose among gambles, referred to as lotteries. They began by specifying a set of axioms (assumptions) intended to define what it means to choose rationally. Although this initial work dealt with choosing among lotteries with well-defined probabilities, Leonard Savage and others subsequently expanded the theory so as to apply to any uncertainties a decision maker might face, including those for which lack of data means probabilities must be assigned subjectively [6].

Axioms of Rationality

Morgenstern and von Neumann

The axioms proposed for defining rational choice can be expressed in several different ways [7]. For example, one axiom, called transitivity, states: If you prefer item A to item B, and you prefer item B to item C, then you must prefer item A to item C. An axiom called monotonicity states: If you are offered two gambles involving prizes A and B and you prefer prize A to prize B, then you must prefer the gamble where the probability of winning A is higher. An axiom called continuity states: If you prefer item A to item B and item B to item C, then there must be some probability p that would make you indifferent between item B and a gamble that provides item A with probability p and item C with probability 1-p. A full list of axioms for rational choice is provided in my glossary.

Most people find the axioms for rational decision making reasonable and easy to accept [8]. Even so, the axioms underlying decision theory aren't meant to describe how people actually make decisions, they describe a proposed ideal for how people ought to make decisions. In practice, and for complex decisions especially, studies show people routinely violate the axioms, in part due to the common biases and errors described in Part 1 of this paper.

Expected Utility Theorem

Remarkably, Morgenstern and von Neumann, and then Savage, were able to prove that, if a decision maker accepts the axioms of rationality, a function called a utility function exists that assigns numbers (utilities expressed in units called "utiles" or "utils") to outcomes such that the outcomes with higher utility numbers (more utiles) are always more preferred. Furthermore, if outcomes to a choice are uncertain, and the decision maker's utilities and probabilities are assigned to each of the possibilities, then the most preferred choice is always the one with the highest expected value of utilities (possible numbers of utiles multiplied by their probabilities and then summed) [9]. If the outcomes are undesired (rather than desired) the computed utility numbers will be negative, in which case, the highest utility will be the one with the least negative number (as illustrated by an example below).

Leonard Savage

The utility function concept would not be of much use if it were not for the fact that decision analysts have devised methods for deriving utility functions, including using an interviewing process whereby an individual's personal utility function is assessed by asking a series of questions [10, 11]. Methods have also been developed for probability encoding; that is, eliciting an individual's subjective probabilities for an uncertain quantity [12]. Following Savage, the probabilities that are assigned to an uncertain outcome must reflect the decision maker's beliefs about how likely the possible outcomes are. However, if objective data are available for deriving probabilities, those probabilities may be used, provided, of course, that computed results reflect the decision maker's beliefs.

Building Value Models Based on Utility Functions

Utility functions and the methods for assessing them provide the means for creating value models. However, in contrast to the bottom up approach often used to build other types of models, value models based on utility functions are created from the top down, starting with specifying and structuring the decision maker's objectives. A performance measure (called an attribute) is defined for each objective. For example, as noted above, cost and quality might be objectives for purchase decisions. Product price in dollars might be chosen as the performance measure for cost, and the number of years the product will last might be the performance measure for quality. If the utility function is constructed with performance measures for more than one objective, it is referred to as a multi-attribute utility function. The performance measures are the attributes for a multi-attribute utility function.

Deterministic vs. Probabilistic Decision Models

When building a value model, the analyst can decide whether to ignore uncertainties and make the model deterministic, or, alternatively, to make the model probabilistic by assigning probabilities to one or more uncertain variables. In either case, a decision maker's preferences can be modeled with utility functions. However, by convention, if the model is deterministic, the utility function is termed a value function [13,14]. A value function is simply a special case of the utility function applicable where it is assumed that there is no uncertainty over decision outcomes. The value function measures preference under conditions of certainty. The utility function measures preference under conditions of uncertainty.

The general mathematical forms for utility functions, plus three of the most useful special forms are presented below.

Utility Function Mathematics

Math wise, a utility function is usually denoted U(x), where x is the outcome to be evaluated by the decision maker. So, according to the theory, if the decision maker is choosing among alternatives that lead to different x, then the best (most preferred) choice will be the one producing the outcome x that maximizes U(x). If x is uncertain and the decision maker assigns a probability distribution, denoted p(x), describing his or her beliefs about how likely the possible values of x are, the best alternative will be the one that maximizes the expected value of U, denoted E[U(x)].

If the utility function is multi-attribute and based on N objectives, designated Oi, for i = 1 to N, there will be N corresponding attributes denoted Xi, for i = 1 to N. Each attribute Xi measures the performance of the project with respect to its corresponding objective Oi. If, for example, the decision maker is a manager at a construction company, worker safety might be one objective. The number of injuries that occur while conducting a project might be chosen as an attribute for measuring the level of safety attributed to that project. Depending on the particular project or projects that the decision maker conducts, each Xi attribute will take on some particular level. For example, the number of injuries that occur to workers might be "3." The outcomes for each attribute Xi are denoted xi (in other words, upper case is used to represent the variable and lower case is used to represent a particular outcome for that variable).


Utility function components

Figure 4:   Building a value model from the top down.


A collection of outcomes for the attributes, denoted (x1,x2...xN) is sometimes referred to as an outcome bundle. The utility achieved given a specific outcome bundle is denoted U(x1,x2...xN). A utility function is a mapping of a multi-dimensional attribute space into a single dimensioned preference space.

The standard notation for expressing value functions is analogous to that used for the more general utility functions. If there are N objectives designated Oi, for i = 1 to N, the corresponding attributes for measuring performance against those objectives are designated Xi, for i = 1 to N. A possible outcome bundle for the attributes is denoted (x1,x2...xN) where each xi is a specified level of the i'th attribute, Xi.

Once a set of fundamental objectives and associated attributes are defined, a value model may be constructed by specifying the equation that will associate with each outcome bundle (x1,x2...xN) a number indicating the decision maker's preference for that outcome bundle.

For the purposes of project selection, the most important results in the theory of value functions is this: If the decision-maker's preferences for the Xi satisfy a particular set of independence conditions (described on the next page), an additive value function may be constructed [15]:


Additive value function:

V(x1,x2...xN) = w1V1(x1) + w2V2(x2) + ... + wNVN(xN)

(Eq.3)

In this equation, V is overall value, the Vi are single attribute value functions that express preferences for consequences that differ in terms of only a single attribute, and the wi are weights. The single attribute value functions are sometimes called scaling functions.

The notation and simplifications useful for utility functions are analogous to those for value functions. As with value functions, there is a similar additive form for utility functions. An additive utility function computes total utility U by weighting and summing single-attribute utility functions Ui(xi) for each of the individual attributes Xi.


Additive utility function:

U(x1,x2...xN) = w1U1(x1) + w2U2(x2) + ... + wNUN(xN)

(Eq.4)

Additive Value Function Example

For rent sign

The following example shows how the assumption of an additive form simplifies the construction of a value model. Suppose, a young woman is choosing an apartment. For simplicity, assume there are just two objectives, (1) minimize the commute distance from her apartment to her job and (2) minimize monthly rent. Further, assume there are just three alternatives: (1) an apartment in the building where she works, (2) a nearby, downtown apartment, and (3) an apartment in the country. The objectives are to be "minimized" because commuting and paying rent are both undesirable activities. The example reflects this by using negative utilities.

Natural scales are selected for the two attributes: commute distance in miles and rent in dollars. Assuming the additive value function of the form shown in Equation 1 applies, for convenience, the single attribute value functions may be scaled to go from zero (best level) to minus one (worst level). For the rent scale, we assume that the woman feels the "undesirability" of rent in proportion to the dollars to be paid. A value of zero is assigned if the rent is zero (like living in her parent's house) and a value of minus one is assigned for a rent of $2000 per month (the highest rent that she can afford). (If, as in this example, a scaling function is linear in the attribute, it can be removed from the equation since the weight may be used to convert the attribute level to the value of that attribute level.)

With regard to commute distance, as shown in the figure below, we suppose the woman assigns a value of zero to any distance between zero and one mile (a distance such that she could walk to work). Above one mile she feels the level of undesirability is linear in miles up to 60 miles (picked as the limit because it is the longest distance she is willing to commute). The maximum commute distance of 60 miles is assigned a value of minus one.


Scaling function

Figure 5:   Assumed scaling function for commute distance.



Weights of 0.4 and 0.6 are selected to represent the relative value the women would experience (how much she would enjoy) moving from minus one to zero on each scale (these are assumed to be assessed using the swing weight method).

As a result of these assumptions the utilities of possible alternatives must lie between zero and minus one utile. Since one of the objectives is measured in dollars, and utility is assumed to be linear in dollars, the ratio of dollars to utiles for rent can be computed and used to express the value of commute distances in dollar equivalents. Thus, we compute the total (negative) value of each alternative as shown below. The alternative with the least negative equivalent cost is, assuming the assumptions are correct, the most desirable choice.


Example Additive Value Function Analysis

Figure 6:   Example additive value function analysis.



Exponential Utility Function

In addition to the additive forms for value and utility functions, there is one other form that is extremely useful for project selection. An exponential utility function is the mathematical form obtained when an exponential funtion is used to transform a value function. Because an exponential transformation is non-linear, a cardinal value function in the exponent loses its inteval scaling. However, the function is still such that higher utility numbers are better. Furthermore, the exponential function can be inverted using a logarithm function to revert to value numbers that are interval scaled (how to do this is illustrated in subsequent pages). The most popular version of the exponential utility function is one where the value function in the exponent has the additive form:


Exponential function:

U(x1,x2...xN) = 1 - e-V(x1,x2...xN) / R

(Eq. 5)

Here U is the overall (ordinal) utility function quantifying preferences, V(x1,x2...xN) is an additive value function, e denotes Euler's number (~2.78128), and R is a parameter called risk tolerance that can be set to reflect different decision-making attitudes toward risk. The exponential transformation converts a value function into a utility function capable of accounting for risk. In order to specify the utility function, it is only necessary to specify the single attribute value functions Vi, the wi weights, and the risk tolerance parameter R [16].

Value model

Utility Theory Provides the Mathematical Foundation for Creating Value Models

Compare the utility function concept to my definition of project value as summarized above and described more fully on the first page of this Part 3 of my paper. Since value maps to preferences exactly and utility maps to preferences exactly, utility and value must be the same, save only for the scaling factor needed to express utility in financial units. Thus, if we have a way to obtain a utility function for projects from the organization's senior decision makers, we will have a way to value projects. If we divide each project's value by its cost, we have the correct measure for prioritizing independent projects [17]. In summary, utility theory provides a solid basis for constructing models for valuing projects, even projects whose outcomes are uncertain.

Metrics for Measuring Value

Now that you've been introduced to value models based on utility theory, the appropriate metrics for measuring project value should be obvious. The desired metrics are the inputs to a project value model; that is, the attributes of a utility function designed to measure the value of projects. Utility theory and a top-down modeling approach will provide the metrics needed to measure the value of your projects.

References

  1. R. G. Cooper, R. G. & S. J. Edgett, Portfolio Management For New Products 2nd ed., Perseus Publishing, 2002.
  2. Y. P. Petit, Y. B. & B. Hobbs, Project Portfolios in Dynamic Environments: Organizing for Uncertainty Project Management Institute, 2012.
  3. P. F. Drucker, "What Makes an Effective Executive." Harvard Business Review, June 1, 2004.
  4. The recommendation to develop a decision model and, more generally, the views and ideas expressed in this and the next part of this paper are shared by many decision analysts. See especially "Choosing the Right Metrics for Measuring, Monitoring, and Maximizing Shareholder Value," C. Spetzler and R. Arnold, www.sdg.com, May 2003. The book Value Focused Thinking by R.Keeney, 1992, describes many of the concepts and techniques for building decision models.
  5. J. von Neumann & O. Morgenstern, Theory of Games and Economic Behavior, Princeton University Press 1947.
  6. L. J. Savage, The Foundations of Statistic Courier Corporation, 1972.
  7. E. Karni, "Axiomatic Foundations of Expected Utility and Subjective Probability," Chpt 1 in Handbook of the Economics of Risk and Uncertainty, Elsevier 2014.
  8. R. F. Nease, "Do violations of the axioms of expected utility theory threaten decision analysis?" Medical Decision Making Oct-Dec; 16(4), 399-403, 1996.
  9. R. L. Keeney & H. Raiffa, Decisions with Multiple Objectives: Preferences and Value Trade-Offs, Cambridge University Press, Jul 1, 1993.
  10. C. L. Hwang & A.S.M. Masud, Multiple Objective Decision Making - Methods and Applications: A State-of-the-Art Survey, Springer Science & Business Media 2012.
  11. C. W. Kirkwood, Strategic Decision Making: Multiobjective Decision Analysis with Spreadsheets Duxbury Press, 1997.
  12. M. W. Merkhofer, "Quantifying Judgmental Uncertainty: Methodology, Experiences, and Insights," IEEE Transactions on Systems, Man, and Cybernetics 17(5), 1987.
  13. R A. Howard, Decision Analysis: Applied Decision Theory Stanford Research Institute, 1966.
  14. R. L. Keeney, "Decision Analysis: An Overview," Operations Research 30(5), 803-838, 1982.
  15. J. S. Dyer & R. K. Sarin, "Measurable Multiattribute Value Functions," Operations Research,, 27(4), 1979.
  16. J. S. Dyer & R. K. Sarin, "Relative Risk Aversion," Management Science, 28(8), 1982.
  17. P. Goodwin & G. Wright, Decision Analysis for Management Judgment, Wiley, 1998.