Lee Merkhofer Consulting Priority Systems
Implementing project portfolio management

Metrics for Prioritizing Projects

Effective project portfolio management (ppm) requires having the ability to identify the best projects to include within the project portfolio. The best projects are, as I have argued on the previous two pages, the ones that generate the most value. The value of a project is the worth, expressed in dollars or in another currency, to the organization, of the consequences of conducting the project. Thus, in order to identify the best projects, you must be able to quantify project value.

What, then, are the appropriate metrics for measuring project value?

Bottom up or top down

Most organizations have trouble answering this question. The natural tendency is to measure what's easy to measure, not necessarily what's most important or most useful. Organizations typically use a bottom-up approach. They start by defining what appear to be useful metrics, but then can't come up with equations for combining those metrics into a measure of project value. They end up using arbitrary aggregation equations, such as unjustified weight-and-add scoring models, or vague and dubious concepts such as "balance" and "strategic alignment." The resulting numbers assigned to projects have little if anything to do with the actual worth of those projects. Unless there is a logical way to combine the selected metrics into a measure of project value, those metrics will not be of much help for identifying value-maximizing project portfolios.

So, how do you determine the metrics and associated equations for computing the value of projects?

Create a Value Model!

The answer is—Reverse the process, use a top-down approach and create a value model. [3].

A value model is a model for quantifying a decision maker's preferences (e.g., the preferences of an organization's most senior executives) for the various alternatives available for a decision (such as a choice over the projects to include in a project portfolio). Like other kinds of mathematical models, a value model consists of variables and relationships, has inputs and outputs, and may be implemented in software. So, how can you construct a project value model?

Formal Decision-Making Methods

A promising place to look for advice on how to create a value model is the literature on formal decision-making methods. Because decision making is important and often difficult, especially when there are many decision objectives, researchers in various fields have proposed analytic methods for evaluating and comparing alternatives for the purpose of identifying the "best" choices. Most such methods create models that assign numbers to alternatives—the numbers so assigned may be referred to as measures of desirability, quality, value, usefulness, or with some other noun to indicate the method's chosen figure-of-merit for comparing alternatives. In most cases, the decision-making approaches literarily provide step-by-step instructions for selecting variables, constructing equations, and implementing solution techniques for computing their selected figures-of-merit.

Dozens of formal decision making approaches have been devised, as suggested by the alphabet soup of acronyms in the table below. These are examples, with a little research you'd find many more. For illustration, I've provided in my glossary brief descriptions for each of the examples in the table.


Acronyms for Some Formal Decision-Making Methods
AIRM ELECTRE SMAA
AHP ER SMART
ANP GP TOPSIS
COMET MACBETH VIKOR
DEX PROMETHEE WSM

Most of the above methods evaluate alternatives based on assigning numbers to them such that the numbers represent a decision maker's preferences for those alternatives. Thus, most decision-making methods can be regarded as methods for creating value models.

From the large number of available formal decision-making methods, the method that is best suited for project selection is, in my opinion, multi-objective decision analysis (MODA). MODA, also called multi-attribute utility analysis (MUA), is a collection of modeling, analysis, and assessment techniques for implementing decision theory, a theory for making rational decisions.

The Theory of Rational Decision Making

Morgenstern and von Neumann

A theory of rational decision making was first outlined more than a half century ago by Oscar Morgenstern and John von Neumann in their book, "Theory of Games and Economic Behavior" [3]. The mathematicians were interested in how one should choose among gambles, referred to as "lotteries." They began by specifying a set of axioms (assumptions) intended to define what it means to choose rationally. Although this initial work dealt with choosing among lotteries with well-defined probabilities, Leonard Savage and others subsequently expanded the theory so as to apply to any uncertainties a decision maker might face, including those for which lack of data means probabilities must be assigned subjectively.

Axioms of Rationality

The axioms that have been proposed for defining rational choice can be expressed in several different ways. For example, one axiom, called transitivity, states: If you prefer item A to item B, and you prefer item B to item C, then you must prefer item A to item C. An axiom called monotonicity states: If you are offered two gambles involving prizes A and B and you prefer prize A to prize B, then you must prefer the gamble where the probability of winning A is higher. An axiom called continuity states: If you prefer item A to item B and item B to item C, then there must be some probability p that would make you indifferent between item B and a gamble that provides item A with probability p and item C with probability 1-p. A full list of axioms for rational choice is provided in my glossary.

Most people find the axioms for rational decision making reasonable and easy to accept. Even so, the axioms underlying decision theory aren't meant to describe how people actually make decisions, they describe a proposed ideal for how people ought to make decisions. In practice, and for complex decisions especially, studies show people routinely violate the axioms, in part due to the common biases and errors described in Part 1 of this paper

Expected Utility Theorem

Remarkably, Morgenstern and von Neumann, and then Savage, were able to prove that, if a decision maker accepts the axioms of rationality, a function called a utility function exists that assigns numbers (utilities expressed in units of "utiles") to outcomes such that the outcomes with higher utility numbers (more utiles) are always more preferred. Furthermore, if outcomes to a choice are uncertain, and utilities and probabilities are assigned to each of the possibilities, then the most preferred choice is always the one with the highest expected value of utilities (possible numbers of utiles multiplied by their probabilities and then summed). If the outcomes are undesired (rather than desired) the computed utility numbers may be negative, in which case, the highest utility will be the one with the least negative number.

Math wise, a utility function is typically denoted U(x), where x is the outcome to be evaluated by the decision maker. So, according to the theory, if the decision maker is choosing among alternatives that lead to different x, then the best (most preferred) choice will be the one producing the outcome x that maximizes U(x). If x is uncertain and the decision maker assigns a probability distribution, denoted p(x), describing his or her beliefs about how likely the possible values of x are, the best alternative will be the one that maximizes the expected value of U, denoted E[U(x)].

Leonard Savage

The utility function concept would not be of much use to us if it were not for the fact that decision analysts have devised methods for deriving utility functions, including using an interviewing process whereby an individual's personal utility function is assessed by asking a series of questions. The assessment techniques will be described shortly.) Methods have also been developed for probability encoding; that is, eliciting a subject's subjective probability distribution for an uncertain quantity. Following Savage, the probabilities that are assigned to an uncertain outcome must reflect the decision maker's beliefs about how likely the possible outcomes are. However, if objective data are available for deriving probabilities, those probabilities may be used, provided, of course, that computed results reflect the decision maker's true beliefs. Thus, the utility function together with available assessment techniques provide the foundation for mathematically identifying optimal, value-maximizing choices.

Multi-Objective Decision Analysis

MODA is a version of decision theory that applies to the common situation where the decision maker has multiple objectives. Multiple objectives complicate decision making because the objectives often conflict with one another, requiring the decision maker to accept tradeoffs. For example, when choosing to purchase a product, you might care about cost and quality. Higher quality often means a higher price.

Top-Down Model Construction

In contrast to the bottom up approach often used to build other types of models, MODA models are created from the top down, starting with specifying and structuring the decision maker's objectives. A performance measure is then defined for each objective. For example, product price in dollars might be chosen as the performance measure for cost, and the number of years the product will last might be the performance measure for quality. The performance measures are the attributes for the multi-attribute utility function.

In the language of math, if the decision maker has N objectives designated Oi, for i = 1 to N, there will be N corresponding attributes denoted Xi, for i = 1 to N. If the decision maker is a manager in a construction company, for example, worker safety might be one objective. The number of injuries occurring annually might be chosen as an attribute for measuring the level of safety achieved. As a result of the alternatives that the decision maker selects, each Xi attribute will take on some particular level. For example, the number of injuries that occur might be "3." The outcomes for each attribute Xi are denoted xi (in other words, upper case is used to represent the variable and lower case is used to represent a particular outcome for that variable).


Utility function components

Figure 22:   Building utility functions from the top down.


A collection of outcomes for the attributes, denoted (x1,x2...xN) is sometimes referred to as an outcome "bundle." The utility achieved given a specific outcome bundle is denoted U(x1,x2...xN). A MODA utility function can be regarded as a mapping of a multi-dimensional attribute space into a single dimensioned preference space.

Deterministic vs. Probabilistic Decision Models

When building a model, the analyst can decide whether to ignore uncertainties and make the model deterministic, or, alternatively, to make the model probabilistic by assigning probabilities to one or more uncertain variables. In either case, a decision maker's preferences can be modeled with MODA. However, by convention, if the model is deterministic, the utility function is termed a value function. A value function is simply a special case of the utility function applicable when it is assumed that there is no uncertainty over decision outcomes. The value function measures preference under conditions of certainty. The utility function measures preference under conditions of uncertainty.

More detail on MODA mathematics is provided in the remainder of this section of the paper. The subsequent section describes how to assess utility functions and value functions from decision makers.

Utility Function Mathematics

Value/utility functions represent the main components of MODA value models. As will become clear if you read through the next few sections of the paper, a practical approach for specifying a value/utility function begins with selecting objectives and their performance measures in such a way as to achieve a degree of independence for expressing preferences. The desired independence assumptions allow the function to be decomposed into component parts that can be individually estimated.

The general mathematical representations for utility functions and value functions, plus two of the most useful forms for decomposition are presented below.

Value Functions

The standard notation for expressing value functions is analogous to that used for the more general utility functions. If there are N objectives designated Oi, for i = 1 to N, the corresponding attributes for measuring performance against those objectives are designated Xi, for i = 1 to N. A possible outcome bundle for the attributes is denoted (x1,x2...xN) where each xi is a specified level of the i'th attribute, Xi.

Once a set of fundamental objectives and associated attributes are defined, a value model may be constructed by associating with each outcome bundle (x1,x2...xN) a number indicating the decision maker's preference for that outcome bundle.

Additive Value Functions

Perhaps the most important practical result in the theory of value functions is this: If the decision-maker's preferences for the xi satisfy a set of conditions known as additive independence, an additive value function may be constructed:


Additive value function:

V(x1,x2...xN) = w1V1(x1) + w2V2(x2) + ... + wNVN(xN)

(Eq.1)

In this equation, V is overall value, the Vi are single attribute value functions that express preferences for consequences that differ in terms of only a single attribute, and the wi are weights. The single attribute value functions are often termed scaling functions. The following example shows how the assumption of an additive form simplifies the estimation of value.

Additive Value Function Example

Morgenstern & von Neumann

A young woman is choosing an apartment. For simplicity, assume there are just two objectives, (1) minimize the commute distance from her apartment to her job and (2) minimize monthly rent. Further, assume there are just three alternatives: (1) an apartment in the building where she works, (2) a nearby, downtown apartment, and (3) an apartment in the country. The objectives are to be "minimized" because commuting and paying rent are both undesirable activities. The example reflects this by using negative utilities.

Natural scales are selected for the two attributes: commute distance in miles and rent in dollars. Assuming the additive value function of the form shown in Equation 1 applies, we specify, for convenience, that the single attribute value functions go from zero (best level) to minus one (worst level). For the rent scale, we assume that the woman feels the "undesirability" of rent in proportion to the dollars to be paid. A value of zero is assigned if the rent is zero (like living in her parent's house) and a value of minus one is assigned for a rent of $2000 per month (the highest rent that she can afford). (If, as in this example, a scaling function is linear in the attribute, it can be removed from the equation since the weight may be used to convert the attribute level to the value of that attribute level.)

With regard to commute distance, as shown in the figure below, we suppose the woman assigns a value of zero to any distance between zero and one mile (a distance such that she could walk to work). Above one mile she feels the level of undesirability is linear in miles up to 60 miles (picked as the limit because it is the longest distance she is willing to commute). The maximum commute distance of 60 miles is assigned a value of minus one.


Scaling function

Figure 22:   Assumed scaling function for commute distance.



Weights of 0.4 and 0.6 are selected to represent the relative value the women would experience (how much she would enjoy) by moving from minus one to zero on each scale (these are assumed to be assessed using the swing weight method).

As a result of these assumptions the utilities of possible alternatives must lie between zero and minus one utile. Since one of the objectives is measured in dollars, and utility is assumed to be linear in dollars, the ratio of dollars to utiles for rent can be computed and used to express the value of commute distances in dollar equivalents. Thus, we compute the total (negative) value of each alternative as shown below. The alternative with the least negative equivalent cost is, assuming the assumptions are correct, the most desirable choice.


Example Additive Value Function Analysis

Figure 22:   Example additive value function analysis.



Utility Functions

As described above, multi-attribute utility functions may be used to represent an individual's preferences over uncertain outcomes described by multiple attributes. The notation and simplifications useful for utility functions are analogous to those for value functions.

Additive Utility Functions

As in the case with value functions, there is a similar additive form for utility functions. An additive utility function computes total utility U by weighting and summing single-attribute utility functions Ui(xi) for each of the individual attributes xi.


Additive utility function

U(x1,x2...xN) = w1U1(x1) + w2U2(x2) + ... + wNUN(xN)

(Eq.2)

Exponential Utility Function

Another special form for the utility function is an exponential function. As we've seen, utility functions and value functions are such that higher is better. Any constantly increasing scaling applied to the functions (i.e., any monotonic transformation) will preserve the order of the utility numbers and, therefore, have no effect on either the choice that maximizes the function or the preference order of alternatives. Thus, a possible mathematical form for the utility function is an exponential function wherein an additive value function appears in the exponent of the function. More specifically, a possible (in fact, fairly common) mathematical form that may be selected for a utility function is:


U(x1,x2...xN) = 1 - e-V(x1,x2...xN) / R

Here U is the overall utility function quantifying preferences, V(x1,x2...xN) is an additive value function, e denotes the Euler's number (~2.78128), and R is a parameter called risk tolerance that can be set to reflect different decision-making attitudes toward risk (see Part 4 of this paper for more discussion of risk tolerance). The exponential form for the utility function is attractive because it allows an additive value function to be converted to a utility function applicable to a probabilistic decision model. In other words, if the condition of additive independence is satisfied for the value function, the specification of the above form for the utility function requires simply specifying the Vi, wi and the risk tolerance R. The utility function in this case is sometimes called the "relative risk aversion function," since it is the risk aversion expressed relative to the value function.

Metrics for Measuring Value

value means worth

Now that you've been introduced to the mathematics of value models, the appropriate metrics for measuring project value should be obvious. The appropriate metrics are the inputs to a multi-objective value model, which will be either a utility function or a value function specified to capture the values of your organization. The model will assign numbers (utiles) to projects such that the higher the number is, the more desirable the project is. As shown in the above example, provided that the value or utility function is additive and linear in money, the preference measure can be expressed in equivalent monetary units, the most convenient unit for project prioritization.

One way to obtain a value function or utility function for your MODA value model is to elicit it directly from a decision maker. There are other ways as well, but there are times when eliciting the function from the decision maker is most convenient. The next section of this paper describes the interviewing process for eliciting a value or utility function from a decision maker.