Lee Merkhofer Consulting Priority Systems
Implementing project portfolio management

Project-Selection Decision Models

So, if the goal is to choose projects based on value generated, what kinds of metrics reflect impacts on value? Many organizations have trouble answering this question. Organizations tend to measure what's easy to measure, not necessarily what's important or useful. Most organizations use a bottom-up approach. They define interesting metrics, but then can't come up with the equations or algorithms for computing project value based on those metrics. They end up using arbitrary aggregation equations, such as weighted summations, or vague and unjustified concepts, such as "balance" or "strategic alignment." Unless there is a logical way to combine metrics to determine the value added by projects, the metrics will not be of much help for identifying value-maximizing project portfolios. How do you determine the metrics and associated equations for computing the value added by projects?

Create a Decision Model

The answer is—You need to reverse the process, use a top-down approach and create a project-selection decision model [3]. A decision model is an analytic representation, typically implemented in software, that captures the key considerations for decision making and derives recommended choices from them. As illustrated in Figure 21, the three basic components of a decision model are: (1) the alternatives that are available (what you can do), (2) objectives and preferences (what you want), and (3) information and beliefs about what will happen depending on the alternative that is selected (what you know, believe or suspect).

Components of a decision model

Figure 21:   A project-selection decision model represents the 3 components to any decision.

To support decision-making, a decision model needs to capture understanding about (1) what might likely happen depending on the alternatives that are selected, and (2) how desirable or undesirable those consequences or impacts would be. The model is expressed in terms of variables and mathematical relationships.

If the decision model is a project-selection decision model, it will likely take as input characteristics of the project, the project's anticipated impacts, and other factors, and produce as output the value that would be added by doing the project. As indicated in Figure 22, such models generally consist of two sub-models. The first (labeled "simulation") specifies or generates the relevant consequences of doing the project. This sub-model is a "consequence model"; it captures or predicts the possible consequences of selecting alternative projects by incorporating the facts, judgments and uncertainties relevant to project selection. The second (labeled "valuation") computes the value of those consequences. Termed a "value model," its purpose is to evaluate the potential decision consequences by incorporating decision-maker values and value tradeoffs. Since the value of a project is what the organization would logically pay to obtain the project consequences, value is expressed in dollar (or other currency) units. The model may be used to identify those project choices that maximize value added.

Components of a decision model

Figure 22:   A project-selection decision model values projects in two steps.

Decision models are created the same way that other types of models are constructed. The variables to use in the model are selected, the mathematical relationships among those variables are determined, and the parameters of the model are quantified based on available data together with judgment. The process is as much art as science, however, in the case of decision models, step-step-by-step model-building processes have been developed. Most of these model-building proceses have evolved as ways to implement established theories and approaches for decision making, including the analytic hierarchy process (AHP), cost benefit analysis, ELECTRE, and multiattribute utiity theory (MUA) (I have more to say about these later). The table below summarizes the model-building steps that I most frequently use.

Steps for Building a Project Selection Decision Model
Model Component Step Description
Value model Structure objectivesObjectives should be fundamental, measurable, and observable, with no overlap or double counting
Specify performance measures Measures or scales for indicating the degree to which each objective is achieved
Consequence model Identify driversFactors that influence or determine project performance against objectives; influence diagrams are useful for this purpose.
Choose model form For relating drivers to performance measures; may be deterministic (causal model) or probabilistic (e.g., a decision tree)
Build consequence model Construct model per the techniques and process recommended for the selected model form
Value model Elicit scaling functionsFor translating performance improvements to the value of those improvements
Define the aggregation equation The value function for combining the single attribute value functions
Elicit weights Tradeoff weights indicating the relative value of achieving each objective
Combined model Implement model In software (e.g., Excel) for convenience

The above sequence of model-building steps is based on MUA (also called multi-attribute decision analysis or multi-objective decision analysis—MADA or MODA). The step-by-step processes for implementing othrer decision making approaches are fairly similar. In nearly all cases, the decision model is constructed "from the top down," beginning with the definition and structuring of objectives.

Structuring Objectives

Structuring objectives means appropriately defining, characterizing, and organizing the objectives for conducting projects into an objectives hierarchy. The objectives in the hierarchy define the ways in which projects might create value, since a project that contributes to the achievement of any objective produces some amount of value depending on how much that particular objective is advanced and the importance of advancing that objective. The key is to structure the objectives hierarchy in such a way that the mathematical equation for correctly aggregating the various value components (the aggregation equation) can be derived from the hierarchy.

Procedurally, the process for constructing an objectives hierarchy begins with identifying the organization's highest-level, most fundamental objective. Then sub-objectives that specify or clarify how that fundamental objective may be achieved are identified and added. Typically, objectives are stated in terms of maximizing some desired, or minimizing some undesired, object of interest. The process of identifying and adding sub-objectives continues to the extent necessary to produce a comprehensive and sufficiently detailed mapping of how projects can create value.

For example, suppose the organization's fundamental objective is maximizing shareholder value. As shown previously (Figure 19), shareholder value may be regarded as the net present value (NPV) of the company's projected cash flows plus option value. Thus, a shareholder value objective could be split into two sub-objectives: (1) maximizing the NPV of future cash flows and (2) maximizing option value. Relevant sub-objectives for increasing NPV would include (1) increasing future revenue and (2) decreasing future costs. Sub-objectives for maximizing option value depend on the organization and the nature of its business, but generally such sub-objectives involve achievements that would increase the ability of the organization to identify and take advantage of future opportunities and avoid future risks, as well as other sub-objectives that, if achieved, would enhance investor expectations and, therefore, increase the market value of the company.

Sample objectives hierarchy

Figure 23:   Sample objectives hierarchy for shareholder value.

If, on the other hand, the fundamental objective of the organization is maximizing stakeholder value, then the objectives hierarchy may be established by identifying who the stakeholders are and then identifying and structuring their objectives. An example of an objectives hierarchy for a case wherein the organization's fundamental objective is to maximize value for stakeholders is provided in a following sub-section.

Much has been written about how to define, characterize, and structure decision objectives so that the mathematical form of aggregation equation for combining value components can be derived. Basically, the goal is to specify objectives that are (1) fundamental (a common error is to specify as an objective something that is only one means for achieving what is really desired), (2) well defined (such that people readily agree on the meaning of each objective), (3) measurable (it is possible to know and measure how well each objective is being achieved), (4) comprehensive (the set of objectives is sufficiently complete to capture all of the strengths and weaknesses of available alternatives), and (5) non-redundant (at any specified level of the hierarchy, there is no overlap or double counting among the objectives at that level). A serious, and all-too-common error made by those designing project selection and prioritization models is to assume that value components can be weighted and added when an analysis of the associated objectives hierarchy would prove that this is not the case (more discussion on this is provided later).

Characterizing Distinctions

Once objectives have been structured, the next step for constructing a decision model is specifying measures for characterizing the distinctions that determine how well the objectives are being achieved. The distinctions that we seek to identify are the attributes or characteristics that we care about. For example, suppose the context is decision making by a utility that distributes electric power. Customers (and therefore the utility) care about power outages, so one distinction that matters when thinking about electricity delivery is whether or not power is available to the customer. Any measure we define must have at least two possible states, for example, "yes power" or "no power," but we can define measures to have many more states if it helps us to differentiate alternatives and measure how well we are achieving our objectives. For example, the measure for electric power to the customer could be defined as the average annual number of hours and minutes that the customer is without power. If there are additional sub-objectives relevant to providing electric power, measures may be defined for those as well. For example, if customers should be spared from brownouts, the voltage available from the customer's power outlet, which, for example, could be zero, 110 volts, 109 volts, 108 volts, etc.

As illustrated by the above example, multiple measures may be required to characterize with adequate precision the multiple sub-objectives that determine the achievement of a particular decision objective. In general, sub-objectives and associated measures should be defined whenever (1) the degree to which the objective is achieved depends significantly on performance against that measure, (2) performance against the measure doesn't track or exactly correlate with performance against some other measure that has already been defined (i.e., you need the additional measure to distinguish possibilities), and (3) available understanding is sufficient to conclude that the level of performance against the measure will differ depending on the project choices that are made (i.e., the measure differentiates among the alternatives).

Measures must be defined for each of the lowest-level sub-objectives in the objectives hierarchy. For example, customers care about the accuracy of their bills. Thus, the percentage of customer invoices with errors might be a relevant measure for an organization considering projects for improving customer billing. As an example relevant to measuring option value, customer brand loyalty might affect investor expectations. A possible measure for impact on brand loyalty might be the percent of customers making repeat purchases or the company's ranking in customer satisfaction surveys.

The selection of measures is one of the more creative aspects of model building. In addition to choosing measures that quantify the achievement of the objectives, measure selection should be based in part on the types of data the organization does or can collect to assess performance. Influence diagrams (illustrated in a following sub-section), can be very effective for identifying useful performance measures.

Describing Possibilities and Likelihoods

Measures that characterize the distinctions that matter provide the framework needed to capture project consequences. To illustrate the concept, imagine that we combine the measures that we've defined into a tree structure. The tree starts with a set of branches indicating the possible states for the first measure, which are connected to branches for the second measure, and so forth. The paths through the tree define all of the possibilities, and that set of possibilities is the same regardless of the order in which the tree has been drawn. Some of the possible combinations will, of course, be more desirable than others, and it will be the job of the value model to sort this out (see the next section). The job of the consequence simulation model is to capture how the choice of project alternatives affects the likelihood of the various performance combinations (paths through the tree).

Sometimes, a project choice that we make will ensure that we end up on a specific branch for some measure (reflecting a specific outcome or level of performance for that measure). For example, if the electric utility from the example above conducts a project to add a distribution line that will reach 100 new homes, and the number of residential customers receiving service is a measure of performance, then, assuming the project is successfully completed, the number of customers will be incremented by 100. If there is uncertainty about project consequences, a choice merely affects the likelihood of the various branches for that uncertain measure (e.g., the number of new customers that will be added is somewhere between 0 and 100, depending on how many of the new homes in the housing development that the line will serve will be built and sold to homeowners).

The above illustrates that one way to capture the relevant beliefs about the consequences of choices is to assign probabilities to the possible performance outcomes that correspond to the paths through the tree of possibilities. We could do this by assigning probabilities to the branches for the first measure, conditional probabilities to the branches for the second measure (that indicate how likely each level of performance is for the second measure given each possible level of performance for the first measure), and so forth. If we then multiply all of the probabilities along any path, we will obtain the probability of that scenario. Since the goal of the consequence model is to express our beliefs about how choices affect our ability to achieve the distinctions we care about, a modeling approach based on assigning probabilities to the branches in a tree of possibilities would provide a fully adequate consequence model.

Creating a Consequence Model

The purpose of above description is not to suggest that developing a tree structure and assigning probabilities is always the best way to construct a project consequence model. Rather, it is to demonstrate the basic problem that must be solved and to illustrate that there is at least one solution approach that will always work. Consequence models built on tree structures (e.g., decision trees and event trees) are, in fact, quite common. For example, for evaluating pharmaceutical projects, tree models are typically used to lay out the likelihood of success and timing and to simulate the outcomes of the various tests and regulatory approvals that are required.

There are, however, many other model forms that may be used to infer or simulate project consequences. The simplest approach (and perhaps the most common) is direct estimation (What do you think will happen if the project is conducted?). A popular approach for the case where data are available on the outcomes of similar projects is to use statistical analyses to establish correlations and suggest cause-effect relationships between project characteristics and project consequences. Sometimes sophisticated simulation models are available for estimating project consequences. Many states and municipal governments use transportation simulation models to predict the impacts of transportation projects on traffic congestion, travel times, pollutant emissions, and other consequences of concern. The models may simulate the paths of individual vehicles along a particular road, or operate at a more macroscopic level, representing, for example, the speed, flow and density of traffic in the various sections of a transportation network.

Importantly, there are many existing models for measuring performance against numerous types of business and organizational objectives. For example, perhaps the most common sub-model used in project evaluation is the financial, or "business case model". The typical project financial consequence model computes the NPV of cash flows from the project (free cash flows), accounting for the timing of incremental costs and revenues, taxes, the time value of money, and other considerations that affect the equivalent, current financial worth of the cash flows to the business. Thus, project consequence models typically include a cash-flow NPV model as the sub-model for measuring the project's direct financial benefits.

Other models have been developed for capturing understanding regarding many different processes, activities, and impacts relevant to evaluating project impacts on business objectives, and these models can likewise be utilized. For example, there are well-established models for estimating R&D success, the impact of project characteristics on customer choice, the rate at which new products penetrate exiting markets, the effectiveness of projects at reducing pollution, the productivity and reliability of business assets, and so on and so forth. Selecting, customizing, refining, and piecing together various sub-models that have worked well in other applications is the most common approach for creating the consequence model component of project-selection decision models.

Valuing Consequences

Project consequences must be valued in a common unit of measurement; otherwise they can't be combined to obtain a total project value. Since financial gain is an expected consequence for many projects, dollars (or another unit of currency) is the obvious choice for valuing all project consequences, including non-financial benefits. Because people regularly use money to make transactions, it is easy to interpret worth in monetary units. Also, unless we express project value in the units used to measure costs, we won't be able to determine whether project benefits justify project costs. Money is the clear choice for the unit for measuring project value.

One approach for determining the monetary value of potential project consequences would be to ask the organization's leaders to provide the numbers directly. However, the task would be difficult and it would be hard to obtain consistent answers. Thus, methods based on formal theory and analysis are used instead. The available theories and methods for quantifying project value are presented in the next section of this paper.