Implementing project portfolio management

Project Valuation Metrics

Once you have a project-selection decision model, it is easy to specify metrics. The desired metrics are "observables" (discussed below) that influence the model's value drivers; that is, those project characteristics and impacts (i.e., model parameters) that have the greatest influence on value. These typically include forward-looking financial metrics, like NPV, but also factors and considerations on value paths that don't directly impact cash flows. Building a decision model leads to metrics that capture the variety of ways that projects contribute value.

In my experience, a well-designed decision model will identify metrics for measuring some or all of the following types of value:

Financial value. Metrics are needed to account for any increases in revenues or reductions in costs that may result from conducting a project. Typically, the appropriate financial metrics are the incremental cash flows attributable to conducting the project, or standard financial metrics that are derived from such cash flows (see the next subsection for more discussion). Revenue generated from projects to develop new products is often estimated with the aid of sub-models that simulate the various development and commercialization stages, in which case metrics indicating the likelihood of success at each stage are also included.
Health, safety, and environmental (HS&E) value. If some projects impact the health and safety of workers or the public, or the natural environment, metrics may be needed to account for such impacts. The field of risk assessment provides many well-established metrics for this purpose. Typically, the desired metrics indicate the scope, nature, likelihood, and seriousness of the health, safety, or environmental impacts.

Customer value. For organizations that sell or otherwise provide products and services, metrics for estimating the impacts of projects on customers are needed. The field of economics provides various metrics for measuring customer value, including the concept of consumer surplus. The field of market analysis has developed numerous customer-satisfaction metrics. Another common approach is to adopt metrics that describe specific product or service characteristics that customers care about (e.g., attributes of product and service quality and price). such models may include custoemr choice and market penetration models for predicting the dynamics of new sales.
Stakeholder value. In additional to customers, organizations typically have other stakeholders whose attitudes and perceptions are important and should be managed. Examples include organized labor, regulators, business partners, etc. The relevant metrics typically depend on the type of stakeholder and ways in which the attitudes of those stakeholders are typically expressed or impact the interests of the organization. Oftentimes, metrics for shareholder value consist those metrics needed to identify and characterize the importance of the concerned stakeholder group and metrics that indicate the anticipated reaction of those groups.
Mission value. Public sector organizations aren't the only ones that have missions. Many private sector organizations have adopted mission statements that identify goals beyond maximizing shareholder value. The appropriate metrics in this case measure the contribution of proposed projects to the achievement of these goals.
Community socio-economic quality. Some organizations conduct projects that significantly impact local communities. For example, a project to build a new manufacturing plant might have a significant impact on community jobs. The fields of sociology and economics provide numerous metrics potentially useful for such situations.
Option value. Consistent with the theory of real options, projects that contribute to the organization's platform for future success provide a source of value. For example, an IT (information technology) project may give the organization new capability. An R&D project may provide new knowledge or understanding important to the business. Also, a project may have a distinct, strategic value. Metrics may be defined that capture these additional sources of value. The key is not to double count.

As suggested by the above, numerous metrics may be needed to capture all of the potential components of project value. If metrics representing some sources of value are omitted, the value of projects will be underestimated. Furthermore, there will be a bias against doing those projects that provide the types of value that are not captured due to the omitted metrics. Note that metrics need to represent timing, that is, when the project benefits are likely to occur and how long they will persist, and, oftentimes, risks (e.g., the likelihood that the project will actually produce its anticipated benefits). When all such metrics are specified, the decision model defines the algorithm that allows the value of a project to be expressed in dollar terms.

Obviously, there is a limit to how many project evaluation metrics should be used. The 80/20 rule applies. The goal should be to include the minimum number of metrics necessary to roughly capture every significant source of project value, not numerous metrics that more completely capture just a subset of components of value. In other words, don't make the mistake of defining multiple, potentially redundant metrics for capturing things that are relatively easy to address (like financial value), while omitting metrics for something that may be important but hard to address (like impact on learning and capability). Since few if any projects will provide significant contributions under each type of value, having lots of metrics doesn't necessarily create a significant burden for evaluating proposed projects. Estimates need only be provided for the subset of metrics that are relevant to capturing the specific motivations for doing that project.

Metrics as "Observables" and the Clairvoyant Test

To the extent possible, metrics should be observables; that is, characteristics of projects or project outcomes that can be observed and measured in the real world. Since estimating project value requires forecasting the future, metrics don't, obviously, all have to be things we can observe today. Metrics can, for example, include a projected future state of some observable, for example, an improvement in a reliability-of-service statistic important to customer satisfaction.

Clarivoyant test

A useful device for checking whether a metric is observable is the so-called "clairvoyant test" devised by my college mentor, Professor Ron Howard. Before accepting what appears to be a good metric, consider whether a clairvoyant could give an unequivocal value for that metric given that a project decision is made in a specific way. Oftentimes, the clairvoyant test points out inexactness of what initially appears to be a well-defined metric. For example, "customer satisfaction" doesn't pass the clairvoyant test. However, "percent reduction in recorded customer complaints" and "company ranking in the next industry customer satisfaction survey" are metrics that do pass the test.

Metrics that don't pass the clairvoyant test are vague. They create inconsistency and imprecision when used for estimating. More importantly, if the metrics are not observables, they cannot be monitored so that actual values can be compared against estimates.

Financial Metrics

The traditional financial metrics should be used to determine the direct financial components of project value. Project investment cost is, of course, an important financial metric for any project. Projects that impact operations (e.g., projects that create new revenues or that affect future operating costs) produce downstream financial impacts that must also be evaluated. Thus, any and all significant, incremental, period-by-period cash flows that are anticipated to result from projects should be estimated, either as a most-likely or average case or in the form of alternative scenarios. The organization's standard accounting model may then be used to determine the resulting after tax, or unencumbered "free" cash flows, which may be used to compute a project's financial NPV.

Some important principles for estimating financial value in support of project prioritization include:

Ignore previously paid, sunk costs.
Include opportunity costs (the opportunity cost of a resource is the value of the net cash flow that could be derived from it if it were put to its best alternative use).
Include overhead expenses (e.g., administrative expenses, managerial salaries, legal expenses, and rent) that are directly related to a project. Indirect overhead can, if necessary, be prorated across proposed projects.
Include "spill over" effects. For example, if a project introduces a new product or service that draws sales from existing products, include such lost revenue in cash flow estimates.
Interpret expected project cash flows submitted in support of a project proposal as commitments to be achieved by the project manager. If there are cash flow components that are more speculative or for which the project manager cannot be held accountable (e.g., because they are contingent on events beyond the control of the project manager), specify such cash flows separately and assign probabilities.
Identify and include any terminal cash flows, for example, cash flows expected from the disposal of assets at the conclusion of the relevant product or service lifecycle.
Be consistent in accounting for inflation. For example, using an inflation-adjusted discount rate while ignoring inflation in estimating cash flows would result in a bias against accepting projects.
For the purposes of prioritizing projects, remember that the project's financial benefit is its NPV exclusive of its current-period costs.

Be suspicious of long-term, positive NPVs. Keep in mind the economic axiom that excess profits (the source of positive NPV) must be zero in a perfectly competitive market. A long-term, positive NPV requires some sustainable competitive edge—being first, being the best, or being the only. Retaining that edge indefinitely would require some barrier to the entry of competitors. Consider carefully how long it will take competitors to catch up and drive profits back down.

Metrics Provide Justification for Tough Choices

One of the most under-appreciated benefits of having good metrics linked to a defensible decision model is improved justification. Author Anthony O'Donnell quotes a portfolio manager at an insurance company that implemented a portfolio management tool: "People would come to me and ask me to do a particular project...I would tell them I couldn't fit it in, but had a hard time articulating why." Metrics now allow him to give concrete reasons for turning away projects. "Their satisfaction immediately went up, and I still didn't do their projects!"[9].

Each Organization Needs Its Own Metrics

Different organizations conduct different types of projects. The metrics for evaluating new product investments by a software vendor, for example, will be different than the metrics needed to evaluate process improvements for a company operating an oil pipeline. Also, different organizations create value in different ways. An electric utility, for example, creates value differently than does a ballet school. Some organizations will seek to maximize shareholder value, while others will want to value impacts to other stakeholders as well. Thus, each organization will have a different model for how its projects create value and, therefore, will want to use different metrics. There is no one set of project metrics that works for every organization. However, in all cases, good metrics provide a means for computing the value added by projects. Good metrics are observables. And, they are sensitive to project decisions so that they may be used to differentiate the value of alternative project portfolios.

Smart Metrics Decouple Performance Assessment from Project Valuation

A common project prioritization problem is what to do if the magnitude of the benefit available from a project depends on the other projects that are included in the portfolio. Here's an example. The US Department of Energy (DOE) is funding a portfolio of projects designed to increase electricity generation from alternative fuels (fuels other than fossil fuels). With a simple rate-and-weight project prioritization tool, the DOE could "score" the benefit of, for example, a solar project, but the amount of benefit delivered by the solar project would be less if the project portfolio includes investments in nuclear (because nuclear would likewise increase generating capacity, making the additional incremental capacity from solar less valuable). It seemed that it would be necessary to score various combinations of solar and nuclear investments, but this would be very time consuming and make it difficult to maintain consistency.

The solution is to choose a performance metric that depends only on the project in question. The obvious such metric for this case is the amount of new generating capacity delivered by the project. The value of a increment of new capacity can then be computed based on the total available capacity, accounting for the incremental capacity delivered from all other projects contained in the candidate portfolios. This approach requires that the tool provide the capability to apply non-linear value models and identify the optimal portfolio via optimization (capabilities that simpler tools often lack), but the result is an evaluation approach that requires far fewer and simpler inputs. As illustrated, smart metrics (that is; good modeling practice), often provides a way to address what might otherwise appear to be impossibly complex project interdependencies.

The Right Metrics Turn Project Proposals into Performance Contracts

Deriving metrics from a decision model ensures that the organization is seeking the right information about proposed projects; namely, the information necessary to estimate the value to be derived by the organization if the project is conducted. The value of the project can then be compared with its cost, and the resulting "bang for the buck" compared with similar estimates for other candidate projects. This provides a sound basis for making project-selection decisions.

If, in addition, the metrics are observables, the organization further benefits in that project proposals serve as performance contracts. In return for a chance at obtaining a share of the organization's limited resources, project proponents indicate in the clearest and most relevant terms what they expect the project will accomplish. Project results and impacts can then be tracked and compared with the original estimates. Performance contracts document the terms of the agreement, protecting both parties to the contract. Framing the project as a performance contract creates a healthy shift in perspective. Instead of choosing which projects to cut, the focus is on deciding what project opportunities to purchase.

Due to uncertainty, project outcomes may not exactly match forecasts. Thus, what the implicit contract requires is not that project managers invariably be held responsible for achieving all of the performance indicated by their estimates, but that any significant deviations between estimates and actuals be explained. Over time and on average, some projects should exceed expectations while others will fall short. In the meantime, the organization can learn to improve forecasts by tracking and better understanding the uncertainties that are involved.

In situations where the uncertainty is considerable, it may be useful to separate metrics for indicating the benefits that can be expected from benefits that are more speculative and uncertain. The "expected" benefits then become the basis for the performance contract and the speculative benefits can be appropriately discounted based on risk (see Part 4).

References for Part 3

E. Goldratt, The Haystack Syndrome, North River Press, 1991.
R. Foti, "Priority Decisions," PM Network, 16 April, pp. 24-29.
The recommendation to develop a decision model and, more generally, the views and ideas expressed in this and the next part of this paper are shared by many decision analysts. See especially "Choosing the Right Metrics for Measuring, Monitoring, and Maximizing Shareholder Value," C. Spetzler and R. Arnold, www.sdg.com, May 2003. The book Value Focused Thinking by R.Keeney, 1992, describes many of the concepts and techniques for building decision models.
Stanford Center for Professional Development, "How Value-Based Management Delivers Vastly Improved Portfolio Results," Strategic Decision and Risk Management Webinar XSDR001, Stanford University, April 1 2009.
R. Kaplan and D. Norton, The Balanced Scorecard, Harvard Business School Press, 1996.
These quotes were found by searching "strategic alignment" on Google, and were taken from a Project Management Institute project portfolio management seminar brochure, an article entitled "From Crisis to Control: New Standards for Project Management," , and a presentation from a consulting firm. You can find similar statements all over the web.
This description of how to rank projects based on strategic alignment is from pages 140-143 of "Chapter 4.1: Linking Strategy and Project Portfolio Management," by K. C. Yelin, in H. Levine's book Project Portfolio Management, Wiley, 2005.
Ackoff, Russell, as quoted by Stephen Miller, "A Management Philosopher with Heady Ideas About Beer," Obituaries, The Wall Street Journal, November 11, 2009.
A. O'Donnell, "Worth the Effort," Insurance Technology, March 4, 2003.

The Scientific Method

Part 4: Inattention to Risk