Lee Merkhofer Consulting Priority Systems
Implementing project portfolio management

Part 3:  Lack of the Right Metrics


Metrics imply incentives

Metrics = Incentives

The metrics that an organization uses to evaluate projects have a big impact not only on the projects that get chosen but also on the projects that get proposed. "Tell me how you will measure me, and I will tell you how I will behave" [1]. Even if the metrics aren't intended to create incentives, managers interpret them as indicating what the organization regards as important. Adopting and applying metrics that promote the development and selection of projects that best serve the organization can have a real impact on organizational success. Conversely, lacking metrics or using the wrong metrics can be a serious problem. Lack of the right metrics is the third reason organizations choose the wrong projects.

Defining Project Value

The metrics for evaluating projects must support the main goal of project portfolio management (PPM); namely, maximizing the value of the project portfolio. Thus, we need metrics and methods for measuring project and portfolio value. How do you measure project value? More fundamentally, how do you define value? These questions are more perplexing than you might at first think.

The Utilitarian View of Value

Aristotle

People have been arguing about the definition of value for centuries. The relevant concept of value for project selection is termed the "utilitarian concept of value:" The value of a project is a measure of the degree to which that project enables the organization to achieve its objectives. This view of value was first articulated in the fourth century BC by Aristotle —the value of something is not an intrinsic property of that thing, but rather is determined by its usefulness to those who want it.

Organizations conduct projects because they believe that the outcomes or consequences of the projects will be useful. The more useful the project outcomes, the more valuable the project will be. The project consequences that are desired depend, of course, on the organization's objectives.

Suppose, for example, that a company's fundamental objective is to create value for its shareholders (more discussion of shareholder value is provided on the next page). Suppose further that the company is considering a hypothetical project that provides only one consequence: an immediate, one-time cash infusion to the company of one million dollars (to be more precise, suppose that the magnitude of the cash flow from the project is such that, after all tax and accounting considerations are addressed, increases the net worth of the company by $1 million). Under these assumptions, it would seem natural to conclude that the value of that project is $1 million.

Defining Value based on Willingness to Pay

An extension to the above line of reasoning, which applies regardless of what the organization's objectives might be, is to argue that the value of a project is what the organization would be just willing to pay for the consequences of the project. Stated more precisely, project value might be defined as the maximum amount of money that the organization's most senior executives would be willing to pay to obtain the project's consequences, without having to pay the project's costs and including consideration of risk. (In other words, if project consequences are uncertain, project value would be defined as the maximum amount of money executives would be willing to give up in exchange for owning the uncertainty over the project's consequences.) In the portfolio context, the value of a project would be interpreted as an amount of cash that would make executives indifferent between (a) the portfolio without the project and (b) the portfolio with the project's consequences but with a debt equal to that amount of cash.

Indifference Prices

In the literature on valuation, the above definition of value is termed the buying indifference price (also called the breakeven buying price). This term applies because at this price the organization would be indifferent between not getting the project consequences and getting the consequences but having to pay the price (the breakeven price at which the company believes it would neither gain nor lose, but simply break even if it obtained the project consequences at that price). A close relative is the selling indifference price (or breakeven selling price), which argues that the value of a project is the amount of cash that would make executives indifferent between (a) the portfolio with the project consequences and (b) the portfolio without the project but with that amount of additional cash.

Complexities for Determining Indifference Prices

Though indifference pricing may seem non-controversial, at least in theory, the concept is less straightforward when you think about the impact that paying a buying indifference price or obtaining a selling indifference price has on other decisions. Suppose I ask you to tell me the most that you would be willing to pay to buy the consequences of project A. If you consider (as you should) that buying project A's consequences means that you won't have the resources needed to buy some other project B, the true cost of buying A is not getting the most valuable B that you could have bought. Losing the opportunity to buy project B is the opportunity cost of buying project A. With opportunity cost reasoning, the most an organization should be willing to pay to obtain a project should be less if doing the project means foregoing other, very good investments than when it would mean foregoing only marginal investments.

The situation gets more complex still in the portfolio context. If project buying prices are determined sequentially, the remaining resources available for buying projects declines as projects are added to the portfolio. Furthermore, the value of obtaining project outcomes is likely to change as you commit to doing other projects and, therefore, anticipate experiencing the consequences of these other projects. Thus, a troublesome implication of indifference prices, when opportunity costs are considered, is that a project's indifference price logically depends on the order in which projects are added to the portfolio. If the indifference price for a project is determined based on it being the first project in the portfolio (or if its value is determined in isolation to other projects), logic says we should get a different value than if we assume it is being added to other portfolio projects. Also, project indifference prices ignoring other projects will not sum to the indifference price for the project portfolio (however, it can be shown that if you sum sequential project indifference prices the result will equal the indifference price for the portfolio as a whole).

Determining Indifference Prices "On the Margin"

In practice, in my opinion at least, the above complexities need not be of major practical concern. Organizations obtain capital from a variety of sources and spend that capital on numerous different investments. Typically, each individual project in a project portfolio consumes a relatively small portion of the organization's total capital. My experience is that executives asked to estimate a project's indifference price give little, if any, thought to the specific opportunities that would need to be foregone if that amount was actually spent. From a practical standpoint, it is impossible to know either the true opportunity cost of paying a buying indifference price or the opportunity value generated from receiving a selling indifference price. Most projects can be evaluated "on the margin," with the assumption that the value the project will provide can be assessed without worrying too much about what the impact will be on other activities.

Make Sure Projects are Properly Defined

Before saying much more about the definition of project value and the metrics for measuring value, it must be acknowledged that projects need to be defined in such a way that they can be valued. If the usefulness of a project depends to a significant degree on the other projects that are in the project portfolio, then it will not be possible to assign a unique value to it. Fortunately, I've found that projects to be prioritized can usually be specified in a way that allows them to be valued.

Define Projects to be Independent of One Another

The exception to the assumption that projects can be valued individually and on the margin occurs when projects are interdependent, which is typically the case when a collection of related projects are designed to be conducted together. For example, if one project is defined to be the purchase of a desired asset and another project is to use that asset to achieve some desired end, then it would not make sense to try to assess the value of the projects in isolation of one another. In such cases, the obvious solution is to combine the interdependent projects into one larger project such that the larger project is independent of other projects. My advice to organizations implementing PPM is to establish a policy of defining projects in such a way that each project encompasses all the work needed to achieve the ends that justify the work.

Define Multiple Versions of Large Projects

The only problem with combining interdependent projects into larger independent projects is that the approach can lead to "all-or-nothing" choices for some important undertakings. The cost of a large project composed of many interrelated tasks might be so great that it doesn't compete well with other projects based on its ratio of value to cost. To avoid all-or-nothing choices, the project proposer can be asked to submit one or more lower cost "versions" of the large project. In one situation that I can recall, a project portfolio manager established the requirement that whenever a project is defined with a cost estimated to be over a specified dollar threshold, the project proponent was required to submit at least one lower-cost version of the project with a price tag less than the threshold amount. Even if the organization lacks a means for quantifying project value, project proponents understand the concept that a reduced scope version of what they would like to accomplish may have a higher value-to-cost ratio. As you would expect, I've found that most project proponents are eager to define lower-cost versions of their projects to avoid the risk that none of the elements of what they want to do will be funded. Whenever multiple versions of a project are defined, then the version with the highest cost that makes the budget funding cut off is recommended for funding.

This same approach works in the case where the smaller project components of a large project can stand on their own merit, but where there are synergies if they are conducted together. For example, project A and project B might be interdependent, but project A and project B have some merit on their own. Then various collections of the interdependent projects can be considered, for example, project A by itself, project B by itself, and project A and project B together. Similarly, if the value of a project depends to a degree on many other projects being conducted, its value may be assessed based on some most likely scenario regarding the other projects that will be conducted. If, at the point in time when a go versus no-go decision for the project is needed, the earlier assumption about which other projects will be conducted proves wrong, the project whose value depends on those other projects may be reassessed using a more accurate assumption about the other projects to be included in the project portfolio.

A Working Definition for Project Value

Let's return to the original question of how to define project value. I've found that most people find it intuitive that the value of something is the amount of money viewed equally desirable to that thing. Accordingly, my working definition for project value is:

Project value

=

  The amount of money the organization's executives would prefer equally, at present, to the uncertain future consequences of conducting the project

(Eq.1)

If the equivalent monetary amount changes significantly depending on other choices that the organization makes, then, as stated above, contingent values may be specified that depend on those other choices.

Notice the similarity of my definition of project value to the classic financial metric for valuing projects, net present value (NPV). The project NPV is the amount of money, at present, that is of equal worth to the project's incremental cash streams. My definition is likewise an equivalent worth, but consideration is also given to project consequences other than impacts on cash streams. Notice also that my definition is a variation of the selling indifference price—it is a selling price because it is an amount the organization would need to be compensated to voluntarily give up ownership of the project's consequences. It is an indifference price because it is the price point such that the organization is indifferent between obtaining the price in return for giving up the project consequences.

Project Value Does Not Depend on Project Cost

Note that the value of a project, defined in this way and under these assumptions, does not depend on its cost. Projects that cost more typically accomplish more, but the value of a project is determined based on the worth of the project outcomes without regard to what it costs to produce those outcomes. The decision of whether to conduct a project, does, of course, depend on cost. In particular, you would never want to conduct a project whose cost is greater than the value of its consequences. When an organization pays the cost of a project, the amount of value obtained is the net value (value minus cost), but thinking in terms of net value confounds discussions of value. For example, if a friend shows up at your home with a new automobile, you wouldn't think that the value of the automobile changes if you learn your friend received it as a gift. Although the net value of a project depends on cost, value does not.

"'Price' is what you pay, 'value' is what you get." Warren Buffet's warning to investors applies to project decisions as well—project value and project cost are sperate and distinct considerations.

An exception to the rule that project value does not depend on project cost would be a case where paying to conduct a project impacts the organization's ability to benefit from the project or from other projects (in other words, a situation where the opportunity costs can't be ignored). An extreme example would be a project anticipated to produce great project outcomes, but whose cost would bankrupt the company. In such cases, the value of a project would logically be the value of the project consequences taking into account the effects of having to pay for the project on the organization's ability to benefit from the project consequences. However, as indicated above, most organizations conduct numerous projects each of which consumes only a small portion of the budget, so we can typically ignore such considerations.

Project Value Exactly Maps to Organizational Preferences

Project value

Project value is the worth to the organization of the (possibly uncertain) consequences of doing the project—in other words, it is the maximum price the organization would pay to obtain the project's consequences

My definition of project value has a critically important property—it maps exactly to organizational preferences. Suppose there are two projects requiring the same cost and resources, and only one can be added to the project portfolio. The organization will prefer project A to project B if and only if it judges the value of project A to be greater than the value of project B. If the organization were to violate this logic and chose project B even though it valued project A more highly, then, it could be persuaded to pay a fee to trade project B for project A. In other words, violating the value judgment means the organization could be used as a "money pump." Under any definition of rationality, such behavior would not make sense.

Although the mapping between project value and project preference may seem trivial, note that criteria routinely recommended by many other authors cannot be said to have this essential property. For example, if one project has a higher strategic alignment score than another, that doesn't guarantee that, other things being equal, the organization will necessarily want to choose that project. Unless the project evaluation measure is defined in a way that maps to the true preferences of the organization, that measure won't correctly prioritize projects. My definition of project value is a true measure of the relative attractiveness to the organization of the consequences available from its project alternatives, and, because the measure of value is expressed in financial units, project value can be compared with project costs to determine which projects are worth doing and their priorities.

OK, assuming that you agree that the worth to the organization of the project consequences is an appropriate definition of project value, how do we determine how much cash an organization would deem of equal value to the anticipated consequences of a project? Before addressing that question we need to consider more carefully why organizations conduct projects.

Projects Determine the Evolution of the Business

The business of an organization is always evolving, and the projects that the organization chooses affect that evolution. For example, a new technology might become available that would allow the organization to reduce its costs. If a project is conducted to install the technology, the organization would incur lower operating costs than it would have if it had chosen not to do the project. Achieving a significant step change in operating costs would, obviously, dramatically change the course of the business.

It is also true that the projects that an organization chooses not to conduct affect the evolution of the business. For most organizations, standing still means falling behind. There are many reasons for this, including increasing competition, changing customer preferences, and aging of the organization's assets. Thus, to take one example, if the organization chooses not to do projects that maintain or replace aging assets, the service provided by those assets will decline.

Figure 18 illustrates a useful way to think about project value. At the point in time when an organization is considering a new project, it is really facing a choice between two alternative futures. If the project is conducted, that project will, presumably, transform the business to some more desirable state. If the project is not conducted, some other, presumably less-desirable, state will result.


Projects determine business evolution.

Figure 18:   Project choices determine the future state of the business.



The perspective of Figure 18 provides another basis for computing project value. The value of a project is the difference between the values of two potential future states of the business:

Project value  =  Value with the project - Value without the project

(Eq. 2)

Some Observations Regarding Project Value

Based on Equations 1 and 2, we can conclude some important things about project value.

  • To estimate project value, it is necessary to consider what would happen if the project is not conducted as well as what would happen if the project is conducted. Many project scoring systems ignore the implications of not doing projects, which causes some types of projects to be grossly undervalued.
  • The same project can have different values to different organizations. For example, if an R&D pharmaceutical company discovers a way to create a new molecule with therapeutic potential, an obvious follow-on project would be to complete the development of a drug based on the molecule. However, if another company is better suited to develop the drug, it may make sense for the company making the discovery to sell the project opportunity to the other company. Thus, for the purposes of estimating project value it is not sufficient to consider just the characteristics of the project itself. It is also necessary to consider characteristics of the business that determine how useful that project will be to that business. If the project opportunity is marketable, then the market value of the project sets a lower bound on project worth.
  • The value of a project can change depending on what other projects are conducted. As suggested above, because of synergies and economies of scale, for example, the costs and benefits of a project can change based on the other projects that are also conducted. If such dependencies exist, optimizing the project portfolio requires estimating the value of conducting different groupings of the interdependent projects. The key in this case is to define the groupings so that they are independent of one another.

For the purpose of measuring project value, an important implication of Equation 2 is this: Project value is the contribution that the project makes to the total value achieved by the organization. Because project value is the difference between the value that the organization attains in two alternative states, the methods for estimating project value are essentially the same as the methods for estimating organizational value. This is good news, since management scientists have devoted considerable effort to determining how best to measure the value created by organizations. We can use the concepts and methods that they have devised to help us quantify project value.

Inadequacy of Financial Metrics

So, how can you measure the value created by an organization? Interestingly, most businesses get it wrong.

Most businesses use financial metrics computed from cash flows to measure value, for example, return on investment (ROI), internal rate of return (IRR), net present value (NPV), payback period. Using these metrics to evaluate candidate projects requires forecasting the incremental cash flows that would be produced by the project. Some impacts on cash flow may be relatively easy to forecast, in particular, the costs to conduct the project and any direct impacts the project will have on the firm's future costs and revenues. However, it is difficult to translate many project consequences into impacts on cash flow. For example, how would a project designed to collect better information about customer preferences impact future cash flows? From a practical standpoint, cash flow analysis will not capture many project impacts.

So, at best, financial metrics provide only a partial representation of what is important to a business. According to a study by Research Technology Management, companies that rely mostly on financial metrics obtain "unbalanced portfolios" that are not well matched to the strategy of the firm [2].

The limitations of financial metrics are even more obvious when it comes to evaluating projects in the public sector. Public-sector organizations have social missions and may not even sell goods and services that generate cash flows. Even it they do, earning a financial return may not be a primary objective. For example, a water utility has a mission that includes serving community water needs. A public school has a mission that includes educating students. Cash flow analysis will not do a good job of indicating how well these organizations are accomplishing their missions. In summary, financial metrics fail to measure the value of projects intended to achieve non-financial objectives.