Lee Merkhofer Consulting Priority Systems
Implementing project portfolio management

Part 3:  Lack of the Right Metrics


Metrics imply incentives

Metrics = incentives

The metrics that an organization uses for evaluating projects have a big impact not only on the projects that get chosen but also on the projects that get proposed. "Tell me how you will measure me, and I will tell you how I will behave" [1]. Even if the metrics aren't used to create incentives, managers interpret them as indicating what the organization regards as important. Lack of the right metrics is the third reason organizations choose the wrong projects.

Defining Project Value

The metrics for evaluating projects must support the goal of project portfolio management (PPM), which is to maximize the value derived from the project portfolio. Thus, we need metrics for measuring project value. How do you measure project value? More fundamentally, how do you define value? These questions are more perplexing than you may think.

A Definition of Value

People have been arguing about the definition of value for centuries. The relevant concept of value for our purposes is termed the "utilitarian concept of value:" The value of a project is a measure of degree to which that project enables the organization to achieve its objectives. This view of value was first articulated in the fourth century BC by Aristotle —the value of something is not an intrinsic property of that thing, but rather is determined by its usefulness to those that want it.

Aristotle

Organizations conduct projects because they believe that the outcomes or consequences of the projects will be useful. The more useful the project outcomes, the more valuable the project will be. The project consequences that are desired depend, of course, on the organization's fundamental objectives. Suppose, for example, that an organization's fundamental objective is to create value for its shareholders (more discussion of shareholder value is provided below). Suppose further that the organization is considering a hypothetical project that provides only one consequence: an immediate, one-time cash flow to the company of one million dollars (to be more precise, suppose that the magnitude of the cash flow from the project is such that, after all tax and accounting considerations are addressed, increases the net worth of the company by $1 million). Under these specified assumptions, it would seem reasonable to conclude that the value of that project is $1 million.

An extension to this line of reasoning, which applies regardless of what the organization's objectives might be, is to argue that the value of a project is the worth, to the organization, of obtaining the consequences of the project. Thus, project value might be defined as an amount of money that is equally as desirable as the project, including consideration of risk. (In other words, if project consequences are uncertain, project value could be defined as the amount of cash deemed equally desirable as the gamble over those uncertain project consequences.) In the portfolio context, this may be interpreted as an amount of cash that would make executives indifferent between (a) the portfolio with the project and (b) the portfolio without the project but with that amount of incremental cash.

In the literature on valuation, the above definition of project value is termed the project's "selling indifference price" (also called the "breakeven selling price"). A close relative is the "buying indifference price" (or "breakeven buying price"), which argues that the value of a project is the amount of cash that would make executives indifferent between (a) the portfolio without the project and (b) the portfolio with the project plus a debt equal to that amount of cash.

In my experience, most people find it intuitive that the value of something is what you are (just) willing to give up for it. For this reason, my working definition for project value is the maximum amount of organization wealth executives would be willing to trade away to obtain the (uncertain) project outcomes. This is a variation of the buying indifference price—it is a buying price because it is an amount the organization would pay to buy the project consequences, and it is an indifference price because it is the price point such the organization is indifferent between declining the project versus paying the price and obtaining the project consequences.

Although conceptually intuitive, determining the buying indifference price for a project gets more complicated when you consider that paying a buying price means not investing in something else. Buying A means that you cannot buy B, so the true cost of buying A is not getting the most valuable B that you could have bought. Thus, the most an organization would be willing to pay to obtain the consequences of a project should be less if doing the project means foregoing other, very good investments than when it would mean foregoing only marginal investments.

Also, if the organization were to pay a buying price, the budget available for investing would change, so the opportunity cost of incremental investments may change. In the portfolio context, as indicated above, a project indifference price is an incremental value, determined relative to other projects in the portfolio. If the projects are interdependent, the indifference price will depend on those other projects, Thus, if the indifference price for a project is determined based on it being the first project in the portfolio (or if its value is determined in isolation to other projects), we'd likely get a different value than if it is being added to other portfolio projects. In general, therefore, project indifference prices depend on the order in which the projects are added to the portfolio.

Fortunately, these complexities need not be of much concern for our context. Organizations obtain capital from a variety of sources and spend that capital on numerous different investments. As a practical matter, it is impossible to know when candidate projects are being considered which other investments would be foregone if there are incremental reductions to available investment capital and what the opportunity costs would be. More importantly, our concern is identifying the set of projects that can be conducted within a specified budget that will produce the greatest value. Project prioritization, with projects ranked based on the ratio of value to cost (as explained in the paper on Mathematical Theory) will identify the value-maximizing portfolio, provided projects are defined to be independent of one another. If the projects are independent, they can be considered one-by-one, in isolation to one another, and with value being relative to the case where the project is not done. So long as we assume the same opportunity cost when establishing the worth of the possible consequences of the different projects, the project ranking will be correct. Thus, the subtleties for setting indifference prices are unlikely to pose significant problems for project prioritization, and defining project value as the worth of project consequences is reasonable for our purposes.

Note that the value of a project, defined in this way and under these assumptions, does not depend on its cost. (The decision of whether to conduct a project, does, of course, depend on cost. In particular, you would never want to conduct a project whose cost was greater than its value.) An exception to the rule that project value does not depend on project cost would be a case where paying for a project impacts the organization's ability to benefit from the project or from other projects. An extreme example would be a project anticipated to produce great project outcomes, but whose cost would bankrupt the company. In such cases, the value of projects would logically be the value of the project consequences taking into account any effects of having to pay for the project. Since we are concerned with organizations that conduct numerous projects, each of which consumes only a portion of the budget, we can typically safely ignore such effects.

Project value

Project value is the worth to the organization of the (possibly uncertain) consequences of doing the project—in other words, it is the maximum price the organization would pay to obtain the project's consequences

Our preferred definition of project value has a critically important property—namely, it maps to organizational preferences. Suppose there are two projects requiring the same cost and resources, and only one can be added to the project portfolio. The organization will prefer Project A to Project B if and only if our measure of project value is higher for Project A than it is for Project B (because any organization would be willing to pay more for the consequences of Project A than for Project B if and only if it prefers Project A's consequences). Many common approaches to project prioritization don't use a project evaluation measure with this essential property (e.g., strategic alignment). Unless project value is defined in a way that maps to the true preferences of the organization, that measure cannot correctly prioritize projects. Our definition of project value is a true measure of the relative attractiveness to the organization of its project alternatives, and that measure can be compared with project costs to determine the projects that are worth doing and their priorities.

OK, assuming we accept that the worth to the organization of the project consequences is a reasonable definition of project value, how do we determine how much an organization should be willing to pay to obtain those consequences? Before addressing that question we need to consider more carefully why organizations conduct projects

Projects Determine the Evolution of the Business

The business of an organization is always evolving, and the projects that the organization chooses affect that evolution. For example, a new technology might become available that would allow the organization to reduce its costs. If a project is conducted to install the technology, the organization would incur lower operating costs than it would have if it had chosen not to do the project.

It is also true that the projects that an organization chooses not to do affect the evolution of the business. For most organizations, standing still means falling behind. There are many reasons for this, including increasing competition, changing customer preferences, and the aging of organizational assets. Thus, to take one example, if the organization chooses not to do projects that maintain or replace aging assets, the service provided by those assets will decline.

Figure 18 illustrates a useful way to think about project value. At the point in time when an organization is considering a new project, it is really facing a choice between two alternative future states. If the project is conducted, that project will, presumably, transform the business to some more desirable state. If the project is not conducted, some other, presumably less-desirable, state will result.


Projects determine business evolution.

Figure 18:   Project choices determine the future state of the business.



The perspective of Figure 18 provides a basis for computing project value. The value of a project is the difference between the values of the two potential future states of the business:

Project value  =  Value with the project  -  Value without the project

(Equation 1)

Some Observations Regarding Project Value

Based on Equation 1, we can conclude some important things about project value.

  • To estimate project value, it is necessary to consider what would happen if the project is not conducted as well as what would happen if the project is conducted. Many project scoring systems ignore the implications of not doing projects, which causes some types of projects to be grossly undervalued.
  • The same project can have different values to different organizations. For the purposes of estimating project value, therefore, it is not sufficient to consider just the characteristics of the project itself. It is also necessary to consider characteristics of the business that determine how useful that project will be to that business.
  • The value of a project can change depending on what other projects are conducted. As suggested above, because of synergies and economies of scale, for example, the costs and benefits of a project can change based on the other projects that are also conducted. If such dependencies exist, optimizing the project portfolio requires estimating the value of conducting different groupings of the interdependent projects.

For the purpose of identifying metrics for measuring project value, the most important implication of Equation 1 is this: Project value is the contribution that the project makes to the total value achieved by the organization. Because project value is the difference between the value that the organization attains in two alternative states, the methods for estimating project value are essentially the same as the methods for estimating organizational value. This is good news, since management scientists have devoted considerable effort to determining how best to measure the value created by organizations. We can use the concepts and methods that they have devised to help us to quantify project value.

Inadequacy of Financial Metrics

So, how can you measure the value created by an organization? Interestingly, most businesses get it wrong.

Most businesses use financial metrics computed from cash flows to measure value, for example, return on investment (ROI), internal rate of return (IRR), net present value (NPV), payback period. Using these metrics to evaluate candidate projects requires forecasting the cash flows that would be produced by the project. Some impacts on cash flow may be relatively easy to forecast, in particular, the costs to conduct the project and any direct impacts the project will have on the firm's future costs and revenues. However, it is difficult to translate many project consequences into impacts on cash flow. For example, how would a project designed to collect better information about customer preferences impact future cash flows? From a practical standpoint, cash flow analysis will not capture many project impacts.

Thus, at best, financial metrics provide only a partial representation of what is important to a business. According to a study by Research Technology Management, companies that rely mostly on financial metrics obtain "unbalanced portfolios" that are not well matched to the strategy of the firm [2].

The limitations of financial metrics are even more obvious when it comes to evaluating projects in the public sector. Public-sector organizations have social missions and may not even sell goods and services that generate cash flows. Even it they do, earning a financial return may not be a primary objective. For example, a water utility has a mission that includes serving community water needs. A public school has a mission that includes educating students. Cash flow analysis will not do a good job of indicating how well these organizations are accomplishing their missions. Financial metrics fail to measure the value of projects intended to achieve non-financial objectives.