- Customer value. For organizations that sell or otherwise provide products and services, metrics for estimating the impacts of projects on customers are needed. The field of economics provides
various metrics for measuring customer value, including the concept of consumer surplus. The field of market analysis has developed numerous customer-satisfaction metrics. Another common approach is
to adopt metrics that describe specific product or service characteristics that customers care about (e.g., attributes of product and service quality and price).
- Stakeholder value. In additional to customers, organizations typically have other stakeholders whose attitudes and perceptions are important and should be managed. Examples include unions,
regulators, business partners, etc. The relevant metrics typically depend on the type of stakeholder and ways in which the attitudes of those stakeholders are typically expressed or impact the
interests of the organization. Oftentimes, metrics for shareholder value consist those metrics needed to identify and characterize the importance of the concerned stakeholder group and metrics that indicate the
anticipated reaction of those groups.
- Mission value. Public sector organizations aren't the only ones that have missions. Many private sector organizations have adopted mission statements that identify goals beyond maximizing
shareholder value. The appropriate metrics in this case measure the contribution of proposed projects to the achievement of these goals.
- Community socio-economic quality. Some organizations conduct projects that significantly impact local communities. For example, a project to build a new manufacturing plant might have a
significant impact on community jobs. The fields of sociology and economics provide numerous metrics potentially useful for such situations.
- Option value. Consistent with the theory of real options, projects that contribute to the organization's platform for future success provide a source of value. For example, an IT project may give
the organization new capability. An R&D project may provide new knowledge or understanding important to the business. Also, a project may have a distinct, strategic value. Metrics may be defined
that capture these additional sources of value. The key is not to double count.
As suggested by the above, numerous metrics may be needed to capture all of the potential components of project value. If metrics representing some sources of value are omitted, the
value of projects will be underestimated. Furthermore, there will be a bias against doing those projects that provide the types of value that are not captured due to the omitted metrics. Note that
metrics need to represent timing, that is, when the project benefits are likely to occur and how long they will persist, and, oftentimes, risks (e.g., the likelihood that the project will actually produce
its anticipated benefits). When all such metrics are specified, the decision model defines the aggregation equation that allows the value of a project to be expressed in dollar terms.
Obviously, there is a limit to how many project evaluation metrics should be used. The goal should be to include the minimum number of metrics necessary to roughly capture every significant
source of project value, not numerous metrics that more completely capture just a subset of components of value. The 80/20 rule applies. Since few if any projects will provide significant contributions
under each type of value, having lots of metrics doesn't necessarily create a significant burden for evaluating proposed projects. Estimates need only be provided for the subset of metrics that are relevant
to capturing the specific motivations for doing that project.
Metrics as "Observables" and the Clairvoyant Test
To the extent possible, metrics should be observables; that is, characteristics of projects or project outcomes that can be observed and measured in the real world. Since estimating
project value requires forecasting the future, metrics don't, obviously, all have to be things we can observe today. Metrics can, for example, include a projected future state of some observable, for
example, an improvement in a reliability-of-service statistic important to customer satisfaction.
A useful device for checking whether a metric is observable is the so-called "clairvoyant test" devised by my college mentor, Professor Ron Howard. Before accepting what appears to
be a good metric, consider whether a clairvoyant could give an unequivocal value for that metric given that a project decision is made in a specific way. Oftentimes, the clairvoyant test points out
inexactness of what initially appears to be a well-defined metric. For example, "customer satisfaction" doesn't pass the clairvoyant test. However, "percent reduction in recorded customer complaints"
and "company ranking in the next industry customer satisfaction survey" are metrics that do pass the test.
Metrics that don't pass the clairvoyant test are vague. They create inconsistency and imprecision when used for estimating. More importantly, if the metrics are not observables,
they cannot be monitored so that actual values can be compared against estimates.
Financial Metrics
The traditional financial metrics should be used to determine the direct financial components of project value. Project investment cost is, of course, an important
financial metric for any project. Projects that impact operations (e.g., projects that create new revenues or that affect future operating costs) produce downstream financial impacts that must also
be evaluated. Thus, any and all significant, incremental, period-by-period cash flows that are anticipated to result from projects should be estimated, either as a most-likely or average case or in
the form of alternative scenarios. The organization's standard accounting model may then be used to determine the resulting after tax, or unencumbered "free" cash flows, which may be used to compute
a project's financial NPV.
Some important principles for estimating financial value in support of project prioritization include:
- Ignore previously paid, sunk costs.
- Include opportunity costs (the opportunity cost of a resource is the value of the net cash flow that could be derived from it if it were put to its best alternative use).
- Include overhead expenses (e.g., administrative expenses, managerial salaries, legal expenses, rent) that are directly related to a project. Indirect overhead can, if necessary, be prorated
across proposed projects.
- Include "spill over" effects. For example, if a project introduces a new product or service that draws sales from existing products, include such lost revenue in cash flow estimates.
- Interpret expected project cash flows submitted in support of a project proposal as commitments to be achieved by the project manager. If there are cash flow components that are more speculative
or for which the project manager cannot be held accountable (e.g., because they are contingent on events beyond the control of the project manager), specify such cash flows separately and assign
probabilities.
- Identify and include any terminal cash flows, for example, cash flows expected from the disposal of assets at the conclusion of the relevant product or service lifecycle.
- Be consistent in accounting for inflation. For example, using an inflation-adjusted discount rate while ignoring inflation in estimating cash flows would result in a bias against accepting
projects.
- For the purposes of prioritizing projects, remember that the project's financial benefit is its NPV exclusive of its current-period costs.
Be suspicious of long-term, positive NPVs. Keep in
mind the economic axiom that excess profits (the source of positive NPV) must be zero in a perfectly competitive market. A long-term, positive NPV requires some sustainable competitive
edge—being first, being the best, or being the only. Retaining that edge indefinitely would require some barrier to the entry of competitors. Consider carefully how long it will
take competitors to catch up and drive profits back down.
Each Organization Needs Its Own Metrics
Different organizations conduct different types of projects. The metrics for evaluating new product investments by a software vendor, for example, will be different than the metrics
needed to evaluate process improvements for a company operating an oil pipeline. Also, different organizations create value in different ways. An electric utility, for example, creates value
differently than does a ballet school. Some organizations will seek to maximize shareholder value, while others will want to value impacts to other stakeholders as well. Thus, each organization will
have a different model for how its projects create value and, therefore, will want to use different metrics. There is no one set of project metrics that works for every organization. However, in all
cases, good metrics provide a means for computing the value added by projects. Good metrics are observables. And, they are sensitive to project decisions so that they may be used to differentiate the
value of alternative project portfolios.
Metrics Provide Justification for Tough Choices
One of the most under-appreciated benefits of having good metrics linked to a defensible decision model is improved justification. Author Anthony O'Donnell quotes a portfolio manager
at an insurance company that implemented a portfolio management tool: "People would come to me and ask me to do a particular project...I would tell them I couldn't fit it in, but had a hard time
articulating why." Metrics now allow him to give concrete reasons for turning away projects. "Their satisfaction immediately went up, and I still didn't do their projects!"[7].
The Right Metrics Turn Project Proposals into Performance Contracts
Deriving metrics from a decision model ensures that the organization is seeking the right information about proposed projects; namely, the information necessary to estimate the value
to be derived by the organization if the project is conducted. The value of the project can then be compared with its cost, and the resulting "bang for the buck" compared with similar estimates for
other candidate projects. This provides a sound basis for making project-selection decisions.
If, in addition, the metrics are observables, the organization further benefits in that project proposals serve as performance contracts. In return for a chance at obtaining a share
of the organization's limited resources, project proponents indicate in the clearest and most relevant terms what they expect the project will accomplish. Project results and impacts can then be
tracked and compared with the original estimates. Performance contracts document the terms of the agreement, protecting both parties to the contract. Framing the project as a performance contract
creates a healthy shift in perspective. Instead of choosing which projects to cut, the focus is on deciding what project opportunities to purchase.
Due to uncertainty, project outcomes may not exactly match forecasts. Thus, what the implicit contract requires is not that project managers invariably be held responsible for
achieving all of the performance indicated by their estimates, but that any significant deviations between estimates and actuals be explained. Over time and on average, some projects should exceed
expectations while others will fall short. In the meantime, the organization can learn to improve forecasts by tracking and better understanding the uncertainties that are involved.
In situations where the uncertainty is considerable, it may be useful to separate metrics for indicating the benefits that can be expected from benefits that are more speculative
and uncertain. The "expected" benefits then become the basis for the performance contract and the speculative benefits can be appropriately discounted based on risk (see Part 4).
References for Part 3
- E. Goldratt, The Haystack Syndrome, North River Press, 1991.
- R. Foti, "Priority Decisions," PM Network, 16 April, pp. 24-29.
- The recommendation to develop a decision model and, more generally, the views and ideas expressed in this and the next part of this paper are shared by many decision analysts. See especially
"Choosing the Right Metrics for Measuring, Monitoring, and Maximizing Shareholder Value," C. Spetzler and R. Arnold, www.sdg.com, May 2003. The book Value Focused Thinking by R.Keeney, 1992,
describes many of the concepts and techniques for building decision models.
- R. Kaplan and D. Norton, The Balanced Scorecard, Harvard Business School Press, 1996.
- These quotes were found by searching "strategic alignment" on Google, and were taken from a Project Management Institute project portfolio management seminar brochure, an article entitled
"From Crisis to Control: New Standards for Project Management," , and a presentation from a consulting firm. You can find similar statements all over the web.
- This description of how to rank projects based on strategic alignment is from pages 140-143 of "Chapter 4.1: Linking Strategy and Project Portfolio Management," by K. C. Yelin, in H. Levine's
book Project Portfolio Management, Wiley, 2005.
- A. O'Donnell, "Worth the Effort," Insurance Technology, March 4, 2003.
|
Printable
version
The Scientific Method