Lee Merkhofer Consulting Priority Systems
Choosing the Wrong Portfolio of Projects

Inability to measure project value is the fourth reason organizations choose the wrong projects.

Part 4: Inability to Measure Project Value


Project-based organizations with limited resources face a common challenge. There are more good ideas for projects than resources for conducting those projects. Consequently, organizations must decide which projects to conduct and which to defer, ignore, or cancel. An efficient and effective way to select project is to prioritize them based on productivity; that is, based on the ratio of value to cost. If independent projects are ranked by the ratio of value to cost and selected from the top down, the resulting project portfolio will create the greatest possible value for the costs incurred. The logic is compelling, but putting it into practice requires an ability to measure the value of candidate projects. Inability to measure project value is the 4th reason organizations choose the wrong projects.

For organizations in the private sector, maximizing profit is often assumed to be the goal of a project portfolio. It is natural, therefore, to define the value of a project to be the financial return that the project will produce. This logic is often referred to as maximizing shareholder value because the owners of the organization, or its shareholders, are likely to benefit most from the increment of financial value produced by investments. The example system created to prioritize R&D projects for a manufacturer of electronic test equipment is an example of project prioritization based on the principle of maximizing shareholder value.

For many organizations, especially government organizations, generating a financial return is not the primary purpose of projects. Instead, success is defined in terms of creating value for stakeholders, individuals or groups that the organization's investments are intended to benefit. The following example illustrates the development of a prioritization model based on a principle of maximizing value for stakeholders.

Driverless Humvee

Driverless vehicle on track used
to test drivers of regular vehicles

The US Army's Tank Automotive Research, Development and Engineering Center is charged with advancing technology for tanks, military transport vehicles, and ground systems. The Center's project portfolio includes projects funded out of its own budget, projects funded in part or in whole by other defense agencies (its customers), and collaborative projects conducted jointly with private-sector automotive companies like General Motors. To provide an example, one of the Center's projects is a robotic, driverless vehicle intended to reduce risk exposures for military personnel.

To guide the allocation of its multi-million dollar budget, candidate projects were scored against more than a dozen separate criteria, including "alignment with warfighter needs," "technical feasibility," "positioning for the future," "risk," "innovation," and "opportunities for additional funding." The scores were multiplied and added to produce a metric for ranking the projects. A manager of a unit charged with providing support for the prioritization process had recently taken a short course on project prioritization. Concerned that the current scoring system appeared illogical and disconnected from an estimate of either project value or project productivity, he commissioned an educational effort wherein he and his team would be guided in the design of a prioritization model based on valuing projects using a utility function based on multi-attribute utility analysis (MUA).

Objectives hierarchy

The first step for the development of a priority system was, as usual, the creation of an objectives hierarchy identifying the Center's fundamental objectives (left). As shown, financial viability; that is, maximizing Center revenues and finding ways to reduce future costs, are clear Center objectives. Defense agencies that provide funding for projects are customers, and providing value to customers was deemed an important objective. Producing high-value innovations was part of the Center's mission and, therefore, an objective. Improving staff knowledge, assets and capabilities, and relationships with partners were identified as objectives needed for continued, future Center success. The last objective, promoting stakeholder satisfaction, was meant to affirm that consideration needs to be given to the interests of many additional important stakeholders, including Department of Defense policymakers, members of Congress, and the Center's staff.

The value of a project is determined by the degree to which it achieves objectives, so performance measures and performance sub-models were defined for each objective. In the case of the financial objectives, for example, if a project manager could quantify potential annual incremental revenues or cost savings resulting from a project, the net present value of those cash streams multiplied by the manager's estimate of likelihood was used to measure the impact on the Center's financial objectives.

For estimating performance relative to the other objectives, sub-models with one-to-ten scoring scales were created. Influence diagrams were constructed to identify the considerations referenced by the various levels of the scales [1]. For example, to compute customer value for projects that produce products for customers, the sub-model required providing three estimates. One was an estimate from a scale very similar to a scale used in the existing model. As shown below, the scores correspond to rough estimates of possible ultimate values of a project's products to customers. Because projects would often require contributions from partners and other organizations before the ultimate products could be delivered, it was necessary to ask scorers to prorate ultimate customer value by the proportion of costs contributed by the Center.

Computing project value

Scale for estimating the ultimate value of project products to customers.


The second scale used in the customer value sub-model served to indicate how long it would take for customers to begin receiving value:


Computing project value

Scale for estimating the when customers would begin receiving value.


An exponential scaling function was applied to the scores from each scale to obtain linear measures corresponding to the scale definitions. The third estimate for the customer value model was the probability that the project would actually produce customer value. Thus, the equation for the customer value sub-model was:


Project customer value

=

  Ultimate product value to customers prorated by the fraction of costs contributed by Center and discounted to present  

×

Probability of success


Similar performance models with scoring scales were developed for the other objectives. The objectives and their performance measures were assumed preferential independent allowing the value components computed by the sub-models to be weighted and added. Weights were assigned and the value model was implemented as a spreadsheet.

A pilot test was conducted wherein the team was tasked with providing dollar financial estimates and scores for 21 projects. Project managers provided project costs and any estimated Center revenue or cost impacts, which the model discounted to present values. The software allowed each individual to assign scores, and the team challenged individuals whose scores were higher or lower than team averages to explain or change their scores. To rank projects, each project's total score was divided by its remaining costs.

Computing project value

The team was excited by the results of the pilot test. While the priorities obtained from the current system sometimes seemed inconsistent with their own. those from the MUA-based model seemed intuitive, especially with regard to low-cost projects that the Team believed the existing model tended to assign low priority. The team was convinced that the prioritization logic used by their model was more defensible, and that computed priorities were more reasonable. Also, the new model accounted for many considerations deemed important that were not represented in the existing model, yet, all-in-all, the judgment was that it was no more difficult to apply.

Because the effort was funded out of training funds, a serious attempt was made to objectively critique the exercise and pinpoint what lessons and insights could be garnered. When asked what participants didn't like, the unanimous answer was that the MUA-based model required too many inputs and that many of the inputs required answering difficult questions. However, when asked which inputs should be left out, none could be identified. People nodded when one participant commented that she felt that being forced to think about the many questions was useful in that it crystallized understanding of what considerations are actually important when forming opinions about the priorities of possible projects. In other words, a benefit of the MUA-based system was that people felt more knowledgeable and more confident in their opinions after the scoring, even before seeing the model results.

People felt that an important insight from the exercise was provided when a sensitivity analysis of the results indicated that little was gained by spending effort scoring a project on more than the two or three objectives that, in their minds, primarily motivated a project. If they were to repeat the scoring process, it was agreed that less time would be spent on scoring for less important objectives.

Following the exercise, a recommendation was made to replace the current scoring model with a model to be created using MUA. However, instead, Center executives elected to ask the designers of the original model to make changes to better account for considerations captured by the team's model.

The Key to Being Able to Prioritize Projects is a Project Value Model

Note the solution approach used in the example. A value model was constructed for computing project value. The inputs to the model consist of time streams of costs and revenues that represent project financial performance and scores that measure non-financial project performance. The incremental cash flows are discounted to obtain a financial performance measure and the various non-financial performance measures are input to a multi-attribute utility function to compute project value. Projects are ranked by the ratio of expected project value to cost. The approach used in the example is a general approach for prioritizing independent projects for the purpose of identifying value-maximizing project portfolios. The critical step in this solution approach is creating a model for computing project value.

The Challenge of Estimating Project Value

Peter Drucker

Despite the obvious importance of project value, most organizations that conduct projects, including projects to create new products or services, do not find it easy to estimate the value of their projects. Many organizations do not even attempt to quantify project value as part of the project selection process. Why? —Because they don't know how to create a project value model. Inability to measure project value makes it difficult, if not impossible, to effectively manage a project portfolio for the purpose of maximizing value. As Peter Drucker is credited with saying, "You can't manage what you can't measure" [2].

Project Selection is an Ongoing Task

For many organizations, project selection occurs mainly as a component of the capital budgeting process when resource needs for the upcoming budget period are being planned [3]. These organizations typically conduct a comprehensive review of candidate projects and choose a subset for execution in the coming budget period. However, pressing new needs and unexpected opportunities can arise at any time [4]. Also, some projects may need to be terminated due to unforeseen events or because it becomes apparent that the resources being consumed could be better used elsewhere. Project selection is, thus, not just a once-per-year activity, it is an ongoing task. Having a model for computing project value provides continual support for the prioritization of projects required due to constraints on resources.

The Difficulty of Project Selection

I describe the reasons that project selection is difficult throughout this text. To summarize:

  • Projects differ in fundamental ways. One project may help a firm sell a product to more customers. Another might reduce the amount of waste produced in the manufacturing process. A third might increase the level of safety for workers handling hazardous chemicals. It's hard to compare projects whose motivations are so different.
  • Uncertainty—The worth of a project depends on its consequences, but project consequences are often uncertain. Projects with uncertain consequences are like gambles. How much it is worth to undertake a gamble depends in part on the willingness of the organization to accept risk. Uncertainty and risk compound the difficulty of project selection.
  • The costs and benefits of projects occur in different time periods. Costs normally begin almost immediately, but benefits generally occur sometime in the future. Also, project benefits may continue over some extended period of time. Comparing projects when the timing and duration of benefits and costs differ is hard to do intuitively.
  • Interactions among projects—the costs and/or benefits of a project can depend on whether or not other projects are conducted. Thus, it may not be possible to make project choices one at a time and without regard to what other projects get funded.

In sum, choosing projects involves complex considerations. Selecting the best project from many alternatives is often tough. Choosing the most desirable combination of projects; that is, creating an optimal project portfolio, is even harder [5].

A Value Model Programmed into a Computer Allows Project Value to be Computed Quickly and Consistently

The challenges that make project selection difficult for people—multiple considerations, uncertainty, interdependencies, the need to compare costs and benefits in different time periods—are exactly the characteristics that make the problem amenable to solution via a computer programmed with an analytic model. As described in Part 1, people have limited information processing skills, can be biased, and are often inconsistent when making choices [6]. An analytic model breaks a complex problem into pieces, allowing those pieces to be studied and characterized, and their relationships represented mathematically [7]. People can then focus on the simpler task of providing inputs to the model. The processing of those inputs can be handled in accordance with sound mathematical reasoning using a valid solution algorithm and a computer.

Knowledge and effort are required initially to design and construct a project value model. But, once built, a model may be used again and again, both during an annual budgeting exercise and as a means for determining how to properly insert a newly identified project into the existing priority order. With each application, generating inputs becomes easier and less burdensome. A project value model makes the criteria and logic for project evaluation explicit. When the organization has the ability to quickly and systematically evaluate the attractiveness of proposed projects, experience shows that the quality of project proposals increases. In other words, a project value model not only helps the organization to select from among its candidate projects, but it also shows project managers how to design projects that generate greater value. As experience with the model accumulates, the model design can be modified and improved in ways that make its application easier, its predictions more accurate, and the insights it generates more useful.

A Project Value Model Formalizes Judgments and Reasoning that the Organization Should Already be Using

The value of a project is, logically, the worth, to the organization, of the project's consequences. In order to make the right choices, therefore, organizations need to consider the consequences of conducting candidate projects, specifically, the degree to which project consequences advance the organization's objectives (something organizations with good accountability practices do anyway). They, then, must determine the value of those advancements. A model that reliably estimates project value helps the organization to consistently make value-maximizing project choices.

References

  1. M. W. Merkhofer, "Using Influence Diagrams in Multi-attribute Utility Analysis—Improving Effectiveness Through Improving Communication," Influence Diagrams, Belief Nets and Decision Analysis 297-317, 1990.
  2. As credited by R. C. Wolcott, "Don’t Be Tyrannized by Old Metrics," Harvard Business Review, September 23, 2016.
  3. L. J. Gitman and J. R. Forrester, Jr., "A Survey of Capital Budgeting Techniques Used by Major U.S. Firms," Financial Management, 6(3), 66-71, 1977.
  4. Y. Petit, "Project Portfolios in Dynamic Environments: Organizing for Uncertainty," International Journal of Project Management, 30(5), 539-553, July 2012.
  5. S. Elonen and K. A. Artto, "Problems in Managing Internal Development Projects in Multi-Project Environments," International Journal of Project Management, 21(6), 395-402 August 2003.
  6. J. G. March, "Bounded Rationality, Ambiguity, and the Engineering of Choice," The Bell Journal of Economics, 9(2), 587-608, Autumn 1978
  7. P. R. Garvey, Analytical Methods for Risk Management: A Systems Engineering Perspective, CRC Press, Oct 20, 2008.