Lee Merkhofer Consulting Priority Systems
Implementing project portfolio management

Improve the Prioritization Process

The most important step in project portfolio management (PPM) is choosing which projects to fund, and making the right project choices. Managers and their organizations are facing increasing internal and external pressures to cut costs while being more effective at meeting constantly changing demands from customers within narrowing windows of opportunity. In many organizations, projects are repeatedly added, changed, and removed in response to heightened business activity. The backlog of "must do" projects requires resources that often exceed what the organization can provide. Given today's level of business competition, making the wrong project choices and ineffectively using limited resources can threaten the very survival of the business.

In order to continually make the best use of limited organization resources, the project portfolio management office must determine which projects to initiate, which on-going projects to continue to fund, which projects to revamp, and which projects to kill. As illustrated in Figure 27, the PPM office must make project decisions so as to optimally manage the "project pipeline."

The project pipeline

Figure 27:   The Portfolio Management Office manages the project pipeline.      

Since organizations rarely have sufficient resources to conduct all available projects, projects must be prioritized. In the absence of formal models for prioritizing projects, managers can use forced ranking. As explained previously, forced ranking means that managers get together and "force" each project into a strict priority ordering (or into a number of priority groups). Projects are then added to the portfolio in rank order until the business runs out of resources. Below that point, projects are put on hold or killed outright. Considerations that apply at the portfolio level, such as project synergies and portfolio risk, may be used as modifiers to the project-by-project ranking. Needless to say, forced ranking, as well as the final choice of which projects to conduct, are difficult decisions. In the absence of a more formal approach, bias, mental errors, and politics can play a major role in project selection.

As I argued previously, individuals knowledgeable about the projects under consideration have the ability to separate high priority from low priority projects. I've seen ample evidence that a PPM team, using the right processes, can reliably produce a rough priority ranking of projects. Forced ranking becomes more difficult and time consuming, however, if there are:

  • Many project proposals. If there are more than twenty or so requests, decision makers may not be sufficiently informed about the projects or their underlying needs to make consistent choices. This is especially true when proposals come from different departments or jurisdictions.
  • Project complexities. If projects are technologically complex, require "apples versus oranges comparisons," or involve difficult-to-judge considerations (such as risks or critical timing factors), most experienced individuals will be hard pressed to provide reliable priorities without reference to formal criteria.
  • Multiple decision makers. The more people involved in priority setting, the less likely it is that they will agree without first agreeing on the objectives, principles, and criteria to guide project prioritization.
  • A need to justify project choices. There may be stakeholders not involved in setting priorities, including those who provide project proposals, who require an explanation for the funding choices that are made. If decision makers arrive at their choices intuitively, they may have trouble explaining priorities when questioned or challenged.

In nearly all instances, but especially so under the above complications, a key challenge for the PPM team is reaching agreement over how projects should rank. Paired comparisons and point scoring are two techniques that can help groups rank projects.

Using Paired Comparisons to Facilitate Forced Project Ranking

If there aren't too many projects, you can use paired comparison to conduct group rankings. Place the name of each project on a paper sticky or index card. Hold up two of the cards and ask, "Which of these two projects is higher priority?" The essential step is to make sure participants base their priority judgments on the correct consideration. According to the relevant mathematical theory, projects need to be prioritized based on the ratio of project value to total project cost (assuming the projects are independent of one another). People often incorrectly prioritize based on a judgment of relative value, which, as I illustrated back in Figure 12, results in the wrong project choices. Thus, if the resource requirements of the projects differ significantly, you may need to remind participants that higher priority should be assigned to the project with the higher ratio of value-to-cost, which may not be the project with higher value.

Initially, many people will be reluctant to express their opinions. In the interest of time, you'll need to encourage quick responses. Don't waste time with discussion if everyone is in agreement. Make sure there is someone knowledgeable about each project to explain and answer questions in case others are unclear about exactly what the project will do. If people disagree on priorities, encourage opponents to express their arguments, but call for a vote where necessary to avoid getting stuck.

Once the first two projects have been compared, put the higher priority project at the top of the list and the other one below it. Now take a third project card and compare it in turn to each of the two previously considered projects to determine where it should be placed. Continue until the group has compared all of the projects to each other, at which point you've ranked the entire list.

Using Point Scoring to Facilitate Forced Project Ranking

A problem with paired comparisons is that it will take a lot of time if there are more than a few projects and/or people don't quickly agree on the pairwise comparisons. A quicker option is to have each participant assign points to each project, such that the number of points assigned indicates the individual's view of the project's priority. If there are a large number of projects, you may need to partition them into groups of similar projects with no more than ten or so in each group. Each person allocates points to the projects to indicate his or her estimate of the project's value relative to its cost. The only requirement is that no project may be allocated more than 100 points (highest possible priority). The process works best if people are required to assign a unique number of points to each project (no ties). You can pick one project to include in multiple groups to serve as a benchmark (a participant must assign the same number of points to the benchmark each time). Finally, to obtain the group ranking, add up the number of points assigned to each project by the participants.

When people drastically disagree on the number of points, have them discuss what's different about their views. Relevant questions include: How will this project affect the business? Will it make a significant difference in our ability to achieve our mission? What, exactly, are the benefits that will be produced? Will it make a temporary or lasting difference? Is there much risk that the project will fail to achieve its goals? How urgent is the project, would its effectiveness decline significantly if we delay it? Again, be sure that people base priority estimates on the ratio of value to cost.

Be Careful Using Scoring Models

An important characteristic of the above methods is that they can be applied consistent with the rule that each participant's estimate of priority be based on judgments of project value relative to project cost, which is necessary if the goal is to choose projects that create the greatest total value. In contrast, some organizations use scoring rules that do not result in projects being ranked according to this necessary logic. A common approach is to define criteria relevant to judging projects and specifying a simple scoring scales for each criterion. For example, with regard to a given criterion, 1 = poor, 2 = OK, and 3 = good. The scores are tabulated and used to obtain a total score for each project. A total score above a certain level is judged a "must do." Alternatively, projects might be ranked or grouped into priority categories based on their total scores. Other common variations include grouping projects into priority categories, for example, projects that address safety issues or projects related to regulatory requirements, designated "priority one."

In addition to the logical flaw associated with failure to prioritize based on the ratio of value to cost, scoring methods of this sort typically produce practical problems. Frequently, too many projects get high scores and/or are labeled must do's. If some types of projects, such as safety projects, are designated as critical, project proponents may place their projects in the critical categories even though the connection is small or indirect at best. For example, a project might be labeled a safety project because there is some influence on safety, even though it is clear that the very small decrease in the number or severity of accidents that would result from the project could not justify the cost.

Assigning a number to something doesn't necessarily make for a more accurate method of measurement. If scores are subjectively assigned to measures without clear criteria, different people will assign wildly different scores for the same project, and the same person may assign different scores on different occasions. Regardless, middle scores are common for most projects, especially when numerous scoring criteria are used. High scores on some criteria cancel out low scores on others. Furthermore, most scoring models aren't sufficiently precise to trust small differences in total scores.

In any case, ranking projects by project scores will be incorrect unless those scores measure the ratio of project value to project cost. Most scoring systems don't claim to measure value. Even when they do, they often fail to scale results to project cost and, therefore, don't come close to indicating "bang for the buck."

Improve Project Data

Even the best project prioritization process will be worthless without adequate project data. "A micrometer won't help you measure a cloud." Thus, one way to improve the prioritization process is to improve the quality of available project information.

The first step to getting better data is to make sure that information requirements are well-defined. If project proponents are clear about what information is needed to enable their proposals to be considered, they are much more likely to supply that information. Thus, the templates for collecting data on project proposals must be complete and precise.

Second, there should be a culture and expectation that rigor is required to generate project proposals. Estimates and forecasts should be backed by reason and analysis. Project proponents need to do their homework before the project gets proposed up the management chain.

Third, the organization should be prepared to allocate increased resources to project planning. Skill, experience, and true cross-functional collaboration are often needed to generate solid project proposals. Inevitably, increasing the effort devoted to preparing project proposals detracts from the resources available for actually doing projects. However, as previously asserted, the tradeoff in improved decision making will likely be worth it.

Note that a lack of adequate systems for collecting detailed, quantitative project data should not rule out attempts to implement PPM. In my experience, project evaluation systems based mostly on data generated subjectively through "best professional judgment" can perform surprisingly well, so long as those providing the judgments are knowledgeable and unbiased. Invariably, PPM acts as a "forcing function" that causes the organization to improve its ability to collect and document important business data.

Estimate Cost, Value and Risk

Using the right criterion to prioritize projects, the ratio of project value to project cost, is critical. Thus, you can support the prioritization process by providing a value and cost estimate for each project. Estimating project cost (not to mention project value) can be difficult. Be sure to use full cost accounting. When evaluating proposed expenditures, some organizations make the error of detailing only non-routine costs, such as one-time, "build" costs, external contractor and consulting costs, and technology costs. As mentioned earlier, project costs include not just the funding request, but also any funding provided from other sources plus the opportunity cost of using equipment, personnel, raw material and any other "non-costed" resources that will be employed by the project. Also, all future costs necessary to obtain project benefits, including future operating and maintenance costs, should be identified, estimated, and included in the calculation. A project to install a $100,000 building security system, for example, will likely produce future costs associated with the necessary labor to operate and maintain the system.

Some companies still do not track costs at the project level, relying instead on the general ledger system to impute approximate project costs. Tracking project costs is essential to encourage accurate estimating and provide budget data needed to make, monitor, and update project decisions. The foundation for effective PPM includes a finance system that tracks labor costs using fully burdened labor cost rates for roles and individual resources.

Estimating value is even more difficult than estimating costs. Establishing a PPM office, creating a database of available projects, and instituting forced ranking of projects helps, but the project portfolio won't be optimized without the ability to estimate project value. Thus, the key to reaping the full benefit of PPM is implementing a formal, organized, and logical method for measuring project value.

Establishing logic for quantifying project value, in my opinion, is not only critical to obtaining accurate and consistent project priorities, it is critical to justifying the role of the PPM office within the governance system for the enterprise. PPM is fundamentally different from project management with regard to the governance structure. Project management, primarily concerned with achieving project deliverables, is largely a tactical function. Delegating the function without providing formal systems to ensure compliance with executive preferences raises no issues. However, PPM is focused on making project decisions intended to achieve the fundamental and strategic objectives of the organization. To justify the delegation of the portfolio management function to a team other than the organization's most senior executives requires a formal prioritization process that makes explicit what would otherwise be the implicit preferences of senior management. Thus, PPM demands that systems be in place to help managers measure the value of projects consistent with the organizations fundamental objectives and strategy as established by senior executives. This leads to the most interesting part of the discussion—developing the metrics and models for measuring project and portfolio value. The next part of this paper explains how this may be accomplished.

References for Part 2

  1. P. Belliveau, A. Griffin and S. Somermeyer (eds.), The PDMA ToolBook for New Product Development, Wiley, 2002.
  2. "The TOC Multi-Project Management Method," The Product Development Institute, Inc., 628 Route 10, Suite 4, Whippany, NJ 07981, 1999.
  3. J. Rothman, Manage Your Project Portfolio The Pragmatic Bookshelf, Raleigh, NC, 2009.
  4. S. Berinato, "Do the Math," CIO Magazine, October 1, 2001.