Part 1 identified available tools for project portfolio management, Part 2 described key differences, and Part 3 summarized implementation costs and risks. This part identifies the weak link in the design of most tools.
A significant difference among available project portfolio management (PPM) tools is the logic used to make project recommendations (see the side box below). Quantitative methods exist for optimizing project portfolios, but few tools use the best of the available techniques. For most PPM tools the logic used to recommend projects is the weak link.
As noted previously, nearly all PPM Tools allow users to define and enter their own measures for ranking projects. In addition, PPM tools typically allow users to rank projects based on any criterion defined by weighting and adding measures that have been assigned to projects. However, the ability to define and create weighted combinations of measures is not as helpful as it might at first seem.
Tools Use Different Logics to Recommend Projects
The logic used to make project recommendations is crucial. A tool that is incapable of providing reliable recommendations will not be of much help in enabling the organization to generate more value from its project portfolios.
Weighting Project Measures Doesn't Enable Prioritization
Take the simple case of projects that produce financial benefits (e.g., incremental revenue and cost savings. You could evaluate each project by computing its net present value (NPV). However, the formula for computing NPV is not linear and additive, so if the PPM tool only allows weight and add equations it can't be used to compute NPV's. You could compute each project's NPV externally and enter it into the tool. However, this approach makes it hard to ensure consistency in the way that NPV's are calculated and impossible to conduct sensitivity analysis to see the impact on rankings of changing assumptions (e.g., What if we assumed a lower discount rate?). Thus, a tool limited to weight and add evaluations can't reasonably handle financial benefits.
Recognizing the importance of measuring financial benefits, nearly all PPM tools include at least a simple financial model for computing project NPV and other common financial measures. However, what about capability for quantifying other project benefits (benefits other than financial)? As with financial benefits, the equations for quantifying non-financial benefits do not generally comply with the restrictive linear, additive, "weight-and-add" form. For example, like cashflows, benefit streams over time ought to be summed and discounted. If the model for evaluating candidate projects is limited to weight and rate, it can't be used to estimate project value.
What's needed is a tool that contains all of the models needed to compute the types of benefits that accrue from the types of projects that your organization conducts. However, the types of benefits differ from industry to industry, as does the project information available for use as inputs for the models. This is why it is nearly impossible for general purpose tools to provide the analytics to properly value and prioritize projects.
The goal for selecting projects is to obtain the project portfolio that, subject to applicable resource constraints, creates the greatest possible (risk-adjusted) value for the organization. This, of course, requires the capability to compute project value. Although many tools are advertised as having the capability to compute measures of project value, few actually define value in a way that makes sense for PPM. In effect, most tools confuse the issue of value maximization with less appropriate (but easier-to-implement) concepts such balance, strategic alignment, maximizing points, etc.
Webster's Dictionary lists several definitions for the word "value." Near the top is, "the monetary worth of something." This is a reasonable and appropriate definition to use as a basis for evaluating projects. However, quantifying the equivalent monetary worth of a project that includes non-financial as well as financial benefits requires sophisticated analysis (using techniques such as multi-attribute utility analysis and real options analysis, see Part 4). Due to restrictions on the analytics that can be incorporated, most tools are incapable of applying these types of analyses. Webster's alternative definitions for "value" include "a numerical quantity assigned or computed" and a "degree of excellence." Although easier to implement, these definitions do not reflect the fundamental objectives of organizations (to create value) and do not provide a sound basis for finding optimum project portfolios.
Project Value Must be Expressed in Dollars
To be most useful for PPM, value must be measured in dollars. Unless all project benefits are expressed in common dollar units, you can't correctly combine financial and non-financial project benefits. Also, you can't determine whether benefits justify costs. Furthermore, without a common unit for measuring benefit, you can't determine how best to allocate a budget across organizational units responsible for different project portfolios.
Unlike the imprecise definitions of project value implicitly assumed by most PPM tools, defining value as monetary worth ensures that there is an objective basis for validating the numbers computed or assigned using the tool. If someone claims that something is worth X dollars, then he or she should be willing to buy it if it costs less than X dollars, but not if it costs more. A tool that can only assign points to projects doesn't allow such checks (What basis is there for validating the assignment, say, of 870 points to a project?)
Optimizing the Project Portfolio
Most tools that recommend project portfolios don't use optimization methods—they rank projects (based on, as indicated above, individual criteria or on weighted sums of those criteria, possibly dividing by project cost). At best, ranking is an approximate technique that can only work if neither the costs nor benefits of projects depend on the other projects that are conducted (see mathematical theory ). Most tools do not consider dependencies when ranking projects. Note that tools that claim to handle dependencies usually only allow the user to record yes/no dependencies among projects (and account for bundling requirements, e.g., "You can't do Project A unless you also do Project B"). Very few tools account for dependencies in project value or cost (e.g., Project A will be worth more if Project B is conducted as well).
Also, be aware that some tools can't easily handle different versions of the same project (e.g., a lower cost, reduced scope approach to accomplishing the same needs). Instead, these tools assume the only decision allowed for each project is go versus no-go. A no-go may be unacceptable in some applications, for example, prioritizing maintenance projects where the option of eliminating maintenance altogether may be unreasonable.
According to Wikipedia:
"[M]ost PPM methods and tools opt for various subjective weighted scoring methods, not quantitatively rigorous methods." 
Another common limitation relates to the types of constraints that can be specified. Most tools only allow for one type of constraint; namely, a constraint on total funding. It is rarer to find tools that allow for different types of dollar constraints (e.g., choose the optimal set of projects with out-year costs less than or equal to Y) or multiple constraints on different types of resources (e.g., choose a portfolio that can be conducted given our current workers and skill sets).
Unfortunately, as competition has heightened, some tool providers have tended to put more effort into making better-sounding marketing claims than into improving the way that their tools recommend projects. If you visit the websites of the major providers, you'll often see the following self-conflicting claims:
In truth, most portfolio management tools fail to employ even the most basic portfolio optimization techniques. Portfolio optimization is mathematically complex, especially when projects produce non-financial as well as financial benefits, when risks are important, and when project costs or benefits depend on when the project is initiated or on what other projects are conducted. Although the necessary algorithms are well-known and can often be obtained as "plug-ins" for software, most tools ignore such issues and cannot be configured or otherwise adjusted to obtain even a rough approximation to a mathematically correct solution.
Thus, although many tools are strong on project execution, most are very weak on project selection. While it is true that a well-designed PPM tool can provide a consistent, logical way to evaluate and compare project proposals, a poorly designed tool or one that doesn't fit the need can distort decisions, increase costs, and create considerable frustration. This is one of the primary reasons that many organizations are disappointed based on their first experiences using PPM tools.
So, how can an organization interested in project prioritization really be sure that the tool acquired makes sound recommendations? Understanding decision models and how they work is the first step. Part 5 describes in more detail how tools use models to evaluate projects and recommend project selection decisions.