How to Evaluate and Compare Project Portfolio Management (PPM) Software Tools

Purchasing a project portfolio management (PPM) software tool is not the first step to implementing PPM. Among other things, you must secure executive and stakeholder support, define the changes you need to make to organizational structure and processes, and design the framework that you will use to value and prioritize your projects (see Keys to Implementing Project Portfolio Management).

However, when you are ready, selecting a tool that fits your needs is key to reaping the rewards of PPM. The tool you choose will not merely be the most visible component of your portfolio management system, it will determine how well that system functions. Choose the right tool and you have the potential to gain control over your projects and derive greater value while reducing costs and risks. Choose the wrong tool and, at minimum, you waste significant time, money, and effort. Also, a low-quality tool can produce inaccurate recommendations that mislead decision makers into making poorer choices than would be made without the tool. Simply put, a bad software choice can cause your portfolio management implementation effort to fail altogether.

Buyer remorse is common...

In a survey¹ of purchasers of PPM tools, only 38% of respondents said they obtained the value expected, and 29% said that their purchased tool had become "shelfware" (unused software).

This resource is intended to help you evaluate and compare PPM tools. It provides a comprehensive set of criteria, structured evaluation process, and simple aid (Excel® spreadsheet program) that you can use to compare candidate tools. The aid can help you document your considerations and reasoning, useful, for example, if you are conducting a formal, tool-selection process. Importantly, the spreadsheet includes criteria for evaluating tool capability to prioritize projects and optimize project portfolios, considerations that are often poorly addressed by PPM tool reviews and evaluation aids.

What Project Portfolio Management Is

PPM is a tool-supported process for making project decisions so as to generate the greatest possible value (return) from the organization's portfolio of projects. Portfolio optimization methods have proven very successful in the world of financial investing. The idea behind PPM is to apply similar methods to project investments.

...yet, purchasers continue to believe in PPM

Despite the high fraction of disappointed buyers, only 12% said, "We don't believe in the value of these tools."¹

Project Portfolio Management Tools

The number of tool options is staggering. More than 100 software providers currently offer PPM tools (see Project Portfolio Management Tools - Which Approach is Best). The available tools are targeted at different industries and types of projects, and they have very different strengths and weaknesses.

What PPM Tools Do

What all PPM tools do is allow you to:

Create, manage, and view data from a database of proposed, planned, and ongoing projects.

What PPM tools do to varying degrees is help you to:

Prioritize projects and optimize the project portfolio.
Plan projects and manage the execution of approved projects.
Manage the supply and demand for project resources.

The Evaluation Aid

The aid consists of spreadsheets containing 128 tool evaluation criteria organized according to the above categories (32 criteria per category).² You can assign importance weights to the criteria and score candidate tools against the criteria. The program computes the total points assigned to each tool by multiplying weights and scores, and adding.

Using the Aid

There are 3 steps:

1. Decide What's Important and Assign Weights.

No tool excels on every criterion. However, not all criteria (or even all criteria categories) are likely to be relevant given your needs. As you go through the criteria, determine how important each is. The following scale is provided for assigning weights:

Importance of Criterion	Weight
Not needed, not important, or already provided by your existing tools	0
Nice, but of minor importance; not having would result in minimal loss	1
Useful; not having would result in some loss and/or extra work	2
Important; not having would result in significant loss and/or extra work	3
Extremely important; not having would result in major loss and/or much extra work	4
Essential; not having disqualifies the tool	E

2. Screen Tools and Obtain Information

Screen tools to select candidates for evaluation. You can screen out tools aimed at industries different from yours or types of projects different from those within your portfolios. Also, you may be able to screen tools depending on delivery method, for example, desktop software, web-based software, or on-demand services (where the data is stored on a supplier's server and the tool is available for an on-going fee). Identify the candidates by naming them above the columns in the spreadsheet. Contact providers to obtain the necessary information for scoring the candidate tools.

3. Score Candidate Tools

The following scale may be used to score candidate tools:

Estimated Performance Against Criterion	Score
Criterion not achieved or supported	0
Criterion partly achieved or supported	1
Criterion mostly achieved or supported	2
Criterion fully achieved or supported	3

As illustrated in Figure 1, any tool that fails to at least partially achieve a criterion rated "essential" is rejected.³

Tools that fail to achieve essential criteria are rejected

Figure 1: Example—A tool that fails to at least partially achieve an essential criterion is rejected.

Warning: Results Will Be Imprecise

Be aware that the aggregate scores and rankings produced by simple rate-and-weight schemes such as this are notoriously imprecise (especially if there are a great many criteria with some overlap or double counting in the way criteria are defined). Therefore, don't make your choice based solely on total points or rankings. Instead look closely at the short list of all highly ranked candidates. Then, make your decision using the same principles that govern choosing projects for PPM: Select the tool that you believe will generate the greatest value, considering total life-cycle costs, resource constraints, and risks.⁴

Tool Selection Process

Reasons customers say their tools fail to deliver full value:

The tools:¹

Don't integrate properly with other tools
Are too full-featured, too complex
Are too hard to customize
Weren't delivered with adequate training
Don't fit our processes

Although guidance such as this can be helpful, choosing the right tool will take work. Consider proceeding as follows. Use the web to research vendors and create a short list of the most viable candidates. Create a request for information (RFI) with questions based on the criteria that are of most importance to you, and send it to the vendor candidates. Compile the responses and trim the list down to 2-3 vendors. Obtain demonstrations from the finalists. There is no substitute for first hand experience, so ask the tool providers to configure the tool to your application and use it with data from real projects. The demos will point to your preferred choice, but challenge the finalists to address your areas of concern.

By following a structured process, and by having a thorough understanding of tool selection criteria, you can significantly reduce the chance of being "burned" by acquiring the wrong tool for your situation.

Notes

From Figures 5 and 7 in, "ALM/PPM Tools: Investments Gone Awry," a white paper reporting results of interviews with purchasers of application life-cycle management (ALM) and project portfolio management (PPM) tools by large IT organizations, conducted by Forrester Consulting for Wipro Consulting Services, Wipro Technologies, September 2008.
You can add criteria to the spreadsheets if there are things important to you that have been left out.
If an otherwise attractive tool doesn't achieve an essential or highly-weighted criterion, before eliminating that tool, you might want to inquire if the provider may be willing to incorporate the desired capability, and, if so, how long it would take and what the added costs would be.
The decision to acquire a PPM tool is itself a project portfolio management decision. I've not heard of a PPM tool provider offering the use of its tool to evaluate PPM tools, but the concept is intriguing!

Appendix: The Criteria