Home

More Papers

The Scientific Method

Printable version

My advice that organizations create a project-selection decision model is nothing more than a recommendation to follow the "scientific method." The scientific method is the process scientists use to develop accurate (that is, logical, reliable, and non-arbitrary) answers to tough questions. Figure 24 provides one representation of the scientific method.

The scientific method embodies several important principles. One is that the problem should be formulated in terms of the specific question you want solved (What is the value of a project?), not in terms of some other question that you think might be easier to answer (What are potentially useful metrics?). Second, because the scientific method is fact based, information relevant for solving the problem is collected (including understanding of the relevant theories for measuring value). Third, a model (hypothesis) is constructed to explain the relevant phenomenon. The model provides a causal explanation for what is observed (that some projects help organizations more than others). Finally, central to the scientific method is the concept of testing and refining the model. This can involve "thought experiments" or the comparisons of real-world observations with model predictions. Whenever inconsistencies are observed, understanding is refined and the model is revised accordingly.

Figure 24:   Developing a decision model is an application of the Scientific Method.

A decision model is not much different from the types of conceptual models developed in other areas addressed using the scientific method. Because the model is quantitative, it allows statistical measures of the reliability of the results to be established. Initially the decision model is constructed based on judgment, but it is tested and refined based on facts and observations. The key is that the model is available for scrutiny by experts other than those who developed it originally. Thus, decision models, unlike typical scoring systems, have a built-in tendency to get better. This is why scientists claim to "stand on the shoulders of the giants who came before them." The key to progress is defining rigorously logical relationships that can be quantified with estimates that can be tested and refined over time.

Scorecards and Strategic Alignment Are Not Decision Models

The majority of currently available, project prioritization and portfolio management software tools provide capability for defining both financial and non-financial metrics. The tools are often based on a balanced scorecard approach. Scoring scales are developed to measure project contribution to various non-financial objectives. The scores are weighted and added to financial metrics to provide a more "balanced" measure of project performance. This aggregate measure is then used to prioritize projects. Scorecards are useful in some contexts, but the way that they are defined means that they cannot be used to properly prioritize projects.

The first problem with using scorecards to prioritize projects is that the goal for selecting projects should be maximizing value, not creating a balanced project portfolio. Basic scorecard guidance advises that, "The measures represent a balance between external measures for shareholders and customers and internal measures of critical business processes, innovation, and learning for growth" [5]. Assigning weights to measures of this type implies a willingness to accept lower performance in one area (e.g., lower performance for shareholders) in return for better performance in another area (e.g., better business processes). Why would an organization want to accept less value (e.g. lower shareholder value) in order to obtain a higher score (i.e. better "balance") on some internal business process? Value maximization, and not balance, is the goal.

The second problem is that, contrary to typical scorecard mathematics, it is generally not correct to weight objectives that represent means for achieving more fundamental objectives. For example, suppose we include scorecards for measuring the impacts of a project on the quality of business processes as well as scorecards for the impacts on operating costs, customer service, etc. But, improving business processes is merely a means for achieving more fundamental objectives, including reducing operating costs and improving customer satisfaction, etc. Thus, a project might get a favorable score on process improvement, but zero weight should be assigned to this score if the value of that process improvement is completely reflected in the scores assigned to metrics that represent the fundamental objectives that explain why business process improvements are important. If the weight is not zero, there will be double counting.

Failure to account for the hierarchical nature of objectives (including the fact that means objectives contribute to the achievement of fundamental objectives) is a serious error being made by many who are designing tools for project portfolio management. For example, several websites advise, "There are four goals for portfolio management, value maximization, balance, strategic direction and the right number of projects." There is only one goal, value maximization. The proper balance, strategic direction and number of projects are whatever is required to maximize value. A decision model shows how lower-level objectives relate and contribute to the achievement of higher-level objectives (Figure 20 in the previous section provided an example). A decision model provides the capability to determine the levels of performance on lower-level objectives that enable the top-level objective (value maximization) to be achieved.

A third problem is more fundamental—Scorecards do not implement any defensible calculus for project valuation. Contrary to the 'weight-and-add" scorecard approach, it is generally not correct to add different types of value. This statement, which is well established by value measurement theories such as multi-attribute utility analysis, often comes as a surprise to people accustomed to adding and subtracting money values. In fact, being able to weight and add sources of values is an exception. It requires the condition in which the value of achieving any level of performance on any one objective does not depend on the degree to which any other objective is achieved, a condition sometimes referred to as "preferential independence." Multi-attribute utility analysis provides tests for determining whether different types of values may be added, or whether more complicated aggregation equations are needed.

Scoring methods are being advocated that involve weighting and adding scores for criteria such as project risk, internal rate of return, time-to-complete, urgency, and many other criteria that fail to pass the test of preferential independence. It makes no sense, for example, to weight and add a project's score for time-to-complete to weighted scores for other criteria that indicate the value added once the project is completed. Being quick is more valuable if the project adds a lot of value than if the project adds little or no value. Weight-and-add could only make sense, in this case, if the weights are not constants; that is, if the weight assigned to time-to-complete is a function of the ultimate value of the project.

A sound decision model addresses these issues by specifying a logically correct way of quantifying value. Prioritizing projects using a balanced scorecard approach will distort project decisions unless the weights and mathematical form of the aggregation equation are derived consistent with a defensible theory of valuation.

Strategic Alignment — The Most Popular Scorecard Approach

Many providers of project portfolio management software and even some of the recent books on project portfolio management promote an "easy way" to prioritize projects:

"Prioritize projects based on alignment with strategy." "If your budget doesn't have room for the high-end software…, you can always fall back on alignment as a sound way to rank your projects." "The approach needs to be really simple, so that small projects can come up with a strategic alignment rating in about 5-10 minutes." [6]

Strategic alignment (also called strategic fit) is probably the most popular of the project scoring methods being used today to prioritize projects.

Strategic alignment sounds great in principle. An organization's projects should be consistent with and implement its strategy. But, how do you measure alignment? This is where it gets shaky. Typical advice goes something like this [7]:

1. Identify the elements of your organization's strategy, for example, build brand value, invest in core technologies, build organizational effectiveness, and revitalize quality.
2. Rate each project based on the degree of impact you expect it to have on each strategy.
3. Add the individual ratings to provide an overall strategy score.
4. Combine this score with ratings on other criteria, such as net present value or risk.

This approach has all of the flaws identified above typical of most scorecard approaches, plus some additional problems:

• Given the subjectivity involved, it would be hard to get consistent and repeatable scores from different people for something like the "degree of impact" a project has on a vaguely defined concept like "revitalize quality."
• How do you combine an alignment score expressed in points with a financial metric like NPV, which is expressed in dollars?
• Isn't the goal of strategic alignment to get the project portfolio that best implements strategy? The scoring approach does something different: It favors projects that impact the most elements of strategy. Suppose there are individual projects that are each extremely well-aligned with a single element of strategy. Weighted scoring will not rank these projects highly (if there are other projects that have even a mediocre impact on many elements of strategy). Thus, strategic alignment does not produce the portfolio of projects that best promotes strategy.
• How do we know that the organization's expressed strategy is optimal? How do we know that choosing projects that "align" most closely with strategy will produce the best outcomes for the organization?

The fundamental problem is that strategic alignment is not a decision model. To review, a decision model is an analytic model that generates as output a measure of the value of a project; that is, a number with the property that Project A is preferred to Project B if and only if the computed measure is higher for Project A than it is for Project B. We can easily show that strategic alignment is not a decision model by applying the four steps of scientific method:

1. Theory: Is there any theory to support the idea that scoring projects based on alignment with strategy provides a measure of the value of those projects? You can find discussions of project valuation theories in many management science books and professional, peer-reviewed papers. Strategic alignment is not considered a theory of valuation.
2. Hypothesis: Suppose we adopt the tentative hypothesis that projects that align the most closely to the most elements of strategy will also provide the most value to the organization.
3. Thought experiments: It is easy to debunk this hypothesis through thought experiments. Suppose the projects that align most closely with strategy are not very effective at addressing needs, produce high risks, or cost too much. Conducting such projects is not likely to provide the most value for the organization.
4. Empirical evidence: Is there empirical evidence indicating that strategic alignment produces high-value project portfolios? I begin my training seminars on project portfolio management by asking, "Of those of you who are currently using a scoring tool to prioritize projects, how many think the recommendations being made by your tool are wrong?" On average, about 60% of the attendees raise their hands. Most users have only recently put their tools in place, so I expect the percentage to go up.

Project-Selection Decision Models

Project Valuation Metrics & Part 3 References