Lee Merkhofer Consulting Priority Systems
Implementing project portfolio management

Quantifying Uncertainties

The most precise and informative way to clarify risk is to characterize the risk with numbers, specifically, by describing the possible outcomes in quantitative terms, estimating when they will occur (risk timing), and assigning probabilities, in other words, by conducting quantitative risk analysis. Quantitative risk estimates help decision makers understand the impact of uncertainty and the range of potential consequences of project decisions. For example, suppose a local company is considering a project to establish a store in another county. If financial analysis indicates the new store needs to generate $1,000,000 in revenue to break even, and if their quantitative risk analysis tells them that there is a 10 percent chance that revenues will be less than $1,000,000, the company has a clear understanding of the risk it is facing if it decides to undertake the project.

Risk and the Project Selection Decision Model

As I explained in the previous sub-section, there are different types of project risk corresponding to the organization's different objectives. As in the example above, a financial risk could cause the revenues obtained from a new product to turn out lower than expected. A safety risk could result in an accident that injurers workers. A corporate image risk could damage the organization's reputation. In the case of project risk, the concern is failing to obtain the project's anticipated positive impacts on objectives, or obtaining unanticipated negative impacts on objectives. In the case of project deferral risk, the concern is that deferring a project results in unexpected negative impacts to objectives.

A project selection decision model, described in Part 3 of this paper, evaluates each candidate project by comparing the anticipated impacts on objectives from doing versus not doing the project. The model then aggregates and translates those impacts into an equivalent monetary value of the project. Though there is no one standard decision model that works for all organizations, project types, and situations, there are step-by-step model building procedures that apply in all cases (one popular step-by-step model building process was described here). Depending on the complexity of the project's impacts and the degree of precision required, the model could be fairly complex or quite simple. For example, the model inputs might consist simply of direct estimates of the degree to which the project advances each objective, with an aggregation equation with weights that combine those impacts into a quantitative measure of project value.

The project selection decision model, as presented in Part 3, is deterministic in that the inputs to the model are all assumed to be point estimates, or, perhaps, a series of point estimates indicated the project's impacts over time. Suppose, however, that there is uncertainty over one or more of the inputs or parameters of the model. Depending on how that uncertainty turns out, it will likely affect the project's performance relative to one or more of the organization's objectives. Thus, uncertainty over the inputs to the project selection model means there is uncertainty over the performance of the project relative to objectives and uncertainty over the value of the project.

There are methods, described in the next subsection of this paper, for propagating uncertainties over a model's inputs (and even uncertainty over the model's logic, see below) to obtain the uncertainties over the models outputs. Thus, we can quantify uncertainty over a project's performance relative to objectives and project value provided that we can quantify the uncertainties over the decision model's inputs. The key step is to express the model input uncertainties with probability distributions.

What I've just described is the general approach to quantitative risk analysis. Risk assessors create a model that calculates the impact of relevant factors on outcomes that we care about, such as profit and loss, health and safety, environmental consequences, and the like. Uncertainties over the model's inputs are then quantified and propagated through the model to quantify the uncertainties over the model's outputs. In other words, our approach to quantifying project risk follows the same general process that risk assessors use to quantify other kinds of risks. The only difference is that with traditional risk analysis, it is necessary to start by building a model that describes the sources and consequences of uncertainties. In the case of project selection, a model has already been constructed for the purpose of prioritizing projects without regard to risk.

To make the deterministic project selection decision model stochastic and capable of prioritizing projects with risk, the model may need to be expanded by adding variables to better isolate the sources of risks. For example, if a source of project schedule risk is the potential for project delays, the model may need to be expanded by identifying things that could lead to delays, such as extreme weather events. If the decision model was constructed by first creating an influence diagram, model expansion may simply mean adding to the decision model some of the more detailed factors identified in the influence diagram. If there are uncertainties related to the model itself, such uncertainties may be captured by defining alternative model logics. In short, the project selection decision model provides the foundation for conducting quantitative risk analysis of projects.

Quantifying project risks

Figure 35:   Propagating uncertainties through the project selection decision model.

Selecting Probability Distributions

Perhaps the biggest challenge for doing quantitative risk assessment is that the appropriate probability distributions for describing uncertainties and their probabilities are usually not readily apparent. The exception is the case where a good deal of historical data are available (e.g., as might be the case for random events such as weather, accident rates, and commodity prices). In such cases, probability distributions can be obtained from the data using statistical methods.

Though it may be tempting, limiting risk quantification to uncertainties that can be quantified with data can easily result in underestimating risk. For example, a week before hurricane Katrina struck, New Orleans hosted an offshore-drilling conference that included a panel discussion entitled, "What Has the Industry Learned from Ivan" (Hurricane Ivan had struck the previous September). The lesson was that rigs needed to be much better secured. The industry, however, had not yet made any changes prior to Katrina. The engineering approach to risk assessment told them that hurricanes of this size occur infrequently, which led them to believe that there was plenty of time before the next major storm hit. Sadly, Katrina proved this assumption to be incorrect.

In the absence of data, probabilities must still be assigned, and it makes sense to do so directly based on professional judgment. Based on judgment, experts in the appropriate subject matter can assign probabilities even if no frequency data exists. If you can imagine an event, you can assign a probability to it, if not in absolute terms—0.01%, 1%, 10%—then relative to another event whose probability can be measured ("Are the odds greater or less than being dealt three of a kind in a hand of poker?"). To assess judgmental probabilities from experts, decision analysts use a probability wheel for the purpose of creating comparative events whose probabilities can be measured. Before you dismiss the predictive value of subjective probabilities, read the section in Part 7 on predictive markets.

Literally hundreds of probability distributions have been defined mathematically. Some are used with such regularity that they have been given their own names, such as normal distribution, beta distribution, triangular distribution, uniform distribution, and many others. Many real-world random events do, in fact, follow one of these distributions. Others don't, but their uncertainties may be well approximated by named distributions.

Various methods have been proposed to assist the selection of probability distributions. Figure 36 provides a flow chart derived from Aswath [5]:

Probabilistic forecasts

Figure 36:   How to Select Probability Distributions.

As indicated, the selection of a probability distribution requires answering four questions:

  • The first is whether the outcomes to the uncertainty can take on only certain discrete values or whether they are continuous. For example, whether a new pharmaceutical drug gets FDA approval or not is a discrete event. You either get approval or you don't. Conversely, the revenues produced by the drug represent a continuous variable. Most estimates that go into the analysis of projects come from distributions that are continuous; market size, market share and profit margins, for instance, are all continuous variables.
  • The second question relates to the symmetry of outcomes, and if there is asymmetry, which direction it lies in; in other words, are the more extreme outcomes skewed to the left or the right of the mean? The symmetric distribution that is used most often is the bell-shaped, normal distribution. Most real-world uncertainties, however, are not symmetrical, and instead exhibit either large or very large positive or negative skew.
  • The third question is whether there are upper or lower limits on the data. There are some data items like stock prices and revenues that cannot be lower than zero whereas there are others like operating margins that cannot exceed a value (100%). If outcomes of the uncertainty are constrained, a related question is whether the constraints apply on one side of the distribution or both.
  • The final question relates to the likelihood of observing extreme values in the distribution; in some data, the extreme values occur very infrequently whereas in others, they occur more often.

The various probability distributions shown at the bottom of Figure 36 have parameters that allow fine tuning their shapes. Thus, once the form of the distribution has been selected, the distribution's parameters must be specified. Oftentimes, changing the parameter values allows for a wide-range of shapes for the distribution. As an example, the beta distribution is a family of continuous probability distributions often used to model random variables that are bounded above and below. Its shape changes dramatically depending on the values of its two parameters, α and β.

Beta probability distribution

Figure 37:   Shapes available from the beta probability distribution.

Again, if there are data for the uncertainty, statistical techniques may be used to set the parameters so as to fit the distributions to the data. In that case, you can determine how well the distribution you selected fits your data using goodness of fit tests or by visually comparing the empirical (based on sample data) and theoretical (fitted) distribution graphs.

Libraries of Pre-Selected Probability Distributions

Organizations can create libraries of pre-generated probability distributions to describe commonly encountered risks. For example, it has been reported that Chevron is developing a library of accident probability distributions for analyzing capital investment projects at oil terminals, and Shell has developed a library of distributions for hydrocarbon volumes for oil exploration projects. Generating such probability libraries helps ensure consistency in risk assessments and facilitates auditing the assignments over time. The cover story for an issue of the Journal ORMS Today argued that companies need to consider the prospect of a Chief Probability Officer, someone with the job of managing probability distributions used to support project portfolio management [6].

Advantages of Quantifying Uncertainties

In addition to the greater understanding enabled by quantifying uncertainties, other advantages exist as well. Once probabilities have been assigned, mathematical reasoning can be used to avoid many of the errors and biases described in Part 1 of this paper. For example, you can calculate how the number of observations affects the accuracy of estimates (to avoid small sample bias) and how the conditions required for an event affect the event's probability (to avoid conjunctive bias). Also, Bayes' theorem can be used to calculate how a probability should be revised or updated as new information becomes available.

As described in the previous sub-section, the amount of uncertainty caused by project risks and the specific project outcomes that are impacted may be used to better estimate what hurdle rates should be used and the types of benefits to which they should be applied. Quantifying uncertainty also allows more sophisticated methods (such as risk tolerance, explained in a following subsection) to be employed to account for risk aversion.

Although quantifying risks requires more inputs to describe proposed projects, note that the additional inputs need not be very complex. In the case of unlikely, (discrete) risk events (i.e., events that occur very rarely), it is less time-consuming yet normally entirely adequate to seek rough, order-of-magnitude estimates of probabilities and consequences (e.g., is the probability closer to one-chance-in-one-hundred or one-chance-in-one-thousand?). Likewise, if some aspect of a project's performance is uncertain (i.e., a continuous risk), instead of obtaining only a middle-value, point estimate, you can ask for a range of possible values (e.g., a 90% confidence interval) as well as a mean or most-likely value. (As indicated in the section of this paper on errors and biases, techniques should be used to guard against overly narrow ranges caused by overconfidence.) With practice, it takes no more time to specify a range of values to describe some uncertain measure of performance than it does to generate a single point estimate. The necessary continuous probability distribution for the uncertain variable can then be generated based on the range and a mean or most-likely value.

Project Managers Benefit from Using Probabilities to Describe Uncertainties

Although project managers may initially feel uncomfortable with probabilities, my experience is that this group can benefit significantly from moving away from using artificial point estimates. The following is a summary of an example devised by Mark Durrenberger of Oak Associates making this point [8]:

Imagine that a project manager is asked to complete a project in 3 weeks. Suppose the project manager feels that this estimate is unrealistically optimistic; that everything would have to go just right to make the deadline. The project manager may feel apprehensive about going to the project sponsor to address the problem. It may not be easy to explain why an optimistic, aggressive project schedule isn't a good one.

Suppose, instead, that the project manager estimates as a range the time required to complete each project step. Those ranges can be combined (by adding the means and variances) to determine the probability of completing the effort within any specified time. Rather than feeling "at the mercy" of the sponsor, the project manager can now say, "I understand your desire to complete the project within three weeks. However, my calculations suggest that we have less than a 5% chance of meeting that deadline."

The sponsor will want to know more, including how the probability estimate was obtained. This gives the project manager the opportunity to discuss the realities of the job and to negotiate tradeoffs (like providing more resources or eliminating some project deliverables so as to increase the likelihood of meeting the desired schedule).

Note that specifying ranges is not a license for the project manager to make baseless claims. Over time, performance can be compared with range estimates. A project manager whose performance routinely beats the means of his specified uncertainty ranges, for example, will be exposed as one who pads estimates.