Lesson 3: Sensitivity, Uncertainty, and Robustness

1 Modelling context

A model's inputs — parameters, initial conditions, boundary values, exogenous variables — are rarely known exactly. They are estimated from data, taken from literature, or assumed from domain knowledge. Each carries some uncertainty. The question that follows is unavoidable: how does the uncertainty in the inputs affect the uncertainty in the outputs, and do the model's conclusions hold?

Two related activities address this. Sensitivity analysis asks how the output responds to a change in a single input. Uncertainty analysis asks what the distribution of the output is, given the joint uncertainty in all inputs. Together they characterise the robustness of the model's conclusions.

Sensitivity is not uncertainty

Sensitivity analysis changes an input and observes the response. Uncertainty analysis propagates the input distributions through the model to obtain an output distribution. Sensitivity answers "how sensitive is the output to x?"; uncertainty answers "what is the distribution of the output given what we know about x?"

2 Learning objectives

By the end of this lesson you will be able to

Conduct a one-at-a-time sensitivity analysis and report the results.
Conduct a joint uncertainty analysis by Monte Carlo propagation.
Identify the inputs to which the model's conclusions are most sensitive.
Assess whether the model's conclusions are robust under plausible ranges of inputs.
Report sensitivity and uncertainty in a form appropriate for a decision-maker.

3 Prerequisite review

Sensitivity: The response of the output to a change in one input, holding others fixed. Often expressed as a partial derivative or as a percentage change in output per percentage change in input.
Uncertainty: The distribution of an input or an output. Parameter uncertainty, measurement error, and model-form uncertainty are the common sources.
Monte Carlo propagation: The technique of sampling from input distributions, running the model for each sample, and characterising the distribution of the resulting outputs. Introduced in Module 4.
Robustness: The property of a model whose conclusions hold across a plausible range of inputs. A robust conclusion is one that does not depend on a precise value of any single input.

4 Mathematical formulation

One-at-a-time sensitivity analysis

Let y = f(x₁, …, xₙ) be a model output as a function of inputs. A one-at-a-time (OAT) sensitivity analysis varies each input over a range, holding the others fixed, and observes the change in output.

The local sensitivity is the partial derivative ∂y/∂x_i, approximated by a finite difference:

∂y/∂x_i ≈ [ f(…, x_i + Δ, …) − f(…, x_i − Δ, …) ] / (2Δ)

The normalised sensitivity (elasticity) is

S_i = ( ∂y / ∂x_i ) · ( x_i / y )

This dimensionless quantity measures the percentage change in y per percentage change in x_i. It permits comparison of sensitivities across inputs with different units.

A one-at-a-time analysis produces a table or tornado chart showing which inputs the output is most sensitive to.

Joint uncertainty analysis

One-at-a-time analysis holds the other inputs fixed, which ignores the interactions between inputs and their joint uncertainty. A joint uncertainty analysis addresses this by:

Specifying a distribution for each input, based on data or expert judgement.
Sampling input values from their joint distribution (assuming independence for the simple case).
Running the model for each sample, obtaining a distribution of output values.
Characterising the output distribution: mean, variance, quantiles, and other summary statistics.

The output distribution is the fundamental result of uncertainty analysis. It states what the model says given what is known about the inputs.

Robustness assessment

A model's conclusion is robust if it holds across the range of plausible inputs. Formally, let C be a conclusion drawn from the model (e.g. "the project is profitable," "the queue length will not exceed 10"). The conclusion is robust if

C holds for all x in the plausible input region

In practice, the region is characterised by the input distributions, and robustness is assessed by checking whether C holds for the great majority of samples (say, 95 percent or more).

Figure 7.4. A tornado chart summarising one-at-a-time sensitivities. Each bar shows the range of output values as the corresponding input varies over its plausible range, with all other inputs held at their baseline values.

Reporting sensitivity results

Report the elasticity (normalised sensitivity) for each input. Rank the inputs by sensitivity. Identify the two or three inputs that dominate. The decision-maker needs to know which assumptions matter and which do not.

5 Worked example

Sensitivity and uncertainty in a project appraisal model

A project appraisal model computes the net present value (NPV) of a five-year project:

NPV = Σ_{t=1}^5 ( R_t − C_t ) / (1 + r)^t − I₀

where R_t is revenue in year t, C_t is cost, r is the discount rate, and I₀ is the initial investment. The baseline values are:

Baseline inputs
Input	Baseline	Plausible range
Annual revenue (R)	$120,000	$100,000 – $140,000
Annual cost (C)	$60,000	$55,000 – $70,000
Discount rate (r)	8%	5% – 12%
Initial investment (I₀)	$200,000	$180,000 – $230,000

Step 1 — One-at-a-time sensitivity.

Vary each input across its plausible range, holding the others at baseline:

One-at-a-time sensitivity results
Input	NPV at low	NPV at high	Range
Revenue	$27,400	$163,000	$135,600
Cost	$66,700	$27,100	$39,600
Discount rate	$59,900	$19,500	$40,400
Initial investment	$50,000	$0	$50,000

Baseline NPV (all inputs at their central values) is approximately $49,000. The tornado chart shows that revenue is by far the most influential input. Cost, discount rate, and initial investment have similar but smaller influences.

Step 2 — Joint uncertainty propagation.

Assign plausible distributions to each input. For example:

Revenue ~ Normal(120,000, 10,000²).
Cost ~ Normal(62,500, 4,000²).
Discount rate ~ Normal(8%, 2%²).
Initial investment ~ Normal(200,000, 12,000²).

Sample 10,000 times from these distributions, compute NPV for each sample, and characterise the output distribution:

Output distribution of NPV
Statistic	Value
Mean	$49,100
Standard deviation	$38,200
5th percentile	−$8,600
25th percentile	$22,400
Median	$48,700
75th percentile	$75,600
95th percentile	$113,000

Step 3 — Robustness of the conclusion.

The decision is whether to proceed with the project. A common criterion is that NPV should be positive. The distribution shows that approximately 10 percent of samples have negative NPV:

P(NPV < 0) ≈ 0.10

The conclusion "the project is profitable" holds in about 90 percent of plausible scenarios. Whether this is sufficient depends on the decision-maker's risk tolerance. If a 10 percent chance of loss is unacceptable, the model's conclusion is not robust for this decision.

Step 4 — Prioritising effort.

Because revenue is the dominant input, the decision-maker's attention should focus on obtaining a better estimate of revenue. Improving the estimate of the discount rate would reduce the output uncertainty by far less. This prioritisation is a direct output of the sensitivity analysis.

6 Verification check

Checks to perform

Verify the sensitivity at the baseline. With all inputs at their baseline values, the output should equal the baseline output.
Check the direction of effect. The sign of each sensitivity should match the expected direction of influence.
Check the sample size. The uncertainty analysis should have enough samples for the quantiles of interest to be stable. For 5th and 95th percentiles, at least 1,000 samples are typically needed.
Check the input distributions. The distributions used for the inputs should be justified by data or expert judgement, not chosen arbitrarily.
Test the interactions. One-at-a-time analysis ignores interactions. If inputs interact, the joint uncertainty analysis will show greater output spread than the one-at-a-time analysis predicts.
Report the seed and the sample size. As with any Monte Carlo analysis, reproducibility requires recording the seed.

7 Interpretation

Sensitivity and uncertainty analyses convert a single output number into a decision-relevant statement. Instead of "the NPV is $49,000," the analysis says "the NPV is likely to be between $22,000 and $76,000, with a 10 percent chance of a loss."

Three interpretive points:

Sensitivity identifies what matters. The input with the largest sensitivity is the one on which the model's conclusions depend most. Effort to reduce uncertainty should focus there.
Uncertainty quantifies the range of plausible outcomes. The output distribution is the honest summary of what the model says, given what is known about the inputs.
Robustness addresses the decision. A conclusion is robust if it holds across the plausible range of inputs. A model whose conclusions change under small input changes is fragile, and the decision-maker should know this.

Reporting sensitivity and uncertainty

Report: the baseline output, the sensitivity of the output to each input (with units), the output distribution (mean, standard deviation, quantiles), and the robustness assessment. The decision-maker needs all four pieces of information.

8 Uncertainty and limitations

Sensitivity and uncertainty analyses carry their own uncertainties.

Limitations of sensitivity and uncertainty analysis
Limitation	Effect
Input distributions assumed	The output distribution is only as good as the input distributions
One-at-a-time ignores interactions	Sensitivities computed separately may mislead when inputs interact
Local sensitivity is local	Partial derivatives apply near the baseline; nonlinear responses require broader analysis
Correlation between inputs	If inputs are correlated, treating them as independent is wrong
Model-form uncertainty	The analysis treats the model form as fixed; if the form is wrong, the analysis inherits the error

Uncertainty analysis cannot fix a wrong model

Sensitivity and uncertainty analyses characterise the response of the model to changes in its inputs. They do not address whether the model form is correct. A model with a wrong form produces a confident but misleading uncertainty distribution.

9 Common implementation errors

Watch for these

Using unrealistic input ranges. The plausible range should be based on data or expert judgement, not chosen for convenience.
Ignoring interactions between inputs. One-at-a-time analysis holds other inputs fixed. If inputs interact, this understates the joint effect.
Assuming independence when inputs are correlated. Correlated inputs require sampling from their joint distribution.
Reporting the sensitivity as a range without units. The elasticity (normalised sensitivity) is more interpretable than the raw partial derivative.
Not reporting the sample size and seed. As with any Monte Carlo analysis, reproducibility requires these.
Treating a robust conclusion as a correct conclusion. Robustness addresses sensitivity to inputs, not validity of the model.

10 Independent modelling task

Your turn

Choose a model you have built or a model from a previous module. Identify the model's output and its three to five most important inputs.
State the baseline value of each input and a plausible range or distribution for it. Justify the range or distribution.
Conduct a one-at-a-time sensitivity analysis. Compute the elasticity of the output with respect to each input. Produce a tornado chart.
Conduct a joint uncertainty analysis using Monte Carlo propagation. Sample from the input distributions, run the model, and characterise the output distribution.
Report the mean, standard deviation, and 5th/50th/95th percentiles of the output.
State the model's principal conclusion. Assess whether the conclusion is robust under the plausible input ranges.
Write a report including all of the above, plus a paragraph on which inputs the decision-maker should focus on and why.

This exercise forms part of Laboratory 13 and the sensitivity and uncertainty components of the capstone.

11 Reflection prompt

Reflect before continuing

A model's conclusion is robust under the stated input distributions. Does this mean the conclusion is correct?

Two or three sentences. Consider the difference between robustness and validity, and the fact that uncertainty analysis treats the model form as fixed.

12 Lesson summary

Key takeaways

Sensitivity and uncertainty are different activities. Sensitivity studies the response to a change in one input; uncertainty propagates input distributions to obtain an output distribution.
Normalised sensitivity (elasticity) permits comparison across inputs. The inputs with the largest elasticities are the ones on which the model's conclusions depend most.
Monte Carlo propagation produces the output distribution. Report mean, standard deviation, and quantiles.
Robustness addresses the decision. A conclusion is robust if it holds across the plausible input region.
Neither analysis addresses model-form error. A wrong model produces a confident but misleading uncertainty distribution.

Unit 7.1 complete

You have verified models, validated them against held-out data, and characterised their sensitivity and uncertainty. Unit 7.2 turns to the design of scenario experiments, the communication of results to technical and non-technical audiences, and the integration of the full modelling cycle into the capstone.

§ Self-quiz, practice, and homework

Practice exercises

For a simple model y = a x + b, compute the elasticity of y with respect to a and b. Which is larger?
For the project appraisal example, recompute the sensitivity of NPV to revenue if the revenue range were narrower (say, ±5 percent instead of ±20 percent).
For a model with three inputs, describe how you would conduct a joint uncertainty analysis. What sample size would you use and why?
For a model whose conclusion is "the queue length will not exceed 10," describe a robustness assessment.

Homework problems

Sensitivity analysis report. Choose a model with at least three inputs. Conduct a one-at-a-time sensitivity analysis with plausible input ranges. Produce a tornado chart. Identify the most influential input and explain its dominance. 500–700 words.
Uncertainty analysis. For the same model, assign distributions to the inputs and conduct a Monte Carlo uncertainty analysis with at least 1,000 samples. Report the mean, standard deviation, and 5th/50th/95th percentiles of the output. State whether the model's principal conclusion is robust.
Interaction assessment. For two inputs that might interact, compare the OAT sensitivities with the joint uncertainty analysis. Discuss whether the interaction is significant.

Mini project

Produce a sensitivity and uncertainty report for a model of your choice. Include: the baseline output, the sensitivity of the output to each input, the output distribution, and a robustness assessment. Recommend which input the decision-maker should focus on and why. Submit the report and the spreadsheet. This is part of Laboratory 13 and the sensitivity component of the capstone.