Lesson 2: Validation Evidence and Model Comparison

1 Modelling context

Verification establishes that the implementation computes what the specification says. It does not establish that the specification corresponds to the real system. That is the role of validation: the comparison of the model's predictions against evidence from the system it is meant to represent.

The critical discipline of validation is that the evidence must be independent of the model's development. If the model was calibrated on a dataset, validation must use a different dataset. If the model form was chosen after examining the data, validation must use data that were not available during the choice. Without independence, validation is circular.

Validation is purpose-relative

There is no such thing as a validated model in the abstract. A model is validated for a purpose. A model adequate for one decision may be inadequate for another, and the same evidence may support one conclusion but not the other. The purpose must be stated before validation.

2 Learning objectives

By the end of this lesson you will be able to

State the purpose of a model and the standard of adequacy it implies.
Design a validation exercise using data not used in model development.
State the disconfirming evidence in advance of the validation test.
Compare two or more models against the same validation data.
Interpret validation results in terms of the model's purpose.
Recognise the sources of validation error that survive the comparison.

3 Prerequisite review

Calibration data: Data used to estimate the model's parameters. Validation cannot use these data.
Held-out data: Data reserved for validation and not used in any stage of model development. The use of held-out data is the defining discipline of validation.
Disconfirming evidence: A result that would count against the model's adequacy, stated in advance of the validation test. Without this, unexpected results are prone to reinterpretation as successes.
Model adequacy: The property of being fit for the purpose for which the model is intended. Adequacy is judged against the purpose, not against a universal standard.
Model comparison: The activity of evaluating two or more models against the same evidence, to determine which better serves the purpose.

4 Mathematical formulation

Formal statement of validation

Let M(θ) be the model with parameters θ, and let y_obs be observations from the real system that were not used to estimate θ. Validation asks whether

M(θ) ~ y_obs     within the tolerance required by the model's purpose

The symbol ~ indicates approximate agreement. Models are never exact; the question is whether the agreement is adequate for the purpose.

The three practices of rigorous validation

1. Held-out data. The data used in validation must not have been used in any stage of model development, including parameter estimation, model form selection, and sensitivity analysis. Using calibration data for validation is circular.

2. Disconfirming evidence stated in advance. Before running the validation, state what result would count as a failure. This prevents the common tendency to reinterpret unexpected results as successes.

Example: "The model will be considered inadequately validated if
the mean absolute error on held-out data exceeds 5 percent of the
observed mean, or if the residuals show systematic curvature over
the validation interval."

3. Comparison against the purpose. The validation standard is the model's purpose, not the in-sample fit. A model that reproduces the calibration data beautifully but cannot support the decision it was built for has not been validated.

Model comparison

When two or more models are available, the validation exercise compares them against the same held-out evidence. Comparison is useful because it reveals whether the additional structure of a more complex model is justified. Standard practice:

Same data. All models must be validated on the same held-out data.
Same metric. The comparison metric must be the same for all models. Choose a metric appropriate to the purpose.
Same scope. The models must be applicable to the same range of inputs. A model that applies only to a narrow range cannot be compared against one that applies more broadly.
Penalise complexity. More complex models have more parameters. A comparison metric that does not penalise complexity will favour the more complex model, even when the extra complexity is not justified.

Common validation metrics
Metric	Use
Mean absolute error (MAE)	Average magnitude of the discrepancy, in the units of the output
Root mean squared error (RMSE)	Similar to MAE but penalises larger discrepancies more
Relative error	Discrepancy as a fraction of the observed value; useful for comparing across scales
Maximum absolute error	Worst-case discrepancy; useful when the tails matter
Residual structure	Systematic pattern in residuals; indicates model form error
Prediction interval coverage	Fraction of observations falling within the model's prediction interval

Figure 7.3. The data split for validation. The calibration data is used to develop the model; the held-out data is used only to validate it. The validation criterion is stated in advance.

5 Worked example

Validating an epidemic model against held-out data

An SIR epidemic model is fitted to the first 40 days of an outbreak:

dS/dt = −β S I / N
dI/dt = β S I / N − γ I
dR/dt = γ I

with parameters β and γ estimated from the calibration data. The model is then validated against the following 20 days of data.

Step 1 — State the purpose.

The model is intended to support decisions about hospital capacity planning over the next two weeks. The relevant output is the predicted peak of the infected population and the timing of the peak.

Step 2 — State disconfirming evidence in advance.

Before seeing the validation data, the following criteria are stated:

Adequate. The predicted peak of I(t) within the validation window falls within ±20 percent of the observed peak, and the timing of the peak is within ±3 days.
Inadequate. Either criterion is violated, or the residuals show systematic curvature over the validation interval.

Step 3 — Compare against held-out data.

Validation against held-out epidemic data
Quantity	Model prediction	Observed	Difference
Peak infected	842	960	−12.3%
Day of peak	47	45	+2 days

Step 4 — Assess against the criteria.

Peak: within ±20 percent ✓
Timing: within ±3 days ✓
Residuals: plot shows no systematic curvature ✓

The model passes all three criteria. It is validated for the stated purpose.

Step 5 — Compare with an alternative model.

A second model (SEIR, with an exposed compartment) is fitted to the same calibration data and validated against the same held-out data:

Model comparison
Metric	SIR	SEIR
MAE over validation window	41.2	28.7
Peak prediction error	−12.3%	−4.8%
Number of parameters	2	3

The SEIR model performs better on the validation data but has an additional parameter. Whether the improvement justifies the complexity depends on the purpose. For the hospital-planning decision, both models are adequate; the SIR model is preferred for its simplicity. If the decision were more sensitive to the peak timing, the SEIR model would be preferred.

This example illustrates that model comparison is purpose-relative. Neither model is universally better; the better model depends on what the model is being used for.

6 Verification check

Checks to perform on your validation

Is the validation data truly held out? Confirm that it was not used in any stage of model development.
Was the disconfirming evidence stated in advance? Document when the criteria were set.
Is the metric appropriate to the purpose? An MAE may be appropriate for average performance, but a purpose that depends on extremes requires a tail metric.
Are the residuals examined? Structure in the residuals is evidence against the model form, even if the summary metrics are acceptable.
Is the validation scope consistent with the purpose? If the model was validated on one time period, region, or scenario, its applicability outside that scope is not established.
Are the models compared on the same footing? Same data, same metric, same scope, complexity considered.

7 Interpretation

A validation result is a statement about the model's adequacy for a stated purpose within a stated scope. It is not a statement that the model is true.

Three interpretive points:

Validation is purpose-relative. The same model and the same data may yield an adequate verdict for one purpose and an inadequate verdict for another.
Validation is not proof. A model that passes validation on a specific dataset may fail on another. The validation provides evidence for the specific scope tested.
Validation can fail productively. A failed validation is not a failure of the project; it is a discovery that the model form or parameters need revision. The information is valuable.

Reporting a validation result

A complete report states: the purpose, the calibration data, the held-out data, the disconfirming evidence stated in advance, the metric used, the observed result, and the verdict. The scope of the validation should also be stated: over what range of conditions is the model validated?

8 Uncertainty and limitations

Several limitations apply to any validation exercise.

Limitations of validation
Limitation	Effect
Limited validation data	The validation is only as broad as the data it uses
Possible coincidence	A model may reproduce held-out data without a correct mechanism
Measurement error in validation data	The reference itself has uncertainty that is often ignored
Extrapolation risk	Validation within one range says nothing about another
Multiple-testing effects	Repeatedly adjusting the model until it passes validation is not validation

Validation is not a pass-fail exam

A model that fails a validation test is not thereby useless. The failure may identify a specific limitation, and the model may still be adequate for a narrower purpose. Conversely, a model that passes has not been shown to be correct in any absolute sense. Validation is a statement about evidence, not about truth.

9 Common implementation errors

Watch for these

Validating on calibration data. A model tested on the data it was fitted to appears better than it is.
Re-running the validation after seeing the result. Adjusting the model until it passes is not validation; it is fitting to the validation data.
Not stating disconfirming evidence in advance. Without this, unexpected results are prone to be reinterpreted.
Using the wrong metric for the purpose. A purpose that depends on extremes requires a tail metric.
Ignoring residual structure. A good summary metric can coexist with a systematic pattern in the residuals, indicating a model-form error.
Overstating the scope of the validation. A model validated on one scenario is not validated on all.
Comparing models on different data. Comparison requires the same validation data for all models.

10 Independent modelling task

Your turn

Choose a model you have built and a dataset relevant to it. If no dataset is available, generate one with a known ground truth or use data from a published case study.
Split the data into a calibration portion and a held-out validation portion. Document the split.
State the purpose of the model and the metric that will be used to judge adequacy.
State the disconfirming evidence in advance. What result would count as a failure?
Run the model and compare its predictions against the held-out data. Compute the metric and plot the residuals.
Assess the model against the criteria. Report the verdict.
Write the validation report: purpose, data, metric, disconfirming evidence, result, verdict, and scope of validation.

This exercise forms part of Laboratory 13 and the validation component of the capstone.

11 Reflection prompt

Reflect before continuing

A model passes a validation exercise on the data you reserved for validation. Does this mean the model is correct?

Two or three sentences. Consider the scope of the validation, the possibility of coincidence, and the distinction between adequacy for a purpose and correctness in any absolute sense.

12 Lesson summary

Key takeaways

Validation compares the model against evidence from the real system. The evidence must be independent of the model's development.
Disconfirming evidence should be stated in advance. Otherwise unexpected results are reinterpreted as successes.
Validation is purpose-relative. There is no universal standard of adequacy; the standard is the model's intended purpose.
Model comparison requires the same data, metric, and scope. Complexity should be penalised.
A validation result is evidence, not proof. Passing validation on one dataset does not guarantee passing on another.

Coming next

Lesson 3 develops the sensitivity and uncertainty analyses that characterise the robustness of the model's conclusions.

§ Self-quiz, practice, and homework

Practice exercises

For a linear regression fitted to the first 20 points of a dataset, describe a validation exercise using the remaining 10 points. State the disconfirming evidence in advance.
For a model whose purpose is hospital capacity planning, state an appropriate validation metric. Justify the choice.
For two competing models (e.g. linear vs. quadratic fit), describe how you would compare them. State the metric and the complexity penalty.
Explain in two sentences why validation on calibration data is circular.

Homework problems

Validation report. Choose a model and a dataset. Split the data into calibration and validation portions. State the purpose, the metric, and the disconfirming evidence in advance. Run the validation and produce a report. 500–700 words.
Model comparison. Fit two models to the same calibration data and compare them on the same held-out data. Compute MAE and maximum absolute error for each. Discuss which model you would recommend and why.
Scope analysis. For a model you have validated on one scope (e.g. one region, one season, one range of inputs), identify another scope where its adequacy is uncertain. Describe what additional validation evidence would be needed.

Mini project

Produce a validation report for a model of your choice. Include the purpose, the data split, the disconfirming evidence stated in advance, the metric, the result, the verdict, and the scope of validation. If the model fails, document the failure and what it reveals. Submit the report and the data files. This is part of Laboratory 13 and the validation component of the capstone.