Lesson 3: Verification, Validation, and Reproducibility

1 Modelling context

In everyday speech, "verification" and "validation" are often used as synonyms. In modelling they are not synonyms, and treating them as such produces two common and costly mistakes. The first is to believe that a model which computes correctly has been shown to represent the system correctly. The second is to believe that a model which reproduces observed data has been shown to compute correctly. Neither implication holds.

The three activities in this lesson are distinct and must be kept distinct:

Verification asks: does the implementation compute what the specification says it should compute?
Validation asks: does the model represent the real system well enough for its stated purpose?
Reproducibility asks: can another analyst, given the same inputs and files, obtain the same results?

Each question has its own kind of evidence, and each can fail independently of the others. A model can be verified and not validated (it computes exactly what was specified, but the specification is wrong). It can be validated and not verified (it matches observed data, but only by accident of a coding error). It can be both verified and validated and still not reproducible (it is correct but no one else can rerun it).

The order matters

Verification should come first. There is no point in testing whether a model represents reality well if the implementation does not even compute what was specified. A validation that appears to succeed on an unverified model is a coincidence, not evidence.

2 Learning objectives

By the end of this lesson you will be able to

Distinguish verification, validation, and reproducibility, and give an example of each.
Produce at least three kinds of evidence for verification of a spreadsheet model.
Design a validation exercise that uses data not used in calibration.
State what would count as disconfirming evidence before running the validation.
Identify the file, metadata, and documentation requirements for a reproducible model.
Audit an existing model against the three activities and report the evidence found for each.

3 Prerequisite review

Specification: The document that states what the implementation should compute: the model form, the parameters, the units, and the intended outputs. Verification is the comparison of the implementation against this.
Calibration data and validation data: Data used to estimate model parameters versus data reserved for testing whether the calibrated model reproduces observations. The two must not overlap.
Analytical solution: A closed-form solution to the model equations. When one is available for a limiting case, it provides a strong verification and validation check: the numerical implementation should converge to it.
Limiting case: A configuration of the model in which the expected answer is known independently of the numerical implementation. Examples: zero input, infinite time, or a parameter value that reduces the model to a simpler known system.
Seed: The value that initialises a pseudorandom number generator. For stochastic models, the seed is part of the inputs and must be recorded to make results reproducible.

4 Mathematical formulation

Verification

Let S be the specification of the model — a formal statement of what output the implementation should produce for any given input. Let I(x) be the output produced by the implementation on input x. Verification is the activity of establishing that

I(x) = S(x)     for all x in the intended input domain

Since the domain is usually infinite, verification proceeds by testing on a finite set of cases chosen for their diagnostic value. Useful cases include:

Boundary cases. Zero input, empty data, the first and last period, the extremes of the parameter range.
Cases with known answers. Inputs for which the analytical solution is known and can be compared directly.
Conservation checks. If the model should conserve a quantity — mass, energy, total population, total currency — check that it does so at every step.
Symmetry checks. If the model is invariant under a permutation of its inputs, verify that the output is invariant.
Degenerate cases. Parameters set to values that reduce the model to a simpler known form.

Validation

Validation compares the model's output against evidence from the real system. Formally, if y_obs denotes observed behaviour and M(θ) denotes the model with parameters θ, validation asks whether

M(θ) ~ y_obs     within the tolerance required by the model's purpose

The "~" is not equality: models are never exact. The question is whether the agreement is adequate for the purpose. This is why validation cannot be done without a stated purpose — without one, there is no standard against which to judge adequacy.

Three practices distinguish rigorous validation from self-deception:

Use held-out data. Validation data must not have been used in calibration. A model tested on its training data appears better than it is.
State disconfirming evidence in advance. Before running the validation, write down what result would count as a failure. This prevents the common tendency to reinterpret unexpected results as successes.
Compare against the purpose, not the fit. A model that reproduces the data beautifully but cannot support the decision it was built for has not been validated for its purpose, regardless of how good the fit is.

Reproducibility

Reproducibility is a property of the workflow, not of the result. A model is reproducible when another analyst, given the model files and the documentation, can obtain the same outputs from the same inputs. Formally, if R_A(M, x) denotes the result analyst A obtains by running model M on input x, reproducibility requires

R_A(M, x) = R_B(M, x)     for any two analysts A, B

This is a strong requirement, and it depends on several things being recorded:

The inputs. Not just the values, but their units, sources, and the date of observation.
The software. The name and version of the spreadsheet or programming environment. Behaviour can change between versions.
The seeds. For stochastic models, the random seed used in each run. Without it, a rerun produces different sample paths and the result cannot be compared.
The procedure. A description of how to run the model, including the order of operations, the parameter settings, and any manual steps.
The outputs. The results as recorded, including any rounding or formatting applied.

Figure 1.8. Verification, validation, and reproducibility are independent activities with independent evidence. A model can pass any one and fail the others.

5 Worked example

Auditing a cooling model against all three activities

We return to the Newton's law of cooling model from Lesson 2, this time treating the model as an artifact to be audited. The goal is to produce specific evidence for verification, validation, and reproducibility.

Verification.

Analytical comparison. The model has the closed form T(t) = T_env + (T₀ − T_env) · e^−kt. Set T_env = 20, T₀ = 90, k = 0.05, and evaluate the numerical solution at t = 10 with a small step size. Confirm it matches the closed form to within the expected truncation error.
Limiting case — infinite time. Run the numerical solution for a very long time. It should approach T_env = 20.
Limiting case — zero cooling. Set k = 0. The temperature should remain constant at T₀.
Convergence. Halve the step size and confirm the error decreases at the expected rate for the method used.

The evidence produced is: a comparison to an analytical solution, two limiting-case checks, and a convergence test. Each of these addresses a different aspect of the question "does the implementation match the specification?"

Validation.

Hold out data. Of the 30 minutes of observed cooling, use the first 20 for calibration of k and reserve the last 10 for validation. Do not touch the last 10 during calibration.
State disconfirming evidence in advance. Write down: "The model will be considered not validated if the predicted temperature at minute 30 differs from the observed value by more than 2°C, or if the residuals over the held-out window show systematic curvature."
Compare. Run the calibrated model over the held-out window. Record the residuals.
Judge against the purpose. If the purpose is to decide whether to add ice at minute 25, the relevant question is whether the model's prediction over minutes 20–30 is accurate enough to support that decision. It is not whether the in-sample fit is good.

Reproducibility.

Record the inputs. All parameter values, the units used, and the source of the temperature data.
Record the software. The version of the spreadsheet used, including any installed solver or add-in version.
Record the procedure. The step size for the numerical solution, the order of operations, and any manual steps (such as a Solver run).
Record the outputs. The calibrated value of k, its uncertainty, and the predicted and observed temperatures over both the calibration and validation windows.
Test reproducibility. Ask another analyst (or a clean copy of the spreadsheet) to rerun the whole procedure from the documentation alone. The results should match.

Notice that the three activities require three different kinds of evidence, and that none of them substitutes for the others. A model that passes all three has earned more credibility than a model that passes only one.

6 Verification check

This lesson is itself about verification, so the "verification check" here applies to your own auditing process. The question is: does your evidence actually support the claim you are making?

Checks to perform on your own verification

Is your verification evidence independent of the implementation? If you "verified" the numerical solution by comparing it to itself with a different step size, you have tested convergence, not correctness. A correct verification compares against something external — an analytical solution, a limiting case, or a known numerical result from the literature.
Did you test the boundary cases? Zero input, the last period, the extremes of the parameter range. Most implementation errors occur at the boundaries, not in the middle.
Did you test a case where you knew the answer before you ran the model? "Knew the answer before running" is the operative phrase. If you only found out the answer by running the model, you have not verified anything.
Is the validation on held-out data? Confirm that the data used in validation was not used in calibration or in any iteration of the model form.
Did you state disconfirming evidence in advance? If you decided what would count as a failure only after seeing the result, the validation is compromised.
Can the model be rerun from the documentation alone? Test this on a fresh copy of the file, without relying on your memory of how it was built.

7 Interpretation

The three activities in this lesson are not bureaucratic categories. They correspond to three fundamentally different questions about a model, and each answer carries different weight in a decision.

What each activity supports
Activity	What it supports	What it does not support
Verification	"The implementation computes what we specified."	"The specification is correct."
Validation	"The model reproduces behaviour consistent with the real system."	"The model will generalise to conditions outside its validation data."
Reproducibility	"Another analyst can obtain the same result from the same files."	"The result is correct."

The most important interpretive point is that a model is not fully characterised by any one of these activities alone. A model is verified and validated and reproducible, or it is none of these, or some combination. The precise combination is what determines how much confidence the model's outputs deserve.

Reporting template

A model's credibility statement is clearest when it separates the three: "The implementation was verified by [evidence]. The model was validated against [held-out data] and reproduced behaviour within [tolerance], on the condition that [scope]. The model and its documentation are reproducible; a rerun from the documentation alone produced the same results."

8 Uncertainty and limitations

Each of the three activities carries its own limitations. Being explicit about them is part of producing a defensible model.

Limitations of verification

Finite test coverage. Verification tests a finite set of cases. The fact that a model passes them does not prove it is correct for every input; it only establishes that specific known failure modes are absent.
Errors in the test cases themselves. If the analytical solution used for comparison has been derived incorrectly, or the limiting case has been misstated, verification will confirm the wrong thing.

Limitations of validation

Limited validation data. The validation is only as broad as the data it uses. A model validated on data from one region, one season, or one operating condition has not been validated outside those.
Validation is conditional on the purpose. A model adequate for one decision may be inadequate for another. Validation is always validation for a purpose.
Possible coincidence. A model can reproduce held-out data without capturing the correct mechanism, especially if the held-out data is similar to the calibration data.

Limitations of reproducibility

Reproducibility is a property of the workflow, not the result. Two analysts obtaining the same wrong answer does not make the answer right.
Reproducibility can be lost over time. Software versions change; file formats become obsolete; cloud services shut down. A reproducible model today may not be reproducible in five years without deliberate preservation.

The most common conflation

The most common error is to treat verification and validation as one activity. A model that matches observed data is assumed to be correctly implemented, and a model that is correctly implemented is assumed to represent reality. Neither inference is valid. The two activities must be performed and reported separately.

9 Common implementation errors

Watch for these

Verifying against the implementation itself. Comparing the numerical solution at two step sizes tests convergence, not correctness. Verification requires an external standard.
Validating on calibration data. A model tested on the data it was fitted to appears better than it is. Always reserve data.
Deciding what counts as a failure after seeing the result. This is confirmation bias in statistical form. State the disconfirming evidence before running the validation.
Treating reproducibility as a documentation formality. If the model cannot actually be rerun from the documentation alone, the documentation is not sufficient.
Reporting only one of the three activities. A model's credibility statement should separate verification, validation, and reproducibility. Collapsing them into a single "the model was tested" obscures which claims are supported.
Assuming that passing verification implies passing validation, or vice versa. The activities are independent. A well-verified implementation of the wrong model is still the wrong model.
Treating a graphical fit as validation. Visual agreement can coexist with systematic error, especially over short observation windows. Quantitative residual analysis is required.

10 Independent modelling task

Your turn

Take the model you built in Lessons 1 and 2 — with its three-layer architecture and its estimated parameters.
Produce three pieces of verification evidence: an analytical comparison (if possible), a limiting-case check, and a boundary-case test.
Design a validation exercise. If your model was estimated on data, reserve part of it. State in advance what would count as disconfirming evidence. Run the exercise and report the residuals.
Produce a reproducibility package: the model file, a documentation sheet listing all inputs, sources, units, software versions, and procedure, and a short note explaining what another analyst would need to do to rerun the model.
Test reproducibility by making a fresh copy of the file and following your own documentation to rerun the model. Note any steps that turned out to require information not in the documentation.
Write a one-paragraph credibility statement that separates the three activities and states the model's claim-support scope.

This is the core deliverable of Unit 1.2 and the direct basis for Assignment 1.

11 Reflection prompt

Reflect before continuing

Of the three activities — verification, validation, reproducibility — which do you habitually skip or treat as a formality in your own work?

Then consider what it would take to do that activity properly on your most recent piece of modelling work. Two or three sentences. The point is to notice the gap between the standard and your current practice — not to close it today, but to see it clearly.

12 Lesson summary

Key takeaways

Verification, validation, and reproducibility are three distinct activities. Each answers a different question and requires different evidence.
Verification precedes validation. There is no point testing whether a model represents reality if the implementation does not compute what was specified.
Validation must use held-out data. A model tested on its calibration data appears better than it is.
State disconfirming evidence before running the validation. Otherwise unexpected results are reinterpreted as successes.
Reproducibility is a workflow property. It requires recorded inputs, software versions, seeds, and procedure — not just the model file.
Report the three activities separately. A model's credibility statement should make clear which claims are supported and by what evidence.

Unit 1.2 complete

You have built a three-layer model, estimated its parameters, and produced evidence of verification, validation, and reproducibility. Module 2 turns to numerical methods for ordinary differential equations — the first of the course's method-specific modules, and the foundation for everything that follows.

§ Self-quiz, practice, and homework

Practice exercises

For the loan model or the cooling model, write three specific verification checks. State the input for each and the expected output.
For the same model, design a validation exercise that uses held-out data. State the disconfirming evidence in advance.
Produce a reproducibility package for a simple model: the file, a documentation sheet, and a readme that would let another analyst rerun it.
Explain in two sentences why reproducibility is not the same as correctness.

Homework problems

Full V&V&R report. Apply verification, validation, and reproducibility practices to a model of your choice. Produce: (a) three pieces of verification evidence; (b) a validation design with disconfirming evidence stated in advance; (c) a reproducibility package. Write a 1000-word report that presents the evidence and justifies each claim.
Independent rerun test. Ask a classmate (or a clean copy of yourself) to rerun your model following only your documentation. Record what they needed that was not in the documentation and update it accordingly.
Failure modes analysis. Identify five specific ways your model could be verified but not validated, or validated but not verified. For each, describe the scenario and the evidence that would distinguish it.

Mini project

Choose a model you have built in a previous course or project. Produce a full credibility statement, separated into three sections: verification evidence, validation evidence, and reproducibility package. If any of the three is missing, produce it. Submit the model file, the documentation, and a two-page credibility statement. This is the direct basis for the verification and validation component of Assignment 1.