Module 7 Capstone: A Model Credibility Dossier

§ Project brief

The Module 7 capstone is a credibility dossier for a model of your choice. It brings together the three credibility activities — verification, validation, and sensitivity/uncertainty — into a single document, along with a technical report for the analyst and a plain-language executive summary for the decision-maker.

You may choose a model you have built in an earlier module, a model from another course, or a model from published literature that you can reproduce. The dossier is what a review committee would need in order to judge whether the model is fit for its stated purpose.

The dossier is the deliverable

A model without a credibility dossier is a model whose reliability cannot be assessed. The dossier records the evidence, structures the claims, and makes the model auditable.

§ Learning outcomes addressed

This capstone assesses

Design and apply verification strategies appropriate to a model.
Design and execute a validation exercise with held-out evidence.
State disconfirming evidence in advance.
Conduct sensitivity and uncertainty analyses.
Assess the robustness of the model's conclusions.
Communicate results to technical and non-technical audiences.

§ Suggested models

Model A

A model from Module 2 or 3

Take an ODE or PDE model you built earlier and extend it into a full credibility dossier. Best if the model already has an analytical reference.

Model B

A model from Module 4 or 5

Take a Monte Carlo study or discrete-event simulation and extend it with a formal validation exercise and uncertainty propagation.

Model C

A published model

Reproduce a model from a published paper with a data set you can access. Apply the full dossier framework to your reproduction.

Model D

A model of your own

A model from your own work or another course. The dossier framework is general; any model with a stated purpose is suitable.

§ Deliverables

Deliverables and weights
Deliverable	Weight	What it should contain
Purpose statement	10%	The model's intended purpose, boundary, assumptions, and claim-support scope.
Verification report	20%	Five or more diverse tests with inputs, expected outputs, acceptance criteria, and observed results.
Validation report	20%	Held-out data, stated disconfirming evidence, metric, observed result, and verdict. Scope of the validation.
Sensitivity and uncertainty report	20%	One-at-a-time sensitivity with tornado chart. Joint uncertainty via Monte Carlo. Robustness of the model's conclusions.
Technical report	15%	Three to five pages that synthesise the above for a technical reviewer.
Executive summary	10%	One page for the decision-maker. Answers the five questions: question, approach, finding, recommendation, caveats.
Presentation	5%	Five slides for a non-technical audience. One idea per slide. Accessible.

§ Milestones

Choose the model and state the purpose Include boundary, assumptions, and claim-support scope. Two hours.
Design the verification suite Five tests across analytical, limiting, conservation, symmetry, and convergence. Two hours.
Run the verification tests Record inputs, expected outputs, criteria, and results. Two hours.
Design and run validation Held-out data, disconfirming evidence in advance, metric, verdict. Two hours.
Sensitivity and uncertainty analysis One-at-a-time sensitivity, tornado chart, joint Monte Carlo. Three hours.
Write the technical report Three to five pages synthesising the above. Two hours.
Write the executive summary and prepare the slides One page and five slides. Two hours.

§ Rubric

Assessment criteria
Criterion	Excellent	Satisfactory	Needs work
Purpose statement	Complete, precise, with a defensible claim-support scope.	Complete but scope is vague.	Purpose missing or overstated.
Verification	Five or more diverse tests, all with criteria stated in advance.	Three to four tests; criteria mostly in advance.	Fewer than three tests or criteria set after.
Validation	Held-out data; disconfirming evidence in advance; scope stated.	Held-out data; disconfirming evidence stated after.	No held-out data or no criterion.
Sensitivity & uncertainty	Both analyses present; robustness clearly assessed.	Both present but robustness assessment thin.	One or both missing.
Technical report	Complete, self-contained, supports all claims.	Complete but disorganised.	Missing sections.
Executive summary	Answers all five questions; uncertainty communicated plainly.	Answers most questions; uncertainty present.	Missing key questions or suppresses uncertainty.

§ Common pitfalls

Watch for these

Verification tests that would pass for any model. A test is only useful if it can fail.
Validation on calibration data. The held-out discipline is essential.
Setting the acceptance criterion after seeing the result. The criterion must be stated in advance.
Confusing sensitivity with uncertainty. They answer different questions and both are required.
Overclaiming the model's scope. The claim-support statement should follow from the evidence.
Executive summary that hides uncertainty. Plain language is translation, not omission.

§ Submission

What to submit

A single ZIP archive named M7-capstone-yourid.zip.
The archive contains the purpose statement, the verification report, the validation report, the sensitivity and uncertainty report, the technical report, the executive summary, and the presentation slides.
Submit through the course LMS before the deadline listed on the course schedule.