Lesson 2: Parameter Estimation and Calibration

1 Modelling context

Even a well-architected model contains numbers that are not given to you. A decay rate, a transmission probability, a service-time mean, a conversion rate — these must be estimated from data, taken from literature, or assumed from domain knowledge. Each estimation is a modelling decision with consequences for the model's output.

Estimation is not a preprocessing step. It is part of the modelling process, and it carries its own uncertainty. A parameter estimated from a small sample will have a wide confidence interval; a parameter estimated from a proxy measurement will inherit the proxy's biases. Both facts propagate into the model's output, and both must be reported.

This lesson covers the estimation and calibration of model parameters. Estimation refers to deriving parameter values from data. Calibration refers to adjusting several parameters together so that the model reproduces observed behaviour. They are related but distinct, and the distinction matters when reporting results.

The two questions of estimation

What value should the parameter take, and how much confidence should that value be given? The first question produces a point estimate; the second produces an uncertainty. Both are part of the result, and a model that reports only the first is incomplete.

2 Learning objectives

By the end of this lesson you will be able to

State what a parameter means in the model before estimating it.
Choose an estimation method appropriate to the data and the model form.
Report the uncertainty in a parameter estimate, not only its point value.
Distinguish overfitting from adequate fit, and recognise the warning signs of each.
Explain why calibration is not validation.
Test the sensitivity of model output to the estimated parameters.

3 Prerequisite review

Estimator and estimate: An estimator is a rule for deriving a parameter value from data — for example, the sample mean. An estimate is the value the rule produces for a particular dataset. The estimator is a random variable; the estimate is one realisation of it.
Bias and variance: The bias of an estimator is the difference between its expected value and the true parameter. Its variance is a measure of how much the estimate varies across datasets. An estimator is often judged by the tradeoff between the two.
Least squares: An estimation method that chooses parameter values to minimise the sum of squared differences between observations and predictions. It is simple, widely used, and equivalent to maximum likelihood when the residuals are independent and normally distributed.
Residual: The difference between an observed value and the model's prediction at the corresponding input. Residuals contain the information the model failed to capture, and their structure is diagnostic of model error.
Confidence interval: A range of parameter values consistent with the data at a stated level of confidence. A 95 percent confidence interval is one that, if the experiment were repeated many times, would contain the true value in 95 percent of repetitions.
Overfitting: Fitting the noise in a dataset rather than the signal. A model that overfits reproduces its calibration data well and generalises poorly to new data.

4 Mathematical formulation

The estimation problem

Suppose we have observations (x₁, y₁), …, (xₙ, yₙ) and a model y = f(x; θ) with parameter vector θ. The estimation problem is to choose θ so that the model reproduces the data in a stated sense.

Least squares. Choose θ to minimise

S(θ) = Σᵢ ( yᵢ − f(xᵢ; θ) )²

This is a well-defined optimisation problem with a clear objective. It is the default method when the residuals are thought to be independent and roughly symmetric.

Maximum likelihood. Suppose the observations are generated as yᵢ = f(xᵢ; θ) + εᵢ, where εᵢ follows a distribution with density p. Choose θ to maximise the likelihood

L(θ) = Πᵢ p( yᵢ − f(xᵢ; θ) )

Under independent, normally-distributed residuals with constant variance, maximum likelihood and least squares produce the same estimate. When the residuals follow a different distribution — exponential, Poisson, heavy-tailed — maximum likelihood is generally preferred because it respects the actual error structure.

Uncertainty in the estimate

A point estimate alone is not a result. The uncertainty in an estimate is typically reported as a standard error or a confidence interval. For a one-parameter least-squares fit, the standard error has the form

SE(θ̂) = σ̂ / √( Σᵢ (∂f/∂θ)² )

where σ̂ is the estimated residual standard deviation and the denominator measures how strongly the model output responds to the parameter. Two consequences are worth noting:

Smaller residuals reduce the standard error. Better data — or a better-fitting model — narrows the interval.
A parameter the output is insensitive to is hard to estimate. If ∂f/∂θ is small over the observed range, the parameter is poorly identified, and its estimate will be uncertain regardless of how much data you have.

Calibration

Calibration is the same problem applied to several parameters at once, often with a model whose output is a trajectory rather than a single value. The objective may be the sum of squared residuals across time, or a weighted combination of several observable quantities. The estimated parameters are the values that make the model best reproduce the observed behaviour according to the chosen objective.

Calibration is not validation

A calibrated model reproduces the data it was calibrated against. That is not evidence that it will reproduce data it has not seen. Validation, covered in Lesson 3, is the separate activity of testing the model against evidence that was not used in calibration.

Overfitting, formally

Adding parameters to a model always reduces the residual sum of squares on the calibration data. This is true even when the added parameters carry no information about the underlying system. The apparent improvement is a consequence of having more degrees of freedom, not of having captured more structure.

The standard correction is to penalise model complexity. The Akaike Information Criterion (AIC) is one such penalty:

AIC = 2k − 2 ln(L̂)

where k is the number of parameters and L̂ is the maximised likelihood. Lower AIC is better. The penalty term 2k prevents the criterion from rewarding complexity for its own sake.

Figure 1.7. Overfitting versus adequate fit. The overfit curve passes through every observation and oscillates between them. Its residuals on the calibration data are smaller, but it would generalise worse to new observations.

5 Worked example

Estimating a decay rate with uncertainty

A quantity y(t) is believed to follow y(t) = y₀ · e^−kt. Observations are available at t = 0, 1, 2, 3, 4, 5. We will estimate k and report its uncertainty.

Step 1 — State what the parameter means.

k is the decay rate, with units of inverse time. It is the parameter of interest. y₀ is a nuisance parameter: it is necessary for the fit but is not the quantity the model is being built to answer questions about.

Step 2 — Linearize and fit.

Take natural logarithms of both sides:

ln y = ln y₀ − k t

This is linear in t. A least-squares line through the points (t, ln y) gives a slope of −k. Let the estimated slope be −0.42, with units of inverse time. Then k̂ = 0.42.

Step 3 — Report uncertainty.

The standard error of the slope from the regression is, say, 0.03. The estimate is reported as

k̂ = 0.42 ± 0.03 per unit time (1 SE)

A 95 percent confidence interval, using the appropriate critical value for the degrees of freedom, would be slightly wider, roughly [0.35, 0.49].

Step 4 — Check residuals.

Plot the residuals of the log-linear fit against t. If the residuals show a trend, a curve, or increasing spread, the exponential model form is suspect. Structure in the residuals is evidence against the model, not a signal to refine the estimate.

Step 5 — Test sensitivity to the estimate.

Re-run the model at k = 0.39 and k = 0.45 (one standard error below and above the estimate). Compare the resulting outputs. If the conclusions drawn from the model change materially over this range, the uncertainty in k matters, and the model's answer is a range rather than a number.

Step 6 — Consider a nonlinear fit.

If a direct nonlinear least-squares fit on the original scale gives a materially different estimate of k, investigate the discrepancy. The two approaches weight the data differently — the log-linear fit weights small values heavily, the nonlinear fit weights large values heavily — and neither is automatically correct.

The deliverable is a point estimate, an uncertainty, a diagnosis of the residuals, and a statement of how much the model's conclusions depend on the estimate.

6 Verification check

Verification of an estimation procedure asks whether the implementation of the fitting method is correct. It is separate from the question of whether the fitted parameter is meaningful.

Checks to perform

Synthetic-data test. Generate data from the model with a known parameter value, add noise, and fit. The estimate should recover the true value within the expected sampling error. Repeat several times and confirm the estimates are centred on the true value.
Compare against a closed-form solution. For a linear regression, the least-squares slope has a known closed form. Compare the spreadsheet result against the closed form.
Test the objective function directly. Evaluate the residual sum of squares at the reported estimate, and at a few nearby values. The reported estimate should minimise it.
Check units consistency. If k has units of inverse minutes, the time inputs must be in minutes. A units mismatch is the most common cause of an apparently wrong estimate.
Verify the uncertainty calculation. The standard error should shrink as the residual variance shrinks and as the data range widens. Test with synthetic datasets of different sizes and confirm this behaviour.
Reconcile against a second method. If a second estimation method is available — for example, a nonlinear fit alongside the log-linear one — compare the estimates. A material disagreement indicates an assumption has been violated, and should be investigated rather than resolved by choosing the more convenient answer.

7 Interpretation

A parameter estimate is not a fact about the world. It is a statement about the data, conditional on the model. Three interpretive consequences follow, and each is a reason estimation must be reported carefully.

The estimate depends on the model form. A parameter named k in an exponential model and a parameter named k in a power-law model are not the same quantity, even if they are given the same symbol. The name of a parameter does not travel between model forms.
The estimate depends on the data window. A rate estimated from a short observation window may differ substantially from one estimated from a longer window, especially if the underlying process changes. Reporting the window is part of reporting the estimate.
The estimate depends on the estimation method. Least squares and maximum likelihood with a heavy-tailed error distribution can give materially different estimates from the same data. The method is part of the result.

None of these observations makes estimation impossible. They make it necessary to report estimation the way we report other modelling choices: with the assumptions, the data, the method, and the uncertainty stated explicitly.

Reporting template for a parameter estimate

A clean way to report an estimate is: "Under model form M, parameter k was estimated from data D using method E, giving k̂ = value ± uncertainty (units). The estimate is sensitive to [list factors]; the model's output changes by [amount] over the plausible range of k." Every element of this sentence is part of the result.

8 Uncertainty and limitations

Parameter uncertainty is not the only source of uncertainty in a model, and it is often not the largest. It is useful to distinguish the sources so that effort is directed at the ones that matter.

Sources of uncertainty in parameter estimation
Source	Cause	Reduced by
Sampling error	Finite number of observations	More data, better experimental design
Measurement error	Imprecision in the observations of `x` or `y`	Better instruments; error-in-variables models
Model-form error	The functional form does not match the true relationship	Better theory; comparison of multiple forms; residual analysis
Identification error	The data do not constrain the parameter well	Wider or better-placed data; reparameterization
Extrapolation risk	The estimate is applied outside the range of the data	No statistical remedy; must be handled by limitation statements

Of these, extrapolation risk is the most dangerous and the least well-handled by standard techniques. A parameter estimated over a narrow range of x may take on very different values outside that range, and the uncertainty reported for the estimate does not reflect this, because the model form has not been tested over the extrapolation region.

Precision in estimation is not accuracy

A narrow confidence interval on a parameter estimated under the wrong model form is a precise statement about the wrong thing. Better data narrows the interval; only better modelling improves the estimate.

9 Common implementation errors

Watch for these

Reporting a point estimate with no uncertainty. Without a spread, there is no way for a reader to judge whether a conclusion is robust. A parameter estimate without an uncertainty is a number, not a result.
Fitting a parameter that is not identifiable from the data. If the output is nearly insensitive to the parameter over the observed range, the estimate will be dominated by noise. Check the sensitivity before estimating.
Assuming linearization preserves the estimate. Fitting ln y = ln y₀ − k t and fitting y = y₀ e^−kt weight the data differently. Neither is automatically correct; the choice depends on the error structure.
Calibrating and validating on the same data. A model calibrated on data it is also validated against will appear better than it is. Validation must use data that did not enter calibration.
Adjusting parameters until the fit improves, and calling it estimation. This is manual overfitting. It produces a model that memorises the data without capturing its structure.
Ignoring the data window. The estimate is conditional on the window. Stating the estimate without the window makes it look more general than it is.
Treating a proxy parameter as if it were the parameter of interest. If k is estimated from a proxy measurement, it is the proxy's parameter, not the physical parameter. The distinction must be preserved.

10 Independent modelling task

Your turn

Take the model you built in Lesson 1 (or any small model you understand well).
Identify at least one parameter that would have to be estimated from data. State, in one sentence, what the parameter means and what its units are.
Obtain or generate a small dataset. If you generate it, state the noise model explicitly.
Estimate the parameter using at least two methods (e.g. least squares and a linearized fit; or maximum likelihood with two different assumed error distributions). Compare the estimates.
Report each estimate as a value ± uncertainty, in the units stated in step 2. Explain any discrepancy between the two methods rather than choosing the more convenient result.
Re-run the model at the lower and upper bounds of the uncertainty range. Record how much the output changes.
Write one sentence stating the data window and the model form under which the estimate is valid.

This estimation exercise is the basis for Laboratory 2 and will feed the parameterization component of Assignment 1.

11 Reflection prompt

Reflect before continuing

Think of a number you have seen presented as a fact — a conversion rate, an interest rate, a risk coefficient, a growth rate. Was it reported with an uncertainty? Was the data window stated? Was the method described?

Write two or three sentences. The point is to notice how often estimates are presented as facts, and what is lost when they are.

12 Lesson summary

Key takeaways

State what a parameter means before estimating it. The name of a parameter does not travel between model forms.
Choose an estimation method that matches the error structure. Least squares and maximum likelihood coincide under normal, homoscedastic errors and diverge otherwise.
Report uncertainty, not just a point estimate. A parameter estimate without a spread is a number, not a result.
Check residuals. Structure in the residuals is evidence against the model form, not a signal to refine the estimate.
Calibration is not validation. A model that reproduces the data it was calibrated against has not thereby been shown to reproduce data it has not seen.
Test sensitivity to the estimate. If the model's conclusions change over the plausible range of a parameter, the model's answer is a range, not a number.

Coming next

Lesson 3 completes Unit 1.2 by separating verification, validation, and reproducibility, and by making explicit what evidence must be produced for each.

§ Self-quiz, practice, and homework

Practice exercises

Fit the model y = a x + b to the points (0, 1), (1, 3), (2, 4), (3, 6) using least squares. Report the estimates of a and b, their standard errors, and interpret the residuals.
For the fit in exercise 1, compute the predicted value and residual at each point. Plot the residuals against x. Is there any structure?
Repeat exercise 1 but fit y = c x² + d x + e. Compare the residual sum of squares to the linear fit. Is the extra parameter justified?
Choose a dataset of your own (at least 10 points). Estimate a parameter, report its uncertainty, and check the residuals.

Homework problems

Estimation with uncertainty. Choose a real dataset (your own or from a public source, such as a government open-data portal). Fit a model with at least one parameter. Report the estimate, the standard error or confidence interval, and the residuals. Perform at least one robustness check (e.g. refit on a subsample). Write a 500-word report.
Method comparison. Estimate a parameter using least squares and at least one other method (maximum likelihood with a specified error distribution; a different loss function; or a robust regression). Compare the estimates and the standard errors. Explain any discrepancy.
Overfitting demonstration. Using a small dataset (10–20 points), fit polynomials of increasing degree (1, 2, 3, 5, 10). Tabulate the residual sum of squares and the generalisation error (using a held-out set or cross-validation). Show where the model begins to overfit and explain how you detected it.

Mini project

Choose a growth or decay process — population growth, radioactive decay, adoption of a technology, spread of a rumour, or similar. Find or simulate data. Estimate the parameters using at least two methods, report the uncertainties, and compare the estimates. Then re-run the model at the lower and upper bounds of the uncertainty range and record how much the output changes. Submit the spreadsheet, a one-page technical summary, and a one-page plain-language summary.