1 Modelling context

A Monte Carlo estimate without a standard error is a number, not a result. The estimate varies across repetitions of the experiment, and the magnitude of that variation is what determines whether the estimate is useful. Reporting the estimate alone conceals the sampling variability and makes it impossible to compare results from different simulations or to know whether the sample size was adequate.

This lesson develops the standard error and confidence interval for Monte Carlo estimates, and the diagnostic methods for verifying that the number of samples was sufficient. Together with the estimator from Lesson 1 and the integration techniques from Lesson 2, these complete the toolbox for Monte Carlo estimation.

2 Learning objectives

3 Prerequisite review

Sample variance

s² = (1/(N−1)) Σ (Yᵢ − Ȳ)² where Yᵢ = g(Xᵢ) and Ȳ is the sample mean. An unbiased estimate of the variance of g(X).

Central limit theorem

For independent identically-distributed random variables with finite variance, the sample mean is approximately normally distributed for large N. This justifies the use of normal-based confidence intervals.

Standard normal quantile

The value z_α such that P(Z > z_α) = α for Z ~ Normal(0, 1). For a two-sided 95% interval, z_{0.025} = 1.96.

Running mean

The sequence of estimates θ̂_1, θ̂_2, θ̂_3, … computed as samples accumulate. Convergence is verified by observing that the running mean stabilises.

4 Mathematical formulation

Standard error

Let Yᵢ = g(Xᵢ) and let θ = E[Y]. The Monte Carlo estimator is θ̂_N = Ȳ, the sample mean of Y. Its variance is

Var( θ̂_N ) = Var( Y ) / N

Estimating Var(Y) by the sample variance s², the estimated standard error is

SE( θ̂_N ) = s / √N

The standard error is the estimated standard deviation of the estimator. It quantifies how much θ̂_N would vary across repetitions of the experiment.

Confidence interval

By the central limit theorem, for large N the estimator is approximately normal:

( θ̂_N − θ ) / SE( θ̂_N )  ≈  Normal(0, 1)

Rearranging gives an approximate two-sided confidence interval at level 1 − α:

θ̂_N ± z_{α/2} · SE( θ̂_N )

For a 95% interval, z_{0.025} = 1.96. For a 90% interval, z_{0.05} = 1.645. For a 99% interval, z_{0.005} = 2.576.

The interval is approximate because it relies on the central limit theorem. It is typically accurate for N ≳ 30 unless the distribution of Y is highly skewed or heavy-tailed.

Choosing a sample size

Suppose we want the standard error to be at most ε. From SE = σ / √N, the required sample size is

N ≥ ( σ / ε )²

where σ² = Var(Y). Since σ is usually unknown in advance, a pilot run of 100–1000 samples provides an estimate of σ, which is then used to plan the full run.

For a confidence-interval-width criterion, the width of the interval is approximately 2 z_{α/2} σ / √N. Setting this equal to a target width w gives

N ≥ ( 2 z_{α/2} σ / w )²

Convergence diagnostics

Three practical diagnostics confirm that the sample size is adequate:

• Running mean plot. Plot θ̂_n against n. The curve should stabilise as n grows. Persistent oscillation indicates insufficient samples.
• Standard-error scaling. The standard error should scale as 1/√N. A log-log plot of SE against N should be a straight line of slope −1/2.
• Multiple-seed replication. Run the simulation with several independent seeds. The estimates should scatter around the true value with no systematic drift.

5 Worked example

σ̂² ≈ 0.132 · 0.868 = 0.1146
σ̂ ≈ 0.338

N ≥ ( 2 · 1.96 · σ̂ / w )²
   ≥ ( 2 · 1.96 · 0.338 / 0.005 )²
   ≥ ( 265.0 )²
   ≈ 70,225

SE = √( 0.1352 · 0.8648 / 70,000 ) ≈ 0.0013

6 Verification check

7 Interpretation

The confidence interval is the appropriate way to report a Monte Carlo estimate. It gives the reader both the central estimate and the range of plausible values, and it communicates how precise the estimate is.

Three interpretive points are worth emphasising:

• The interval is not a statement about the model. It describes the sampling variability of the estimator, not the uncertainty in the model. Model error and sampling error are distinct.
• The interval narrows as the square root of the sample size. To halve the interval width, quadruple the sample size. To reduce it by a factor of 10, use 100 times as many samples. This is a substantial cost, which motivates the variance-reduction techniques of Unit 4.2.
• The interval assumes the estimator is unbiased. A biased estimator produces a confidence interval that does not contain the true value, no matter how narrow it is.

8 Uncertainty and limitations

The standard error and confidence interval are approximate. Several conditions must hold for them to be reliable.

Heavy-tailed distributions violate the finite-variance condition. For such distributions the standard error can seriously understate the uncertainty, and the confidence intervals can undercover. Alternative methods — bootstrap confidence intervals, or estimators designed for heavy-tailed data — are required.

9 Common implementation errors

10 Independent modelling task

11 Reflection prompt

12 Lesson summary

§ Self-quiz, practice, and homework