Lesson 1: Random Experiments and Estimators

1 Modelling context

Monte Carlo methods are often presented as a computational trick: "just sample a lot and average." The trick works, but only when the random experiment and the estimator have been defined correctly. A Monte Carlo simulation that estimates the wrong expectation, or averages the wrong function of the outcome, produces a confident answer to the wrong question.

This lesson establishes the two modelling decisions that precede any Monte Carlo computation: what random experiment to run, and what function of the outcome to average. Once these are fixed, the implementation is mechanical. Getting them wrong is not a numerical error but a modelling error, and no amount of replication can repair it.

Estimator before implementation

The choice of estimator is a modelling decision, not a technical one. It determines the quantity that the simulation estimates, and therefore the meaning of the result. Write the estimator down before writing any code.

2 Learning objectives

By the end of this lesson you will be able to

Define a random experiment precisely, including the sample space and the probability measure.
Express a quantity of interest as an expectation E[g(X)].
Construct an estimator for the expectation and justify its form.
Explain the unbiasedness and consistency properties of the Monte Carlo estimator.
Distinguish the quantity of interest from the estimator that approximates it.

3 Prerequisite review

Random variable: A function that assigns a numerical value to each outcome of a random experiment. Denoted with capital letters, e.g. X.
Probability distribution: The rule that assigns probabilities to the possible values of a random variable. Discrete distributions are specified by a probability mass function; continuous distributions by a density.
Expectation: The long-run average of a random variable over many independent repetitions of the underlying experiment. Denoted E[X].
Independence: Two random variables are independent when the outcome of one provides no information about the other. The samples used in a Monte Carlo simulation must be independent for the standard error formulas to hold.
Law of large numbers: Under independence and finite expectation, the sample mean converges to the true expectation as the number of samples grows.

4 Mathematical formulation

Random experiment

A random experiment is specified by three ingredients:

Sample space Ω. The set of all possible outcomes.
Random variable X. A function from Ω to the real numbers (or to ℝᵈ).
Distribution P. The probability measure that assigns probabilities to subsets of Ω, or equivalently the distribution of X.

For Monte Carlo purposes, the experiment is fully specified once we have a procedure that generates an independent sample from the distribution of X.

Quantity of interest

The quantity of interest is expressed as an expectation:

θ = E[ g(X) ]

where g is a function chosen to represent the quantity we want. Different quantities require different g:

Quantities of interest as expectations
Quantity	Choice of `g`
Mean of `X`	`g(x) = x`
Probability `P(X ∈ A)`	`g(x) = 1 if x ∈ A, 0 otherwise`
Second moment `E[X²]`	`g(x) = x²`
Tail probability `P(X > a)`	`g(x) = 1 if x > a, 0 otherwise`
Expected shortfall `E[X \| X > a]`	Requires a modified estimator (conditional mean)

Estimator

The Monte Carlo estimator of θ = E[g(X)] is the sample mean:

θ̂_N = (1/N) Σ_{i=1}^N g(Xᵢ)

where X₁, X₂, …, X_N are independent samples from the distribution of X.

Properties of the estimator

Unbiasedness. By linearity of expectation:

E[ θ̂_N ] = (1/N) Σ E[ g(Xᵢ) ] = E[ g(X) ] = θ

So the estimator is centered on the true value, on average across repetitions.

Variance. Because the samples are independent:

Var( θ̂_N ) = Var( g(X) ) / N

So the standard deviation of the estimator decreases as 1/√N.

Consistency. By the law of large numbers:

θ̂_N → θ  as  N → ∞

The estimator converges to the true value with probability one.

Figure 4.2. The Monte Carlo workflow. A random experiment generates independent samples; the estimator averages g over those samples; the estimator targets the expectation θ = E[g(X)].

5 Worked example

Estimating the mean and variance of a normal distribution

Suppose X ~ Normal(μ, σ²) and we want to estimate μ, σ², and P(X > 1), given μ = 0.5 and σ = 2.

Step 1 — Express each quantity as an expectation.

μ = E[X], so g₁(x) = x.
σ² = E[(X − μ)²] = E[X²] − μ², so g₂(x) = (x − μ)².
P(X > 1) = E[ 1{X > 1} ], so g₃(x) = 1 if x > 1, else 0.

Step 2 — Choose a random experiment.

Sample Xᵢ ~ Normal(0.5, 4) independently, using a pseudorandom number generator with a recorded seed.

Step 3 — Compute the estimators.

With N = 10,000 samples:

μ̂ = (1/N) Σ Xᵢ

σ̂² = (1/N) Σ (Xᵢ − μ̂)²

p̂ = (1/N) Σ 1{Xᵢ > 1}

Step 4 — Compare with the truth.

The true values are μ = 0.5, σ² = 4, and P(X > 1) ≈ 0.4013. A typical run gives:

μ̂ = 0.502
σ̂² = 4.01
p̂ = 0.401

The estimates are close to the truth, with discrepancies consistent with the sampling error of N = 10,000 samples.

Step 5 — Note the estimator forms.

Each quantity of interest required a different function g. The estimator is always the same — the sample mean of g(X) — but the choice of g determines what the estimator estimates. Choosing the wrong g produces a confident estimate of the wrong quantity.

6 Verification check

Checks to perform

Check the unbiasedness numerically. Run the same experiment with different seeds and confirm the estimator values scatter around the true value rather than drifting systematically.
Verify against a known case. When the quantity of interest has a known analytical value, compare the estimator against it.
Test with a degenerate case. If X is constant, the estimator should return that constant exactly.
Check the estimator form. Confirm that the function g actually corresponds to the quantity of interest. A common error is to confuse E[X²] with (E[X])².
Verify independence of samples. The standard error formula assumes independent samples. Confirm that the generator does not produce correlated sequences.

7 Interpretation

The Monte Carlo estimator is a random variable. Its value depends on the particular samples that were drawn. Across repetitions of the simulation, the estimate varies according to a distribution centered on the true value θ and with standard deviation Var(g(X))/√N.

This has three interpretive consequences:

The estimate alone is not a result. Reporting a Monte Carlo estimate without its standard error conceals the sampling variability.
Different seeds produce different answers. Two equally valid simulations of the same model will give different estimates. The difference is not a bug but the sampling error made visible.
The estimate is not the true value. It is a realisation of a random variable whose mean is the true value. The uncertainty must be reported.

Reporting template

A clean way to report a Monte Carlo estimate is: "Under the model [description], the estimated value of [quantity] is [estimate] with a standard error of [SE], based on N = [N] replications. The 95% confidence interval is [lower, upper]. The random seed was [seed]." Every element of this sentence is part of the result.

8 Uncertainty and limitations

The Monte Carlo estimator carries two distinct uncertainties that must be distinguished.

Uncertainties in the Monte Carlo estimator
Source	Cause	Reduced by
Sampling error	Finite number of samples	More samples, variance reduction
Estimator bias	A choice of `g` that does not target the quantity of interest	Correct estimator design
Distribution error	The samples do not follow the intended distribution	Correct generator, validated input distribution

Precision is not accuracy

A small standard error means the estimator is precise: it would give nearly the same answer if the experiment were repeated. It does not mean the estimator is accurate: the answer may be precise and wrong, if the estimator, the input distribution, or the model is wrong.

9 Common implementation errors

Watch for these

Averaging the wrong function. Estimating E[X] when the quantity of interest is E[X²] is a modelling error, not a numerical one. Write down the estimator before implementing it.
Confusing the estimator with the quantity. The estimator θ̂_N is a random variable; the quantity θ is a fixed number. In derivations and reports, keep the notation distinct.
Using correlated samples. The standard error formula Var(g(X))/√N assumes independent samples. Reusing samples, or using a generator with visible correlation, invalidates the formula.
Reporting the estimate without a standard error. Without the SE, a reader cannot judge whether the estimate is reliable or how it compares to another estimate.
Using a sample-size convention that is not documented. N must be stated explicitly, along with whether it refers to the number of samples, the number of replications, or something else.
Forgetting to record the seed. Without the seed, the result is not reproducible.

10 Independent modelling task

Your turn

Choose a random variable X from a distribution you know — normal, exponential, uniform, Poisson, or similar.
Define three quantities of interest: a mean, a probability, and a second moment or tail probability.
For each, write down the function g such that the quantity equals E[g(X)].
Implement a Monte Carlo estimator for each quantity, with N = 1000 samples. Record the seed.
Compare each estimate against the analytical value (where available) and compute the standard error.
Repeat with N = 10,000. Confirm that the standard error decreases by roughly a factor of √10 ≈ 3.16.

This exercise forms part of Laboratory 7 and the estimator-design component of Assignment 4.

11 Reflection prompt

Reflect before continuing

Suppose two analysts estimate the same quantity by Monte Carlo and obtain different values. Is one of them wrong?

Two or three sentences. Consider the sampling variability of the estimator and the difference between a random draw and a modelling error.

12 Lesson summary

Key takeaways

A Monte Carlo simulation is defined by a random experiment and an estimator. Both are modelling decisions made before any computation.
Quantities of interest are expressed as expectations θ = E[g(X)]. The choice of g determines what is being estimated.
The Monte Carlo estimator is the sample mean of g(X). It is unbiased and consistent.
The standard error of the estimator scales as 1/√N. Reducing it by a factor of k requires k² times as many samples.
The estimate alone is not the result. It must be reported with the sample size, the seed, and the standard error.

Coming next

Lesson 2 develops the Monte Carlo approximation of integrals and probabilities, including the direct application of the estimator to definite integrals and geometric regions.

§ Self-quiz, practice, and homework

Practice exercises

Write the estimator for each of: (a) E[X] where X ~ Exp(λ); (b) P(X < 2) where X ~ Normal(0, 1); (c) Var(X) where X ~ Uniform(0, 1).
For each quantity in exercise 1, write the corresponding function g explicitly.
Suppose the standard error of a Monte Carlo estimate is 0.05 at N = 1000. Estimate the standard error at N = 4000.
Explain in two sentences why a Monte Carlo estimate is not a deterministic answer to the modelling question.

Homework problems

Estimator design. Choose a random variable X from a distribution you know. Define four quantities of interest: a mean, a second moment, a probability, and a tail conditional mean. For each, write the estimator in full. Implement all four in a spreadsheet with N = 5000 samples and record the seed. Compare against analytical values where they exist.
Standard error scaling. For one of the quantities in problem 1, compute the standard error at N = 100, 400, 1600, 6400. Tabulate the standard errors and confirm the 1/√N scaling numerically.
Estimator bias. Choose an estimator for Var(X) — either the biased (1/N)Σ(Xᵢ − X̄)² or the unbiased (1/(N−1))Σ(Xᵢ − X̄)². Run 1000 independent replications with N = 10 samples each. Compare the mean of the estimates against the true variance. Which estimator is unbiased? Demonstrate numerically.

Mini project

Choose a real quantity that can be expressed as an expectation — a risk measure, a portfolio value, a structural reliability, or similar. Design a Monte Carlo estimator, implement it, and report the estimate with its standard error and a 95% confidence interval. Verify against an analytical benchmark where one exists, or against an independent method. Submit the spreadsheet and a two-page report. This is the basis for Laboratory 7 and Assignment 4.