Lesson 1: Scenario Design and Experimental Planning

1 Modelling context

A simulation model is a piece of machinery that produces outputs from inputs. To use it for a decision, the analyst must choose which inputs to test. This choice — the experimental design — determines what the study can conclude. A well-designed experiment isolates the effects of interest, controls the sources of noise, and produces results that answer the modelling question. A poorly designed experiment produces a scatter of runs that cannot support any conclusion.

The experiment design questions are the same as in any empirical study, but with a distinctive twist: in a simulation, the analyst controls the random seed, so some sources of noise that would be uncontrolled in a physical experiment can be controlled here. The techniques developed for simulation are correspondingly sharper: common random numbers, variance reduction, and factorial designs.

The design determines the conclusion

A study can only answer the questions its design is capable of answering. If the design does not vary an input, the study cannot say what effect that input has. If the design confounds two inputs, the study cannot separate their effects.

2 Learning objectives

By the end of this lesson you will be able to

Translate a modelling question into a scenario design.
Distinguish factors from levels, and treatments from replications.
Choose a factorial design appropriate to the question.
Use common random numbers to isolate the effect of interest.
Plan replication to achieve a target precision.
State in advance what each comparison will mean.

3 Prerequisite review

Factor: An input variable whose effect on the output the experiment is designed to measure. In a queueing study, the factors might be arrival rate, service rate, and number of servers.
Level: A specific value of a factor. A factor with three levels might be tested at low, medium, and high values.
Treatment: A combination of levels of the factors. A full factorial design with two factors each at three levels has nine treatments.
Replication: Independent repetition of a treatment. In stochastic simulation, replication uses a different random seed each time.
Design: The set of treatments and replications that constitutes the experiment.
Confounding: The situation in which the effect of one factor cannot be distinguished from the effect of another because they were varied together. A well-designed experiment avoids confounding.

4 Mathematical formulation

From question to design

The design follows from the modelling question. Three questions structure the design:

What is the primary comparison? Which difference between scenarios does the study exist to measure? This is the question the design must answer.
Which factors must vary to make this comparison? The primary comparison usually involves one factor (the treatment or policy). Other factors may be needed to explore robustness.
Which factors are controls? Factors that must be held fixed to make the comparison clean.

Common designs

One-factor-at-a-time (OFAT). Vary one factor at a time, holding others fixed.

Simple to run and interpret.
Cannot detect interactions between factors.
Inefficient when many factors are involved.

Full factorial. Vary all factors simultaneously, with every combination of levels tested.

Detects main effects and interactions.
Number of treatments grows multiplicatively: L^k for k factors each at L levels.
Feasible for small k and L.

Fractional factorial. Test a subset of the full factorial combinations.

Reduces the number of treatments.
Confounds some effects; appropriate when higher-order interactions are expected to be small.

Common random numbers

In a comparison between two configurations of a stochastic system, using the same random seed for both configurations isolates the effect of the configuration from the effect of random variation. Module 4 introduced the technique; here it appears as an experimental design choice.

Formally, if Y_A(r) and Y_B(r) are the outputs of configurations A and B under the same random stream r, the difference

D(r) = Y_A(r) − Y_B(r)

has variance

Var(D) = Var(Y_A) + Var(Y_B) − 2 Cov(Y_A, Y_B)

Using common random numbers makes the covariance positive and reduces the variance of the difference. The design decision is whether to use independent seeds (simple but noisy) or common random numbers (efficient but requires matched configurations).

Replication and precision

For a stochastic simulation, each treatment should be replicated with independent seeds. The number of replications R determines the standard error of the treatment mean:

SE( mean ) = s / √R

where s is the sample standard deviation of the output across replications. To achieve a target standard error ε, use a pilot run to estimate s and set

R ≥ ( s / ε )²

Figure 7.6. OFAT and factorial designs. OFAT varies one factor at a time; factorial varies all combinations. Factorial designs detect interactions that OFAT misses.

Stating the analysis in advance

Before running the experiment, state how the results will be analysed. For each comparison, specify:

The primary metric (the output quantity to be compared).
The comparison of interest (e.g. "configuration A versus configuration B").
The statistical test or summary statistic (e.g. "mean difference with 95% confidence interval").
The criterion for a meaningful difference (e.g. "a 10% reduction in mean waiting time").

Stating these in advance prevents the common failure mode of choosing the analysis to fit the result.

5 Worked example

Designing a study of a hospital emergency department

An emergency department wants to compare two triage policies. The primary decision is whether to adopt the new policy, and the primary concern is patient waiting time.

Step 1 — State the primary comparison.

The comparison is between the current triage policy (A) and the proposed policy (B). The primary metric is the mean time from arrival to first treatment.

Step 2 — Identify the factors.

Factor 1 — Triage policy. Two levels: current (A), proposed (B).
Factor 2 — Arrival rate. Three levels: low (weekday), medium (weekend day), high (weekend evening).
Factor 3 — Number of physicians. Two levels: 4, 6.

The primary comparison concerns Factor 1. Factors 2 and 3 characterise the operating conditions under which the comparison is made.

Step 3 — Choose the design.

A full factorial design with 2 × 3 × 2 = 12 treatments. Each treatment will be replicated 30 times, giving 360 simulation runs.

Step 4 — Use common random numbers.

For each combination of arrival rate, physician count, and replication index, run both triage policies with the same random seed. This isolates the effect of the triage policy from the random variation in arrivals and service times.

Step 5 — Plan the replication.

A pilot run of 10 replications gives a sample standard deviation of s = 4.5 minutes. For a target standard error of ε = 0.8 minutes:

R ≥ ( 4.5 / 0.8 )² ≈ 32

Choose R = 30 for a small safety margin. If the achieved precision is not adequate, additional replications can be added.

Step 6 — State the analysis in advance.

Primary metric. Mean time from arrival to first treatment.
Primary comparison. Policy B versus Policy A, across all combinations of arrival rate and physician count.
Summary statistic. Mean difference with 95% confidence interval.
Criterion for adoption. Policy B reduces mean waiting time by at least 10% under at least two of the three arrival-rate conditions, without increasing it under the third.

Step 7 — Document the design.

The design is documented with: the factors, the levels, the treatments, the replication count, the random seed policy, the primary metric, and the analysis plan. The documentation is the basis for the analysis and for the reproducibility of the study.

6 Verification check

Checks to perform on your design

Does the design answer the question? Confirm that the primary comparison is enabled by the design.
Are the factors and levels appropriate? Each level should be plausible and physically meaningful.
Is the design balanced? Balanced designs (same number of replications for each treatment) simplify analysis.
Is confounding avoided? No two factors should be varied together in a way that prevents separating their effects.
Is the replication adequate? A pilot run determines the required number of replications for the target precision.
Is the analysis stated in advance? The primary metric, the comparison, and the criterion for significance should be documented before the experiment is run.

7 Interpretation

The design of an experiment is a modelling decision. It determines what the study can conclude, and it constrains how the results can be interpreted.

Three interpretive points:

The design defines the comparison. A study can only compare the treatments it included. Factors that were held fixed cannot be evaluated.
Interactions are effects too. The effect of one factor may depend on the level of another. A factorial design detects these interactions; OFAT does not.
Common random numbers are a design choice. They reduce variance in comparisons, but they require matching runs between configurations. The choice depends on the question being asked.

Report the design

A complete report of a simulation study includes the design: the factors, the levels, the treatments, the number of replications, the random seed policy, and the analysis plan. Without the design, the results cannot be interpreted.

8 Uncertainty and limitations

Experimental design in simulation inherits the limitations of experimental design in general.

Limitations of simulation experiments
Limitation	Effect
Finite resources	Not all combinations of factors can be tested
Model scope	The study is limited to the factors included in the model
Replication costs	High replication reduces variance but increases runtime
Model-form error	No design can detect a wrong model form
Convergence to steady state	Transient effects may dominate if warm-up is inadequate

The design cannot fix a wrong model

A carefully designed experiment on a wrong model produces a carefully documented wrong answer. Design improves the precision of the answer, not its validity.

9 Common implementation errors

Watch for these

Running one-off trials. A single run of each configuration is not an experiment. Without replication, there is no way to distinguish systematic effects from random variation.
Confounding factors. If two factors are always varied together, their effects cannot be separated.
Using independent seeds for comparison. Common random numbers reduce the variance of the difference substantially. Independent seeds waste the opportunity.
Choosing the analysis after seeing the results. The primary metric and comparison should be stated in advance.
Insufficient replication. A study with 5 replications per treatment may not have the precision to detect the effect of interest.
Not documenting the design. The design is part of the method. Without it, the results cannot be interpreted or reproduced.
Ignoring warm-up in stochastic simulations. Transient effects can bias the treatment means if the warm-up period is not discarded (see Module 5).

10 Independent modelling task

Your turn

Choose a modelling question that involves a comparison. It can be from the capstone or from a smaller study.
State the primary comparison: what difference does the study exist to measure?
Identify the factors and their levels. Justify each level.
Choose a design. If the number of treatments is manageable, use a full factorial. Otherwise, use a fractional factorial and document the confounding.
Decide on the seed policy. Will you use common random numbers, independent seeds, or a mixture?
Plan the replication. Conduct a pilot run to estimate the required number of replications for a target precision.
State the analysis in advance: the primary metric, the comparison, the summary statistic, and the criterion for a meaningful difference.
Write a one-page experimental design document.

This exercise forms part of the capstone's experimental design component.

11 Reflection prompt

Reflect before continuing

A study uses a one-factor-at-a-time design to compare five factors. What can it conclude, and what can it not conclude?

Two or three sentences. Consider the effects the design can detect and the effects it cannot.

12 Lesson summary

Key takeaways

The design follows from the question. The primary comparison determines the factors, levels, and replications.
Factorial designs detect interactions; OFAT does not. Choose the design that matches the question.
Common random numbers reduce variance in comparisons. Use the same seed for matched runs.
Replication determines precision. Conduct a pilot run to estimate the required number of replications.
State the analysis in advance. The primary metric, comparison, and criterion should be documented before the experiment is run.

Coming next

Lesson 2 addresses the communication of results to technical and non-technical audiences.

§ Self-quiz, practice, and homework

Practice exercises

For a study comparing two queueing disciplines, identify the factors and levels. Choose a design and justify it.
For a full factorial design with 4 factors at 3 levels, how many treatments? If each treatment is replicated 20 times, how many total runs?
For a comparison between two configurations, describe how common random numbers would be implemented.
For a pilot run with s = 12, compute the required replications for a target standard error of 1.5.

Homework problems

Design document. For a modelling question of your choice, produce a complete experimental design document: the primary comparison, the factors and levels, the design, the seed policy, the replication count, and the analysis plan. Justify each choice. 500–700 words.
Comparison of designs. For the same question, sketch both a one-factor-at-a-time design and a factorial design. Compare the number of runs, the effects detectable, and the interpretive strength. Recommend one and justify.
Pilot study. Conduct a pilot run of 10 replications of a simulation. Compute the sample standard deviation of the primary metric and derive the required replications for two target precision levels.

Mini project

Produce a complete experimental design document for the capstone project. Include the primary comparison, factors, levels, design, seed policy, replication plan, and analysis plan. Submit the document and a pilot-study table. This is the basis for the experimental design component of the capstone.