What is it?
What question does it help answer?
How sensitive is a result to the particular order or sample path that happened to be observed?
What can it tell you?
- Whether a reported result is unusually dependent on one path ordering under the chosen simulation rule.
- How a distribution of synthetic outcomes differs from a single headline path.
- Which percentile or tail questions deserve closer investigation.
What can it not tell you?
- It cannot prove future profitability.
- A percentile is only meaningful relative to the simulation assumptions that generated it.
- Naive resampling can destroy serial dependence, volatility clustering, regime structure, or other features that matter.
Same trades, different drawdown.
Synthetic example: start from $1,000 and use four fixed-size net trade outcomes of +$60, −$40, +$60 and −$40. The ending balance is $1,040 and the largest peak-to-trough drawdown is $40. Reorder the same trades to +$60, +$60, −$40, −$40: the ending balance is still $1,040, but drawdown reaches $80. This additive calculation uses no compounding and already-net outcomes. Resampling with replacement is different: it can repeat or omit a trade and therefore change total profit. Neither operation creates evidence about trades absent from the sample.
Open this concept in Research Lab →Record what the test assumes.
- State whether you shuffle without replacement, resample with replacement, resample blocks, perturb execution costs or rerun the strategy on altered inputs. These answer different questions.
- Record the input period, trade count, random seed, number of runs, sizing, leverage, compounding, costs and the exact drawdown definition.
- Independent trade resampling treats observations as exchangeable. It may erase serial dependence, volatility clusters and the ordering of regimes; block choices introduce their own assumptions.
- A percentile of simulated drawdowns is conditional on the simulation rule and input sample. It is not a guaranteed loss limit or an unconditional probability of a future outcome.
Read the supporting sources.
- NIST/SEMATECH: Bootstrap Plot — Describes resampling with replacement to estimate uncertainty for a statistic.
- Bailey, Borwein, López de Prado and Zhu: The Probability of Backtest Overfitting — Examines selection across many investment backtests and the limits of standard holdout evaluation.
About the TraderCockpit team
We build TraderCockpit, a Windows desktop app for trading strategy research that is in development, and we write these guides to explain the methods behind testing a strategy. How we write and correct the guides →
Educational material, not financial advice. No backtest or simulation guarantees future results.