In-Sample vs Out-of-Sample Backtesting: Avoiding Data Leakage

Plain-language definition

What is it?

In-Sample vs Out-of-Sample Backtesting: Avoiding Data Leakage: In-sample backtesting uses data that helped develop, fit or select the strategy. Out-of-sample backtesting evaluates it on data withheld from those decisions. A holdout stops being unseen for a later decision when its results inform that decision, even if the file is still labelled out-of-sample.

Research question

What question does it help answer?

What evidence was genuinely withheld from the decisions that created or selected the candidate?

Bounded interpretation

What can it tell you?

Whether evaluation evidence was separated from the decisions being assessed.
Where a research process risks feeding holdout information back into selection.
Why a clean information boundary matters even when the numerical split percentage is unchanged.

Limitations

What can it not tell you?

A holdout cannot erase bias already introduced by data cleaning, feature design, universe selection, or repeated experimentation.
One successful holdout result does not guarantee stability in later regimes.
Calling a dataset OOS does not make it independent if it influenced the process indirectly.

Synthetic example

A holdout becomes feedback.

Synthetic example: a researcher develops a moving-average rule on 2020–2023 data and reserves 2024 for one final test. After seeing a losing month in 2024, the researcher changes the entry threshold and reruns 2024 until the result improves. The first test evaluated the frozen rule; the improved rerun now evaluates a rule selected with knowledge of 2024. Renaming that year OOS does not restore its independence. Record the original result and the change, then evaluate the revised rule on a genuinely untouched later period under a predeclared protocol.

Open this concept in Research Lab →

Assumptions

Record what the test assumes.

Split time in order and freeze the strategy, parameter search, scoring rule, data preparation and execution assumptions before the final evaluation.
Indicators, universe membership, corporate-action data and features must use information actually available at each simulated decision time. Fit preprocessing on development data only.
Account for overlapping labels, trades crossing a split and temporal dependence. A chronological split alone does not remove every route for leakage.
Record every candidate tried and every consultation of validation data. Data used for model selection is validation data, not an untouched final test.

Primary references

Read the supporting sources.

scikit-learn: data leakage — Explains why fitting preprocessing and choosing models using test data can bias evaluation.
Bailey, Borwein, López de Prado and Zhu: The Probability of Backtest Overfitting — Examines selection across many investment backtests and the limits of standard holdout evaluation.

About the authors

About the TraderCockpit team

We build TraderCockpit, a Windows desktop app for trading strategy research that is in development, and we write these guides to explain the methods behind testing a strategy. How we write and correct the guides →

Educational material, not financial advice. No backtest or simulation guarantees future results.

Apply the concept

Use the idea without turning it into a shortcut.

Keep the context: use this concept to interpret evidence and assumptions. For TraderCockpit controls and workflow details, open Docs. The app is in development; these examples are educational.

Continue learning