The rules were fixed before the test
Were the entry, exit, sizing and pass/fail thresholds written down before any results were seen, and is every later change disclosed?
TRADERCOCKPIT // TEST THE CLAIM
Trading Strategy Backtest Checklist: How to Evaluate the Evidence
A trading strategy backtest checklist should establish the rules, data, costs, selection history and genuinely withheld evaluation behind a performance claim. Request reproducible evidence and record missing assumptions before interpreting the headline return.
By the TraderCockpit team · Updated · Educational material, not financial advice.
Claim / exact wording
Source / creator
Date encountered
Use one rule: a polished chart is not evidence. Check what was fixed before the result, which data influenced the design, how many variations were tried, and what data is still unseen.
Every number can be traced
Can each figure be traced to a trade list or report, with the timeframe and symbol matching the chart being discussed?
Costs and fills are stated
Are commissions, spread, slippage and the fill assumptions (market or limit, next bar or same bar) stated, and do they fit the market and size traded?
Some data was held back
Was an out-of-sample period set aside before development and opened only once, for the final check?
Seen data is not reused as unseen
Once a period has influenced a decision, it cannot later confirm the strategy. Cross-validation inside the same window does not make it unseen.
Every attempt was counted
How many rule sets, parameters and markets were tried? The best of many trials needs a tougher bar than a single planned test.
There is a fair comparison
Was the result compared with a baseline chosen up front, such as buy and hold or random entries of the same size and holding time?
The sample is large and varied enough
Do the trade count, the length of the test and the market conditions covered support the claim, allowing for trades that are not independent?
Paper and live trading follow the same rules
Do forward, paper or live results use the same rules and data handling as the backtest, with any differences explained?
The mechanism is checkable
If the claim offers a market reason, is each step observable, stated as a cause rather than a correlation, and confirmed on a second related market?
The pitch sells evidence, not returns
Does it avoid promised returns, invented testimonials and unexplained figures, and keep the evidence separate from the outcome?
STOP / CANNOT VERIFYA required piece of evidence is missing: the rules, the trade list, the costs or the held-back data.
DISCOVERY ONLYThe evidence comes from data that was used to build or choose the strategy.
TESTED ON UNSEEN DATAThe rules were fixed in advance and checked on held-back or forward data. This still makes no performance promise.
A backtest screenshot reports $1,000 gross profit across 100 trades. At an assumed $8 round-trip cost per trade, net profit is $200. Adding $3 of slippage per trade changes it to a $100 loss. These are illustrative fixed dollar costs, not a broker quote or product result. Request the trade list, fill assumptions and cost model before judging the screenshot.
ASSUMPTIONS AND LIMITATIONSThe checklist assumes the underlying rules, data and experiment history can be inspected. Missing evidence means the claim cannot be evaluated from the supplied material. Passing these checks does not establish future profitability. Trade count alone is not an adequacy test: dependence, risk, regimes and uncertainty matter.
ABOUT THE TRADERCOCKPIT TEAMWe build TraderCockpit, a Windows desktop app for trading strategy research that is in development, and we write these guides to explain the methods behind testing a strategy. How we write and correct the guides. Educational material, not financial advice.
PRIMARY REFERENCES AND RELATED GUIDESThe Probability of Backtest Overfitting examines selection across many trials. scikit-learn’s data leakage guidance explains evaluation contamination. Continue with in-sample vs out-of-sample backtesting, Monte Carlo uses and limitations and reading backtest metrics together.