Turn a validation funnel into one bounded evidence statement.
By the end, you should be able to name the rule, sample, stress or retest performed at each stage; identify where selection information could have leaked forward; and say which stage, if any, actually changes your conclusion.
Recover the chain before interpreting the colors.
- Which candidate or result is being evaluated?
- Which metric definitions and cost assumptions are in force?
- Which stages were specified before the result was seen?
- Which sample belongs to development and which belongs to evaluation?
- Can you recover the evidence identity for the exact run?
Ask a different question at every stage.
01 · AcceptanceWas the decision rule predefined?02 · RetestDid a more demanding execution model preserve the result?03 · Time slicesIs the aggregate result concentrated in a few years?04 · RegimesDoes behavior depend on a narrow market state?05 · Path stressHow sensitive is the conclusion to sequence uncertainty?06 · HoldoutWas evaluation information protected from selection?07 · EvidenceCan the conclusion be tied to the exact evaluated artifacts?
Passing more stages does not automatically make the conclusion stronger.
A later stage is useful only when it adds genuinely different information. Re-running a related test on the same decisions and same sample can create the appearance of independent confirmation without adding much independent evidence.
Good form: “The result met its stated acceptance rule, remained similar under a higher-precision retest, was not concentrated in one calendar year, remained sensitive to one regime, and showed material path uncertainty under the stated Monte Carlo rule. The holdout remained separate and the evidence chain identifies the evaluated artifacts.”
Treat failure as information about the claim, not as a prompt to shop for a friendlier test.
- Acceptance failure: revisit the research premise or rule; do not move the threshold after seeing the result and call it confirmation.
- Higher-precision failure: investigate execution assumptions, fill modeling, costs, and data resolution before trusting the coarser result.
- Year or regime concentration: narrow the claim to the conditions actually supported, then test whether that conditioning was selected after the fact.
- Monte Carlo weakness: report the uncertainty rule and failure region instead of hiding the unfavorable tail.
- Out-of-sample weakness: treat the selection story as unresolved; repeated holdout peeking can invalidate the intended separation.
- Evidence-chain break: stop before interpreting the result if you cannot establish which exact artifact was evaluated.
A seven-stage funnel can still be seven ways to overstate the same evidence.
- Counting correlated checks as independent confirmation.
- Changing stage thresholds after seeing failures.
- Running many alternative robustness tests and reporting only the favorable ones.
- Calling a calendar slice a market regime without a defined state model.
- Treating simulation replicate count as additional historical information.
- Letting the holdout influence feature, parameter, or candidate selection.
- Confusing a complete record with economic validity: identity can be perfect while the research design is weak.