State the claim before choosing the evidence.
Write down what you are trying to learn and what observation could count against it. A vague goal such as “find a good strategy” makes it too easy to change the question after seeing results.
Why selection changes interpretation →Separate development information from evaluation information.
Track which observations influenced design, tuning, filtering, or candidate choice. Evidence that remained unused during development answers a different question from evidence that helped produce the result.
Open out-of-sample evidence →Keep definitions, units, costs, and sample scope attached.
A metric name is not enough. Record the exact definition used, unit, cost assumptions, position-sizing basis, observation count, and sample so later comparisons refer to the same thing.
How to read result metrics together →One realized path is not the whole distribution.
Inspect path shape, drawdown, dispersion, skew, tails, and alternative outcomes under clearly stated assumptions. Uncertainty analysis narrows a question; it does not turn a historical result into a forecast.
Look beyond the single point that won.
Ask whether conclusions change under nearby parameter choices, time windows, regimes, execution assumptions, or other justified perturbations. A broad stable neighborhood usually supports a different interpretation from one isolated optimum.
Count the search that produced the reported result.
The more candidates, metrics, thresholds, or variants inspected, the more important it becomes to retain that search context. “Best observed” is not the same evidential claim as “prespecified before evaluation.”
Open selection bias →Preserve enough context to know what was actually evaluated.
Keep the source, sample, configuration, run, revision, result, and validation identity distinguishable where those records exist. A complete record supports auditing; it does not prove that the method was valid.
Read about evidence chains →