What is it?
Selection bias and multiple testing: Selection bias appears when the same evidence helps choose which result to report. Multiple testing intensifies the problem: as more candidates are compared, extreme-looking results can arise from noise even when no candidate has a true advantage.
What question does it help answer?
How much of the selected result could be explained by searching many candidates and keeping only the most impressive one?
What can it tell you?
- Why the number and dependence of tried candidates matter when interpreting the selected result.
- Why a maximum should be compared with a selection-aware reference rather than treated like a prespecified observation.
- Where an experiment needs stronger holdout, correction, or resampling logic.
What can it not tell you?
- A simple independent-Normal example does not model the dependence structure of real candidate searches.
- Counting visible trials alone can miss hidden researcher degrees of freedom.
- A correction method is only as defensible as its assumptions and accounting of the search process.
See the shape before reading the formula.
The Research Lab draws independent synthetic Normal scores under a no-signal assumption. As the family expands from 64 to 1,024 candidates, the approximate typical maximum under that null rises even though the generating process contains no edge.
Open this concept in Research Lab →Use the idea without turning it into a shortcut.
Keep the context: use this concept to interpret evidence and assumptions. For TraderCockpit controls and workflow details, open Docs. The app is in development; these examples are educational.