Test catalog — every test the project has run, by kind

As of 2026-07-25. Sources of record: bot/runs/registry.db (execution-level runs), bot/runs/experiments.db (scans/measurements, 137/2000 hypotheses), preregistrations in bot/preregistrations/, scan declarations in engine/scans/.

Kind 1 — Walk-forward backtests (parameters FITTED, then tested out-of-sample)

Grid-searched on in-sample windows, evaluated only on stitched out-of-sample windows (walkforward.py). Used whenever a spec has tunable parameters.

Kind 2 — Zero-free-parameter event-follow backtests (nothing fitted → no walk-forward needed)

Every constant inherited from a prior declared scan and frozen in a prereg BEFORE the run; sequential simulation with stops/costs; coin-flip control on identical events. Walk-forward exists to stop fitted parameters leaking; with zero fitted parameters there is nothing to leak — the residual risk is hypothesis selection, handled by Kind 4.

Kind 3 — True holdout test (the only genuine out-of-sample evidence)

Frozen spec, never-scanned data, pre-registered criteria, machinery guard (bit-exact reproduction of the research run first), one shot, fail-is-final.

Kind 4 — Layer-3 event-study scans (no execution; four-null battery)

Declared-then-run through Edge Lab (or its bespoke predecessor). No fills, stops, or costs — pure "does the market move after X" with unconditional, frequency-matched-random, label-shuffle, and ISO-week block-bootstrap nulls. All on the Research split.

Kind 5 — Descriptive measurements (Track M; no returns evaluated)

Kind 6 — Robustness characterization (not evidence of edge)

Kind 7 — Machinery verification (proves the tools, not the market)

• pytest suite: 191 tests — fills/R-math, cost itemization + frozen hashes, kill criteria, gate fail-closed, registry lineage, executor/MT5, level-engine causality (truncation identity), paper-loop streaming==batch equivalence, MC block-vs-iid proof.
• Reproduction guards: validation runner reproduced run 19 bit-exact before touching Validation; Edge Lab acceptance reproduced the displacement scan bit-for-bit from a config; dashboard renderers assert numeric identity with frozen CSVs before rendering.
• Published-record anchors: London Breakout / Dual Thrust / Donchian reproduced published-consistent failures (the harness finds nothing where nothing should be).

The overfitting question, answered honestly

Protections in force: pre-registration before every run (constants frozen, one run per registration); zero-free-parameter execution specs; every hypothesis logged (137/2000) with project-wide BH-FDR; four-null batteries; coin-flip controls on identical events; dose-response/neighborhood checks instead of single-point thresholds; two-instrument replication requirements; the ex-ante cost kill rule; and the Research/Validation/Vault split with a 10-look lifetime budget enforced in code.

Residual risk, stated plainly: research-split results are still selected — the streak finding emerged from scanning that same data, so its +0.107R is best read as an upper bound. That selection effect is exactly what the Validation split exists to catch — and run 20 proves the catch works: displacement looked at least as good on research and failed the holdout decisively. Nothing in this project is called an edge until it survives a Validation look, and nothing is called deployable until Vault + paper. By design, the honest answer to "did we overfit?" is: the research numbers probably flatter; the process assumes so; the holdout decides.