Skip to content
Quantis Trade
Get started
← Back to the Education Centre

Prove it, shown on a search

Catching an overfit rule

Walk-through

In everyday words

Ask enough people to toss a coin a handful of times and somebody will get heads every time. They were not better at tossing. They were the luckiest of many, and their next toss is a coin toss like anyone's.

The short version

Try enough settings on past prices and one will look brilliant. This page shows what our checks did to the best-looking ones.

Why we did this

Showing that a check fails a rule that was never any good is the easy part. The harder question is whether the checks catch a rule that looks good only because it was picked out of many tries.

So we made one on purpose: we tried many settings of three simple rules on past prices, kept the best-looking rule of each kind, and then ran every check Quantis has on it.

The winners, check by check

Each card below is one winner: first its backtest as the search saw it, then every check that tried to break it, the sealed months, and the many-tries check: the deflated Sharpe ratio with every try counted.

RSI(2) below 30, exit above 70, on Advanced Micro Devices (AMD)

Its backtest as the search saw it

In-sample and fitted: this is the best of the candidates on the prices it was picked on, so it looks as good as it ever will.

Sharpe ratio 1.04, fitted: the highest of the 588 RSI candidates, and the 42nd highest of all 840 tries.

From 21 Oct 2016 to 21 Oct 2025 it grew 40.1% a year. Buying and holding Advanced Micro Devices over the same dates grew 49.1% a year.

What Prove it found

Survives 4 of 7 attacks. Main weakness: the settings chosen on earlier data kept less than half of their fitted in-sample Sharpe on the folds that followed.

  • Did not break itRandom-entry baseline

    Beats 98.5% of 200 random-timing runs on Sharpe and 97.5% on CAGR (the pass mark is 90% on both).

  • Did not break itPlacebo (shuffled prices)

    On real prices the rules beat 100% of 100 shuffled-price runs on Sharpe and 100% on CAGR (the pass mark is 95% on both).

  • Did not break itSignal delay / look-ahead check

    With one bar of delay the Sharpe ratio goes from 1.04 to 0.91 (keeps 88%); it degrades gracefully.

  • Broke itTriple trading costs

    At 3× retail costs the Sharpe ratio drops to 0.75, below 0.96 for buy-and-hold AMD (it was 0.99 at normal costs).

  • Did not break itParameter neighbourhood

    The 2 neighbouring settings within ±20% keep a median Sharpe of 0.95 vs 1.04 chosen (91.8%), and 9 of 9 grid points stay positive: a plateau, not a knife-edge.

  • Broke itWalk-forward, fold by fold

    Over 5 out-of-sample folds (1358 bars) the settings chosen in sample gave a Sharpe of 0.61 and a CAGR of 18.8%; the settings as asked gave 0.61 over the same bars. That is 43.8% of the fitted in-sample mean of 1.39; the pass mark is 50%. 5 folds cannot tell a choice of settings that carries over from one that happened to suit these 5 periods.

  • Broke itRegime and sub-period stability

    Ends 42.9% behind buy-and-hold AMD over the whole period; it beat it in 3 of 9 years.

Each line is Prove it's own sentence for that check, as its panel shows it.

The sealed months

From 22 Oct 2025 to 21 Oct 2026, months no try had looked at, the rule returned +77.7%. Buying and holding Advanced Micro Devices returned +140.0%.

The biggest fall from a high was 22.3% for the rule and 27.8% for buying and holding.

Could it be luck?

With all 840 tries counted, its deflated Sharpe ratio is 26%.

It sets the measured Sharpe of 1.04 against 1.25, the Sharpe the best of 840 tries with no skill would be expected to show. The measured Sharpe is below it.

Counted as if it were the only try, the same returns would give over 99%.

Counted the way the trial ledger groups tries (the 84 tries of the same rule on Advanced Micro Devices), it would give 71%.

The deflated Sharpe ratio says how firmly the measured Sharpe stands clear of what luck alone would hand the best of that many tries. It treats the returns as a stable sample and is not a forecast.

What broke it

  • Triple trading costs: At 3× retail costs the Sharpe ratio drops to 0.75, below 0.96 for buy-and-hold AMD (it was 0.99 at normal costs).
  • Walk-forward, fold by fold: Over 5 out-of-sample folds (1358 bars) the settings chosen in sample gave a Sharpe of 0.61 and a CAGR of 18.8%; the settings as asked gave 0.61 over the same bars. That is 43.8% of the fitted in-sample mean of 1.39; the pass mark is 50%. 5 folds cannot tell a choice of settings that carries over from one that happened to suit these 5 periods.
  • Regime and sub-period stability: Ends 42.9% behind buy-and-hold AMD over the whole period; it beat it in 3 of 9 years.
  • The sealed test: behind buying and holding the same share over the sealed months
  • The many-tries check: with all 840 tries counted, its deflated Sharpe ratio is under the line the result card marks in red

5-day and 250-day moving average crossover on NVIDIA (NVDA)

Its backtest as the search saw it

In-sample and fitted: this is the best of the candidates on the prices it was picked on, so it looks as good as it ever will.

Sharpe ratio 1.40, fitted: the highest of the 182 moving average crossover candidates, and the second highest of all 840 tries.

From 21 Oct 2016 to 21 Oct 2025 it grew 70.4% a year. Buying and holding NVIDIA over the same dates grew 68.4% a year.

What Prove it found

Survives 6 of 7 attacks. Main weakness: its edge over buy-and-hold is not spread across the tested period.

  • Did not break itRandom-entry baseline

    Beats 98% of 200 random-timing runs on Sharpe and 97% on CAGR (the pass mark is 90% on both).

  • Did not break itPlacebo (shuffled prices)

    On real prices the rules beat 100% of 100 shuffled-price runs on Sharpe and 97% on CAGR (the pass mark is 95% on both).

  • Did not break itSignal delay / look-ahead check

    With one bar of delay the Sharpe ratio goes from 1.40 to 1.37 (keeps 98.1%); it degrades gracefully.

  • Did not break itTriple trading costs

    At 3× retail costs the Sharpe ratio is 1.39 vs 1.23 for buy-and-hold NVDA; tripling costs takes 0.12 percentage points a year off its CAGR.

  • Did not break itParameter neighbourhood

    The 8 neighbouring settings within ±20% keep a median Sharpe of 1.35 vs 1.40 chosen (96.6%), and 24 of 24 grid points stay positive: a plateau, not a knife-edge.

  • Did not break itWalk-forward, fold by fold

    Over 5 out-of-sample folds (1358 bars) the settings chosen in sample gave a Sharpe of 1.40 and a CAGR of 70.3%; the settings as asked gave 1.43 over the same bars. That is 98.5% of the fitted in-sample mean of 1.42 (the pass mark is 50%). 5 folds cannot tell a choice of settings that carries over from one that happened to suit these 5 periods.

  • Broke itRegime and sub-period stability

    All of its edge over buy-and-hold NVDA comes from 2018; without that slice it would end 33.0% behind.

Each line is Prove it's own sentence for that check, as its panel shows it.

The sealed months

From 22 Oct 2025 to 21 Oct 2026, months no try had looked at, the rule returned +23.3%. Buying and holding NVIDIA returned +23.3%.

The biggest fall from a high was 20.2% for the rule and 20.2% for buying and holding.

Could it be luck?

With all 840 tries counted, its deflated Sharpe ratio is 68%.

It sets the measured Sharpe of 1.40 against 1.25, the Sharpe the best of 840 tries with no skill would be expected to show. The measured Sharpe is above it.

Counted as if it were the only try, the same returns would give over 99%.

Counted the way the trial ledger groups tries (the 26 tries of the same rule on NVIDIA), it would give over 99%.

The deflated Sharpe ratio says how firmly the measured Sharpe stands clear of what luck alone would hand the best of that many tries. It treats the returns as a stable sample and is not a forecast.

What broke it

  • Regime and sub-period stability: All of its edge over buy-and-hold NVDA comes from 2018; without that slice it would end 33.0% behind.
  • The sealed test: behind buying and holding the same share over the sealed months

100-day breakout on NVIDIA (NVDA)

Its backtest as the search saw it

In-sample and fitted: this is the best of the candidates on the prices it was picked on, so it looks as good as it ever will.

Sharpe ratio 1.40, fitted: the highest of the 70 breakout candidates, and the highest of all 840 tries.

From 21 Oct 2016 to 21 Oct 2025 it grew 68.5% a year. Buying and holding NVIDIA over the same dates grew 68.4% a year.

What Prove it found

Survives 6 of 7 attacks. Main weakness: its edge over buy-and-hold is not spread across the tested period.

  • Did not break itRandom-entry baseline

    Beats 98.5% of 200 random-timing runs on Sharpe and 97.5% on CAGR (the pass mark is 90% on both).

  • Did not break itPlacebo (shuffled prices)

    On real prices the rules beat 99% of 100 shuffled-price runs on Sharpe and 99% on CAGR (the pass mark is 95% on both).

  • Did not break itSignal delay / look-ahead check

    With one bar of delay the Sharpe ratio goes from 1.40 to 1.35 (keeps 96.8%); it degrades gracefully.

  • Did not break itTriple trading costs

    At 3× retail costs the Sharpe ratio is 1.39 vs 1.23 for buy-and-hold NVDA; tripling costs takes 0.15 percentage points a year off its CAGR.

  • Did not break itParameter neighbourhood

    The 2 neighbouring settings within ±20% keep a median Sharpe of 1.22 vs 1.40 chosen (87.4%), and 4 of 4 grid points stay positive: a plateau, not a knife-edge.

  • Did not break itWalk-forward, fold by fold

    Over 5 out-of-sample folds (1358 bars) the settings chosen in sample gave a Sharpe of 1.30 and a CAGR of 61.3%; the settings as asked gave 1.30 over the same bars. That is 93.5% of the fitted in-sample mean of 1.39 (the pass mark is 50%). 5 folds cannot tell a choice of settings that carries over from one that happened to suit these 5 periods.

  • Broke itRegime and sub-period stability

    All of its edge over buy-and-hold NVDA comes from 2018; without that slice it would end 45.5% behind.

Each line is Prove it's own sentence for that check, as its panel shows it.

The sealed months

From 22 Oct 2025 to 21 Oct 2026, months no try had looked at, the rule returned -10.8%. Buying and holding NVIDIA returned +23.3%.

The biggest fall from a high was 23.7% for the rule and 20.2% for buying and holding.

Could it be luck?

With all 840 tries counted, its deflated Sharpe ratio is 68%.

It sets the measured Sharpe of 1.40 against 1.25, the Sharpe the best of 840 tries with no skill would be expected to show. The measured Sharpe is above it.

Counted as if it were the only try, the same returns would give over 99%.

Counted the way the trial ledger groups tries (the 10 tries of the same rule on NVIDIA), it would give over 99%.

The deflated Sharpe ratio says how firmly the measured Sharpe stands clear of what luck alone would hand the best of that many tries. It treats the returns as a stable sample and is not a forecast.

What broke it

  • Regime and sub-period stability: All of its edge over buy-and-hold NVDA comes from 2018; without that slice it would end 45.5% behind.
  • The sealed test: behind buying and holding the same share over the sealed months

What this shows, and what it does not

In one line, from the run:

The search made 840 tries: 3 rule families on 7 shares and funds, on prices from 21 Oct 2016 to 21 Oct 2025, and kept the eligible candidate with the highest in-sample Sharpe in each family. Prove it broke 3 of the 3 winners; counting the sealed test and the many-tries check too, 3 of the 3 were broken by at least one check.

It shows what the checks do to rules picked for how they looked on past prices. It says nothing about how any of these shares or funds will do next.

Passing a check is not proof that a rule has found anything. A check can break a rule; it cannot prove one.

What the run itself says it does not show:

  • This is a demonstration of the method on past prices: what the checks do to rules picked for how they looked in the past. It is not a forecast for any of these shares or funds.
  • A rule the checks do not break has not been shown to work. The checks can break a rule; they cannot prove one.
  • The many-tries check treats every try as a separate draw. Many tries are near neighbours that move together, so it may allow for more luck than there was.
  • Seven attacks were run, and each can be wrong by chance: a sound strategy can fail one, and a strategy with no edge can pass the random-entry test about 1 time in 10 and the placebo test about 1 time in 20. Treat a single failure as a warning to investigate, not proof the strategy is broken, and treat a pass as 'not broken by this attack', not proof of an edge.
  • These months are one short stretch of history: a result like this can come from luck in either direction. It describes these months only, and it is not a forecast.

How this page was made

The script apps/api/scripts/overfit-demo.ts made it on 22 Oct 2026, through the same code the Strategy Agent, the goal search, Prove it and the sealed test run, with no AI model writing any part of it.

It made 1,906 engine runs (at most 1,961 were planned, under a cap of 2,500) on engine-svc 0.1.0. Prove it's random draws used the seed 20260926.

Every figure on this page is read from the report that run wrote. Run again on the same prices with the same seed, it gives the same figures.

Read next