Check whether a result is luck

The problem

Every time you change a rule after seeing a result, you fit it a little more closely to the past.

Fit it enough and the past looks good whether or not the rule has found anything.

Months that played no part in any of your changes are the one check that fitting cannot reach, so they are kept apart until you have finished.

Try enough rules, or enough settings of one rule, and one of them will look good by chance. A single result cannot tell you which kind you have.

Step by step

1. 1

   Finish a test

   Run your idea in the Strategy Agent, or open a saved one on My Strategies.

2. 2

   Open Prove it

   On the result, the Prove it section has one button. It shows its price before anything runs.

3. 3

   Read each attack

   Each attack tries to explain the result another way, such as buying on random days, and says whether the result survived.

4. 4

   Open the sealed months

   When you have finished changing the rule, press "I'm happy with it: open the sealed test" in the full report. It runs once, over months no test of the idea has looked at.

5. 5

   Watch it from today

   Press "Track it with pretend money" and the rule meets prices that have not happened yet.

A worked example from a stored test

This is the report the engine produced for a golden cross on SPY, tested from 1 Oct 2021 to 24 Sep 2026. It made 328 test runs, and the rule survived 2 of 6 attacks.

In the random-entry attack, the rule's Sharpe ratio was higher than in 72% of the random runs. The pass mark is 90%, so it did not pass: buying and selling on random days, as often, did about as well.

Over a longer stretch the same rule lost to simply holding: a stored 10-year test of a golden cross on SPY ended at $27,277 against $40,513 for simply holding, from $10,000, 5 Oct 2016 to 1 Oct 2026. Prove it does not change a result. It shows what the result rests on.

What it cannot tell you

• Surviving every attack does not show that a rule will keep doing well. It shows the result did not fall apart under those attacks, on those years.
• Settings chosen on earlier years and run on later ones give only a few later stretches, fold by fold; few stretches cannot tell a choice that carries over from one that happened to suit them.
• The sealed test can be opened once per idea. After that, every test of the idea says so.
• Prove it costs more than one test, because it makes many runs. It shows its price first.

Try it

Run one of these, then open Prove it on the result. Nothing is sent until you press the button.