Concepts / The sealed test

The sealed test: months you never looked at

3 min read · Concept

In everyday words

A teacher keeps one exam paper locked in a drawer while you practise on all the others. Only that paper shows what you really learned.

An example in pounds

Say a rule turns £1,000 into £1,500 on four years you kept testing it on. On the next year, which you never looked at, it turns £1,000 into £1,030. The second figure is the fair one.

The short version

The newest months are kept apart until you finish, so one fair test is left.

DrawingA drawing of the idea, not a real result.

Each version you try stops before the newest months. Those months stay sealed until the end, when the rule runs on them once.

What it is

Each time you test a rule, you see how it did on past prices. Then you change it a little and test it again. Each change is kept because it looked better on the same old prices.

After enough changes, the rule starts to fit those prices. It fits their ups and downs, even the ones that came from luck. The past then looks good, whether the rule found anything or not.

A sealed test keeps the newest months apart. None of your tests may look at them while you work. When you have finished, the rule runs on them once.

Those months played no part in any change you made. So they are the one check that your changes could not bend.

A worked example

Say you test a rule on five years of prices. Quantis keeps the last 12 months apart. Your test runs on the four years before them.

You try a slower setting, then a faster one. Each try still stops before the sealed months. On the four years, your best version turns £1,000 into £1,500.

Then you open the sealed test. The same rule runs on the last 12 months, once. It turns £1,000 into £1,030. Holding the same shares all along turns £1,000 into £1,100.

The £1,500 came from years you tuned on. The £1,030 came from months you never saw. The second figure is the fair one.

What it does not tell you

What comes next. A few months are one short stretch of the past. They do not say how the rule will do from here.
Much, from one stretch. A good or a bad year can come from luck alone. One sealed test is one reading, not a final word.
Anything fresh, once used. If you change the rule after seeing the sealed months, they are not unseen for the new rule. That is why every later test says the sealed test is already used.
Prices you saw somewhere else. Quantis keeps its own tests away from the sealed months. It cannot stop you looking at a price chart.

This leaves out how Quantis decides that two tests belong to one idea, and what breaks a seal. The exact version is below.

On a Quantis result

The first time you test an idea, Quantis seals its last 12 months. A shorter test loses its last fifth instead. A test under 30 months gets no sealed months, and its result says so.

Buying shares and keeping them has no rule to tune. So a test of that seals nothing, and it covers all the years you asked for.

Every change you then make to the idea stops before the sealed months. The result card says which months are sealed, and the last day your test used.

When you are happy with the rule, the button “I'm happy with it: open the sealed test” runs it once on the sealed months. Holding the same shares over the same months is shown beside it. It costs no tokens.

After that, every test of the idea says the sealed test is already used. That label stays for good.

A goal search breaks a seal. That is a test where you name a target and Quantis tries many rules for you. It looks at the newest years, so the idea has no unseen months left.

The exact detailFor anyone who wants the detail. Nothing above depends on reading it.

The rules

One idea. One idea is every test you run of the same kind of rule (the same strategy family) on exactly the same shares in the same currency, whatever its settings, period or costs; a different kind of rule or a different set of shares is a different idea, with its own sealed months. It is the same grouping the result card's check for luck counts your tests by.
How long. The sealed months are the last 12 months of a test that covers at least 60 months, and the last fifth of a shorter one, in whole months. A test of under 30 months gets no sealed months: a fifth of it would be under 6 months, too short to say much on its own. A test's length is counted from its days, to the nearest month, so a test asked for over five years (60 months) gets the full 12.
Fixed dates. The sealed months end on the last day of the first test long enough for them, and they never move. While they are sealed, every test of the idea ends the day before them, whatever period it asks for. A test that would leave less than 6 months before them is not run, and costs nothing.
Only months nobody has looked at. A seal is made only when no earlier test of the idea in your test ledger reached those months. A goal search's candidates count up to the day of the search, because it ran them on the years it held back.
Opening it. The rule of one result made under the seal runs once, over exactly the sealed months, beside buying and holding the same shares in equal amounts. Before it is shown, its return and its biggest fall are worked out again from the two curves; a result that fails is not shown, and the seal stays closed. Quantis stores when it was opened and the exact rule it ran.
Broken. If anything in the ledger looked at the sealed months while they were sealed, the seal is recorded as broken, with the reason, and the idea has no sealed test any more.
Closed when in doubt. If the record of an idea's seal cannot be read, the test is not run (and costs nothing): run on the whole period, it could show the sealed months.

How much one year can say

The Sharpe ratio of one year of daily returns has a standard error of about 1 when the ratio itself is small (roughly the square root of 1 divided by the number of years). A sealed year that shows a Sharpe ratio of 0.5 is within that error of 0 and of 1. So the sealed test is the one result your changes could not have shaped, and it is still a small amount of history: it is read beside buying and holding over the same months, not on its own.

Where the idea comes from

Keeping a last stretch of data away from every choice, and using it once at the end, is common practice when testing models (a final holdout). The danger it guards against, picking the best of many tries on the same data, is set out by Bailey, Borwein, López de Prado and Zhu. The walk-forward test in Prove it, which picks settings on earlier years and tries them on later ones, runs inside the years before the sealed months.