Full daily price histories, checked and extended without deleting anything

Some symbols had only about six years of daily prices. An earlier job that loaded prices for stocks traded by members of Congress asked for six-year windows, and our loaders skipped any symbol that already had prices, so the rest of the history never arrived. A 10-year backtest on one of those symbols could only start in 2020. From 28 September, histories are checked for this and extended back to the first daily bar our price provider has. Our daily price data now covers about 2,280 symbols.

Examples. Checked against our price provider, IEF (a 7 to 10-year US Treasury fund) has daily bars back to July 2002, 6,079 of them, where we had held 1,506 starting in 2020. Realty Income (O) goes back to 1994, Simon Property Group (SPG) to 1993, Teva (TEVA) to 1982 and Vodafone (VOD.L) to 2000.

How a history is checked. A history counts as complete when its first stored bar is within 14 days of the first trading date the provider reports (or 1970 for US symbols, the earliest date our loaders request) and its bars cover at least 90% of weekdays. Otherwise we fetch the full daily history with the same guards as our other loaders: daily bars only, London prices converted from pence to pounds after a currency check, and unit glitches refused.

Nothing deleted, disagreements stopped. Where the fetched history overlaps what we hold, closing prices must agree within 1%. If more than two bars disagree, or any bar is more than 10% off (typically a share split applied on one side only), that symbol is left exactly as it was and flagged for a person to check. Up to two isolated bars that had been stored mid-session are corrected, with the old values backed up first. New dates are added and nothing is ever deleted. Every check is logged, and extended symbols also get dividend history for the added years.

Going forward. New symbols picked up from congressional trading disclosures now get their full history straight away, and the old six-year setting is ignored, with a log line saying so if anyone sets it. Backtests are still limited to 10 years, but a 10-year test on these symbols now covers the full 10 years.