the temptation of tantalus

If you’ve ever spent enough time around 18-25 year old finance hopefuls you might find yourself momentarily believing you’re rubbing shoulders with a couple of the next Buffetts and Mungers. I’ve sat through a lot of these conversations: overnight crypto pumps, high frequency options trading, and some strange techniques returning 20x that nobody could quite explain. The stories of seemingly boundless reward were tantalising, in the original sense. Luscious fruit hung just overhead, begging to be grabbed. I was itching to put my own money on the line and take a shot at it. Some of them even had backtests to back them up, and those never lie, right?

…Right?

That’s roughly how I got into testing strategies, and, spoiler alert, how I was disillusioned almost immediately. A large chunk of backtests people flaunt are a little less than honest, and oftentimes they don’t even know it. Information leaks in from the future, costs get ignored, and the dataset conveniently forgets every company that went broke. As it turns out, a large part of professional quant research boils down to hunting these lies down in your own work before believing anything. Advanced cynicism, wrapped in maths, which sounded like a fun endeavour to dip my toes into.

I could’ve tested strategies on an existing platform, but existing frameworks have already made all the decisions where the mistakes, and therefore the learnings, lie: when fills happen, when costs count, how the data gets split. I would’ve inherited their answers and none of the thought reflexes. I can’t audit a black box I didn’t build, which is a problem when the black box is the thing under investigation.

So I built all of it myself: this is the strategy-research arm of quantlab, my miniature trading firm. Through the process, I hit just about every pitfall you can imagine, and many of those inform the article you’re about to read.

the engine, and the first lesson

The heart of any of this is the backtesting engine: the simulator that replays market history one day at a time, hands each day to a strategy, executes whatever trades it asks for, and tracks what the portfolio would have been worth. Every result in this article comes out of it, so if it lies, everything downstream lies with it. That’s why I built it first, and why it got the paranoia.

Before any strategy, though, data. I took ten years of daily prices for about 500 current S&P names plus benchmark ETFs, pulled from free sources into Parquet files. A quick validation layer rejected about 19 tickers outright for broken histories. One of the checks I implemented (“no sane stock moves 50% in a day”) falsely rejected PG&E during its 2019 bankruptcy, which was a pretty quick reminder that data cleaning rules encode assumptions about what markets are allowed to do.

The engine itself started as the ‘simplest working’ thing, so to speak. Two hundred lines that read prices, walk forward one day at a time, and track cash and positions. The only real design rule it had was to make lookahead bias structurally impossible, rather than just discouraged. Strategies are handed only the price history up to yesterday, as a sliced array. Signals computed on day T trade at day T+1’s open. In plain terms: if the strategy decides to buy after seeing Tuesday’s close, the earliest a real trader could act is Wednesday morning, so that’s when the engine fills it. Trading Tuesday’s decision at Tuesday’s price means using a number that didn’t exist yet when the decision was made, and that single-day slip is enough to make a losing strategy look brilliant. I even added a deliberate cheat strategy to the test suite: one that tries to trade on a price it shouldn’t be able to see yet. The test confirms the cheat is impossible to write at all, not merely that it loses money. The engine never hands a strategy tomorrow’s data, so there’s nothing to cheat with in the first place. Transaction costs are always on.

The first strategy through the engine was a simple moving average crossover on single names like AAPL and SPY. It lost to just buying and holding (on Apple, over the decade: +332% vs +1,197%), and it barely paid any costs at 11 trades in a decade. Cost drag scales with turnover, a fact that’ll ruin a couple of strategies later in this article.

As for the cheating strategy: the demo below runs the same toy rule twice on real SPY data, once trading at the close it just peeked at, once realistically at the next open. Both curves are drawn from the same signals; hover to step through the dates and watch the gap grow.

pairs trading: a DSP student meets a regime change

With an engine I could finally trust, I needed a strategy actually worth running through it, and a moving average crossover wasn’t going to cut it. I started with the classic statistical arbitrage play: find two stocks whose prices historically move together (cointegrated), then trade the gap when it stretches, betting it closes. There’s no ticker for a pair; you build the position yourself out of two ordinary trades placed at once: short the stock that’s gotten expensive, buy the one that’s gotten cheap. Sized correctly, the two legs cancel out whatever the broad market does, so you’re left betting on one thing only: that the gap between them snaps back to normal. Mechanically, the strategy watches the spread between the two stocks and bets when it stretches more than two standard deviations, exiting when it comes home.

A scan across about 190 candidates surfaced pairs like Google/Home Depot, Microsoft/Walmart, and J&J/UnitedHealth. Tested fairly (pairs selected on 2016–2020 data, traded only on 2021 onward), the textbook version failed, badly. The Google/Home Depot pair lost 41.1%, Microsoft/Walmart lost 16.4%. In practice, the relationships drift, so a fixed hedge ratio goes stale, and recalculating and refitting it daily churns fees.

The textbook version re-fits the hedge ratio from a rolling 250 day window every single day, and that daily re-fitting generated roughly 1,400 trades in five years, paying costs on every single adjustment. The losses weren’t only costs, though: the signal itself decays. Cointegration isn’t an immutable, unbreakable law, but more of a snapshot of two businesses that happened to be pulled by the same forces for a while. Google and Home Depot moved together through 2016–2020; then the pandemic re-priced e-commerce and housing on completely different logic, and the “stable” relationship my scan had certified simply stopped existing. The strategy kept betting the gap would come home, but it never did.

As a bit of a signal processing nerd (colossal understatement, see also: bqst, my DSP project) I realised all I needed was a method of ignoring high frequency noise while accounting for gradual drift. A Kalman filter was the obvious answer. It treats the hedge ratio as a slowly drifting quantity. Each day the filter predicts where the spread should sit if the pair is behaving normally, along with how confident it is. When the real price arrives, any gap between it and that prediction is the pair misbehaving right now: the filter has already accounted for the slow drift, so what’s left over is exactly the stretched-spread signal the strategy wants. Measured against the filter’s own uncertainty, that gap is the z-score directly. That single daily update hands the strategy both things it needs at once. The hedge ratio tells it how many shares of each stock to hold so the two legs stay balanced against each other, and the prediction error tells it when the gap has stretched far enough to be worth a trade. The old textbook approach had to compute these separately, one pass over past data to estimate the ratio and another to measure how stretched the spread was; the filter rolls both into one step, with no windows of history to pick and only two noise settings to tune. And because its ratio estimate barely moves from day to day, the strategy sets its position once at entry and holds it, instead of re-adjusting daily and paying a fee on every tweak. On my first calibration it turned Google/Home Depot’s -41% into roughly breakeven and cut drawdowns three to six times over.

It’s around here that I made the exact mistake I’d built the engine to prevent. That first calibration of the filter’s two tuning knobs had drawn on the test window it wasn’t supposed to see. When I retuned them on the selection window alone, the Google/Home Depot pair lost 20 points and flipped from roughly breakeven to -19.4% net. So the honest result is -41% turned into -19%: still a loss, but half the bleeding. Across all three pairs the truthful filter still beat the static version comfortably (average net +2.3% versus -19.3%), so the adaptivity genuinely helped. But cash-like returns on the good pairs with a losing one in the mix means daily-bar pairs trading on free data just isn’t profitable.

The chart below plots the hedge ratio itself over time, estimated two ways: the textbook 250-day refit whipsawing between extremes, and the Kalman estimate gliding through the same underlying drift. The selection window is shaded; step through the dates and watch the gap between them open up. Every jump in the jagged line is a re-adjustment the strategy would have paid costs on.

factors and survivorship bias

Pairs trading bets on one relationship at a time, and I’d just watched those relationships die. The other end of the spectrum is to stop caring about any single stock and trade the whole market at once: rank all 500 names by some signal, buy the top slice, re-rank every month. Strategies built that way are called factor strategies, and they meant some big engineering decisions first. The daily event engine assumes cleanly aligned histories, and a 500-name universe is ragged with IPOs and gaps. Rather than generalise the engine before it needed it, I ran the factor tests as a separate monthly pipeline: form the signal at month-end, earn the following month’s return. The lookahead protection carries over to this format automatically: the pipeline still can’t reach next month’s data early.

Then I ran the classic factor strategies as a matched pair: 12-1 momentum (buy the past year’s winners) and its defensive opposite, low-volatility (buy the calmest names). Momentum backtested at +840% over nine years; low-vol at +208%, trailing as a defensive factor should, especially over a decade-long bull run.

The +840% is the one that stands out, of course. Maybe those finance hopefuls I was talking about at the start were onto something. That’s the type of number I could have led this article with, but also exactly the kind of number you should treat with a healthy dose of scepticism. The universe is today’s S&P members, so companies that failed along the way have vanished from the data. To investigate, I measured the damage by benchmarking my survivors against the real equal-weight S&P ETF that held the losers as they died: +4.6% per year of pure survivorship bias, persistent in 94 of 125 months, and worst exactly where it flatters momentum.

Momentum, specifically, gets flattered: momentum buys recent winners, and a survivors-only universe is made of winners: every stock that cratered and got delisted has been removed from the list of things the strategy could have bought. So the bias doesn’t inflate all strategies equally; it most inflates the strategy that chases the survivors’ defining trait.

The chart below grows a single dollar two ways: through my survivors-only universe, and through RSP, the ETF that kept the losers as they dropped out of the index. The widening wedge between the two curves is the bias itself. The meter underneath then translates that gap into the haircut on the +840% headline, compounding the measured 4.6%/yr away across the nine years.

the ML layer that lost

Then the machine learning layer. The dataset here is a few hundred monthly cross-sections of noisy, non-stationary tabular data, which puts it firmly in gradient-boosted-trees territory; a neural net on data this small would mostly be a more expensive way to overfit. So: gradient boosting over five price-based features (momentum at two horizons, last month’s reversal, volatility, and distance from the 52-week high), trained to predict which stocks outrank others next month. Predicting what a stock will return next month is close to hopeless, so I didn’t even try. Predicting that stock A probably beats stock B is a gentler target, and a portfolio only needs the ordering anyway.

Again, I evaluated it the strict way: walk-forward with an embargo, costs included. Ordinary cross-validation shuffles time, letting the model train on the future it’d later be tested on. Walk-forward means the model only ever predicts years it hasn’t seen, with a gap so nothing bleeds across the boundary.

The result was +325%, versus plain momentum’s +446% over the same 2020+ window (that’s the full-period +840% shrunk to the shorter walk-forward window, not a contradiction), and it wasn’t even a smoother ride for the trouble: Sharpe 0.96 against momentum’s 1.24. The feature importances came back nearly flat, around 0.20 each; the model found no hierarchy among the five signals, it just blended correlated things into mush, diluting the strongest one and adding turnover. Annoyingly, that’s exactly what the literature warned me would happen. I kept the negative result as the centrepiece: The ML layer, with critical evaluation, lost to the boring baseline. If nothing else, it’s a result I can defend, which is more than I can say for +840%.

mining the community archives

These factors were alright, but it wasn’t close to the mythical success stories I’d heard. Luckily for me, the internet is full of famous strategies with spectacular backtests. Quantopian left behind an archive of them. I picked three of the most-cited and tested them with the same scepticism.

First, the famous one. Peter Guenther’s “In & Out” was the most-hyped strategy of Quantopian’s final year: hold equities normally, but flee to bonds when a basket of proxy ETFs (industrial metals, the dollar, the gold-to-silver ratio) flashes early signs of stress. Its published backtest showed roughly 1,700% since 2008. Now these were the numbers I’d been trying to get at. On my 2016-to-2026 data it returned 306% with a Sharpe of 0.85, while simply holding the Nasdaq-100 returned 600% at 0.94. It shaved three points off the worst drawdown (-32% versus -35%) but gave up half the return to do it, parked in bonds 20% of the time and flitting in and out. The famous number was practically all derived from the 2008 crash, which my data didn’t reach. It was effectively a worse buy-and-hold with a nervous disposition. Despite all that, I’m paper trading it anyway, clearly labelled, to watch it underperform in real time.

The only one I was actually hopeful for. Scaling momentum to a constant volatility target is one of the few momentum improvements most academic literature says survives out of sample, by dodging the sharp reversals that make momentum dangerous. I pre-registered that it would improve risk-adjusted return. It didn’t. It had the same -20% drawdown, a lower Sharpe (1.02 versus 1.24), and barely a third of the return (160% versus 446%, on the same 2020+ window as the ML test). Volatility targeting protects against momentum crashes, and the sample never handed this portfolio one. Scaling into cash during loud months only forfeited upside for insurance that never ended up being claimed anyway.

The one that reused my other project. Chris Cain’s “Quality Companies in an Uptrend” combines a quality score (profitability, low debt) with momentum and a market-regime filter. Building it let me wire this project into the finetuned AI analyst: the quality scores come from the same SEC fundamentals that project extracts, under a strict point-in-time discipline so the backtest never reads a number before it would have been published. It returned 344% against plain momentum’s 840%, with a worse drawdown (-28% versus -20%) and a worse Sharpe (0.86 versus 1.22): the regime filter dragged in a decade that mostly went up, and actually worsened the drawdown by whipsawing, exiting after drops and re-entering after recoveries. The quality tilt gave up momentum in a growth-led market.

the fate of tantalus

The three famous strategies only dug up three unimpressive negatives, with one thread running through all of them. On a survivorship-inflated bull decade, every filter, gate, and clever overlay is just drag. Plain momentum is stubbornly, boringly, and infuriatingly hard to beat. Every spectacular number pulled away the moment I reached for it with a fair test.

I started this whole project with a small, naive hope that I’d leave with a golden ticket. Maybe I’d find one of those unusual strategies those finance hopefuls talked about, and I’d never have to work a day for the rest of my life.

I’m still writing these portfolio articles, so you know how that went.

important lessons, disappointing truths

1. Discipline is the product. Not the strategies, the reflexes. Lookahead made impossible to write rather than merely discouraged, costs always on, and biases measured in numbers someone can argue with (+4.6%/yr).
2. Adaptivity beats refitting. The Kalman filter was a DSP-to-finance transfer that turned -41% into -19% (still a loss, but half the bleeding), and the tuning leak I caught in my own work cost 20 points when confessed. That gives you a pretty good indication of how much the lies are worth.
3. Boring baselines are extremely hard to beat. An ML layer, three famous community strategies, and a stack of clever overlays all lost to plain momentum once tested fairly. Nobody dangles that result in front of you.

where to next?

Research is only one arm of a trading firm. The signals built here feed the execution side, a matching engine, a risk-control gateway, and live paper trading, covered in building a trading firm’s machinery. The AI research arm, fine-tuning a small open model into an analyst whose memos are verifiably correct, is covered in distilling a financial analyst.