Hey Traders,
I built a trading system with AI. Backtested it. The thing showed a 78 percent win rate.
I was already mentally spending the money.
Then I sent it to Tim Colby. Tim runs the Reboot newsletter, has managed nine figures and specializes in vetting trading systems. Forty-five minutes later, the 78 percent win rate I was so proud of? Closer to 40.
That's the problem with using AI to build trading strategies. AI is a brilliant intern who will confidently hand you garbage if you don't know what to ask for. And if you don't know what can go wrong with a backtest, you'll never spot it when AI gets it wrong.
Here's what I was building.
The strategy that fooled me
It's called the ORB system. Opening Range Breakout. The idea is simple: the first 15 to 30 minutes after the market opens form a high and a low. That's your "opening range." If the price closes above the high, you go long. If it breaks below the low, you short. The bet is that the early move sets the tone for the day.
Beginner-friendly. Clean rules. Plenty of YouTube videos selling it as a printing press.
Real-world win rate when traders run it honestly? Around 40 to 55 percent, depending on filters. Most of the profit comes from a handful of big trend days, not a steady drip of wins.
My 78 percent backtest, in other words, was suspicious before I even hit run.
What Tim caught
The first thing Tim asked me: how far back did you test? I said two years. He said test as far back as possible.
The system broke once the window passed five years.
So I tweaked it. Got it looking great over 15 years. Tim said pull the equity curve and the monthly returns for that whole span. I did. And I found that two or three weird anomalies, weather events in the data really, were carrying the entire backtest. Strip those out and the edge disappeared.
There's a name for what I was doing without realizing it. It's called overfitting (designing a strategy that fits the past so perfectly it can't survive the future). And there's a cousin called survivorship bias (testing on stocks that exist today and quietly ignoring the ones that went bankrupt during your test window). Both inflate win rates by 1 to 4 percent a year if you don't check for them. That's the gap between a great-looking backtest and a real one.
After cleaning all that up, I think I'm finally on to something. But here's the next problem.
Just because a strategy wins on stocks doesn't mean it wins on options. The math is completely different. Theta, implied volatility, bid-ask spreads, you can't backtest any of it with the precision you can backtest stock returns. You have to paper trade the live market and watch what actually fills.
And one more Tim catch: open the charts and confirm the bot is actually following your open/close rules. For three days I caught inconsistencies. Trades opening where I said they shouldn't. Exits firing late. Every one of those inconsistencies was probably warping my backtest too.
Get yourself a Tim
I'm good with AI. I mean it. But I'm only as good as what I know to check.
A trading bot is a parrot with a calculator. It'll repeat whatever you tell it to look for, and it'll do the math fast and clean. What it won't do is tell you that you're asking the wrong question. That's a human job. And if the human asking the questions doesn't know what overfitting is, what survivorship bias is, or why a five-year backtest isn't enough, the bot's results are worthless.
I'm lucky. I have Tim.
A lot of traders building AI strategies right now do not. They run a clean-looking backtest, skip paper trading, fund the account, and donate.
So here's my actual advice. If you're building a system, don't trust the first backtest. Don't trust the second one either. And do not fund it with real money until paper trading confirms what you saw on the historical data.
Get yourself a Tim. He runs a free newsletter, Reboot.
And this coming week, Tim is going to reveal a “barometer” he’s put together that uses his more than 16 years of trading experience to put out winning trades.
Sign up to Reboot here and find out exactly when Tim will reveal his new strategy.
Charlie Delvalle
Editorial Director
Option Pit