Hey trader,

Picture this…

You fire up an AI coding tool, describe a trading strategy you’ve been kicking around for years, and by dinner you’ve got a working backtest.

(A backtest runs your trading rules against old market data to see how they would’ve done.)

Out comes an equity curve that climbs like it’s late for a meeting. If you haven’t seen one before, an equity curve is just a chart of what your account balance would’ve been over time if you’d traded the strategy.

Up and to the right means you got rich. Down and choppy means you’d have been better off buying an index fund and taking up golf.

So you stare at this beautiful line and start pricing out boats.

I’ve been that guy more times than I’d like to admit. I’ve been building trading tools with AI for a while now, and that gorgeous curve shows up early and often.

Almost every time, something underneath it was broken.

That’s why i’m starting this today…

From here on, I’m going to write regularly about AI and trading: building tools with it, using it to sharpen your own trading, where it gives you a real advantage, and where it’ll quietly burn you. I’ll share what I’ve run into building trading strategies and working with financial data, especially the stuff that bit me.

I’m opening with the pitfalls, because the biggest one catches almost everybody.

People trust the AI, and their data, way more than either one has earned.

Everything starts with data, and data’s messier than you’d hope.

If you’re testing options strategies, you need historical option prices and current ones, and they have to be clean.

Same deal for stocks.

Plenty of sources won’t give you what you need. Others give you something that looks right but has holes in it: missing days, stale quotes, prices stamped at the wrong time.

Your equity curve won’t warn you about any of that. It just draws a line. If one bad price in, say, March 2022 turned a losing trade into a winner, the line ticks up a little higher and you go right on shopping for that boat.

So now, before I believe a single number, I tell the AI to go back through the backtest and hunt for pricing and timing anomalies that could be skewing the results. I make it check its own homework.

I don’t wait until something smells off. I do it every time, because the AI finds a problem on that first pass every time.

Then I fix what it found and run the backtest again.

That second run tells me whether the strategy actually holds up. Sometimes it still works, and sometimes the whole edge turns out to be a data glitch and the boat goes back on the shelf.

Sometimes there are more data issues to sort out.

If that sounds paranoid, the pros found pretty backtests don’t mean much either.

Back in 2016, Quantopian, a platform where quants built and tested trading algorithms, studied 888 strategies that had at least six months of real trading after their backtests.

They looked hard at the Sharpe ratio, a score for how much return a strategy gets for the risk it takes. Higher is better, and it’s the number people love to wave around to prove a strategy works.

The backtest Sharpe ratio explained less than 2.5% of how those strategies did afterward. Worse, the more a developer tested and tweaked a strategy, the wider the gap got between the backtest and real life. Tinker enough and you’re mostly teaching the strategy to memorize the past.

Programmers are hitting the same wall. Stack Overflow surveyed roughly 49,000 developers in 2025, and 66% named AI solutions that are “almost right, but not quite” as a problem. Forty-five percent said debugging AI-written code eats more time than they expected.

More of them distrust AI’s accuracy (46%) than trust it (33%). These folks write code for a living, and even they don’t take the AI’s word for it. If you’re newer to programming, you’ve got even less reason to.

One of the companies we work with takes this so seriously that it pulls from as many redundant data sources as it can. It compares them against each other, fills the holes, and flags any price that doesn’t match.

That’s the gold standard. It also runs thousands of dollars a month, which is a tough sell when your trading app is a weekend project.

You don’t have to go that far. You can get a long way with free or low-cost sources like Massive (the company formerly known as Polygon.io). Some brokerages also let you pull data through their APIs, which is basically a pipe that lets your program grab data straight from them. You’ll have to dig to find out which ones offer it and what’s included.

On a budget, the checking matters even more. You don’t have a second source around to catch the first one’s mistakes.

That tells you something about working with AI in general. Everything’s more complicated than it looks, and a system that seems to work can still be working badly. You have to go looking for problems even when nothing looks wrong.

And the hunt doesn’t stop once you go live. When you start trading the system, or running the alert service, or whatever you’ve built, new anomalies will show up that the backtest never caught.

So keep checking. Go back on a regular schedule, see what popped up, figure out what it hit, and fix it. Do that long enough and you’ll have a system you can actually trust.

Over the coming weeks, I’ll get into a lot more of what I’ve learned building trading services with AI, including the stuff that’s worked.

Charlie