A trading bot almost never dies the way you expect it to. You imagine a crash, a red stack trace, a pager going off at three in the morning. What actually happens is quieter. The bot keeps running, keeps placing orders, keeps reporting healthy, and somewhere in the middle of that healthy-looking loop it starts doing something dumb. By the time you notice, the equity curve has a kink in it and you are reading logs trying to reconstruct a decision the machine made an hour ago with information that was already wrong when it made it.
I have watched enough of these post-mortems to notice the same handful of failure modes keep coming back. None of them are exotic. Most of them are boring plumbing problems dressed up as strategy problems, which is exactly why they are so hard to see. You go looking for a bug in your alpha and the real culprit is a clock. Here are the seven that show up most, and for each one, how you actually catch it.
The four that come from the world moving under you
The first cluster is about the gap between what your bot believes and what is actually true. That gap is where most of the damage lives.
Stale data feeding fresh decisions. This is the big one and it is sneaky because nothing errors out. Your price feed hiccups, a websocket silently stops pushing updates, or a cache holds a value one tick too long, and your bot keeps computing signals against a number that stopped being real. It sizes a position, it fires an order, and it is trading against a price that moved thirty seconds ago. The detection method is a freshness clock, not a health check. Stamp every piece of market data with the time you received it, and refuse to act on anything older than a threshold you set deliberately. If the last tick is more than a couple of seconds old on a fast market, the bot should not be allowed to have an opinion. The fix is to make staleness a first-class blocking condition, the same way you treat insufficient balance.
Clock drift breaking signed requests. Almost every exchange API signs requests with a timestamp and a receive-window, and rejects anything outside it. If your server clock drifts by even a few seconds, your perfectly valid orders start bouncing with signature or timestamp errors, and if you are not reading the error body carefully you will swear the API is broken. It is not. Your clock is. Run NTP, monitor the offset, and treat a growing drift as an incident before it becomes an outage. A useful habit is to log the exchange server time alongside your own on every rejected request, so the pattern is obvious the moment it starts.
Precision and rounding on quantity. Every market has a minimum order size, a step size for quantity, and a tick size for price. If your position sizer produces 0.0333333 of something and the step size is 0.001, the exchange either rejects the order or silently truncates it, and now your live position does not match what your bot thinks it holds. That mismatch compounds. Reconciliation drifts, stops get placed against the wrong quantity, and you find out during a drawdown. The fix is unglamorous and non-negotiable: pull the instrument filters from the exchange, round every quantity and price to the allowed increment before you send anything, and never use floats for the final rounding step. Use decimals. Floating point will betray you on the eighth digit at the worst possible moment.
Unhandled exchange maintenance windows. Exchanges go down on purpose. They schedule maintenance, they halt specific markets, they enter cancel-only mode where you can close positions but not open them. A naive bot treats a maintenance window as either a stream of errors it retries forever, or worse, it comes back online after the halt and dumps a queue of stale orders into a market that has moved. The detection is to actually parse the exchange status endpoint and the specific error codes for halts, rather than lumping everything into a generic retry. The fix is a state machine that knows the difference between "try again in a second" and "this market is closed, stand down and re-evaluate from scratch when it reopens."
The two that come from your own process rotting
The next two have nothing to do with the market. They are about the fact that a trading bot is a long-running process, and long-running processes accumulate problems that short scripts never live long enough to hit.
Memory leaks in long-running processes. A backtest runs for a minute and exits, so nobody notices it leaks. The same code running live for three weeks slowly eats every gigabyte on the box. Common culprits are unbounded lists of tick history you keep appending to and never trim, event listeners you attach and never remove, and reconnect logic that spawns a new websocket handler on every drop without killing the old one. The symptom is a bot that behaves beautifully for days and then gets slow, then gets killed by the OS out-of-memory reaper mid-trade. Detection is just plotting resident memory over time and looking for a line that only goes up. The fix is bounded buffers everywhere, ring buffers for history, and a hard rule that anything you subscribe to, you also unsubscribe from.
Config drift between environments. Your bot works in staging and misbehaves in production, and the reason is that the two environments are not actually the same. A different API endpoint, a testnet key where you meant mainnet, a leverage setting that was 1x in the config you tested and 5x in the one that deployed, a fee assumption that is right on one venue and wrong on another. This one is dangerous precisely because everything looks fine until real money is on the line. The fix is to make config diffable and to log the full effective configuration at startup, every value, so that when something goes wrong you can compare exactly what each environment was running. If a human is hand-editing production config, you have already lost. Generate it, check it in, and diff it before every deploy.
The one that is not a bug at all
Strategy decay mistaken for a bug. This is the failure mode that eats the most engineering time for the least reason, because there is nothing to fix. The bot is working perfectly. It is executing exactly the strategy you gave it. The strategy just stopped making money, because the market regime that produced its edge went away. Volatility compressed, a spread you were harvesting closed, the flow you were front-running got crowded out by everyone else running the same idea. You will spend a week auditing your order logic looking for the leak, and the leak is that your alpha expired.
The way you tell decay apart from a defect is that a defect usually leaves a fingerprint. Slippage worse than modeled, fills at prices you did not expect, orders rejected, a reconciliation mismatch. Decay leaves clean execution and a flat-to-down curve. So before you go bug hunting, check the boring diagnostics first: were the fills clean, did realized slippage match your model, did every order do exactly what it was told. If the answer is yes across the board and you are still bleeding, you do not have a bug. You have a strategy that needs retiring or re-fitting, and the honest move is to size it down or turn it off rather than keep debugging code that is behaving.
How to actually catch these
The through-line across all seven is that a trading bot needs to distrust itself. Most of these failures are invisible to a plain uptime check because the process is alive and the loop is turning. What catches them is a second layer of monitoring that watches for silent wrongness rather than loud crashes. A short checklist I keep coming back to:
- Freshness clock on every input, and refuse to trade on stale data.
- NTP offset and rejected-request rate as monitored metrics, not afterthoughts.
- Instrument filters pulled live and applied with decimals, never floats.
- A real state machine for halts and cancel-only mode.
- Resident memory plotted over the life of the process.
- Full effective config logged at startup and diffed before deploy.
- Reconciliation between believed position and actual exchange position, on a loop, alerting on any drift.
That last one, continuous reconciliation, quietly catches half the list. If your bot's idea of its position and the exchange's idea of it ever disagree, something above went wrong, and you want to know in seconds, not at end of day. When I built the execution layer at Blockcircle to route across a lot of venues, the reconciliation loop earned its keep more than any single clever piece of strategy code, because it turned silent drift into a loud alert I could act on. The strategy is the part everyone wants to talk about. The plumbing is the part that decides whether you keep the money you make.