The bug that scared me most wasn't in the strategy. It was a rounding error in position sizing that only showed up when the account balance crossed a specific decimal boundary, and it turned a modest long into something about ten times too big. The backtest never caught it because the backtest fed clean, pre-rounded numbers. The bot in production got the raw balance from the exchange, did its own arithmetic, and got it wrong. Nobody tests the arithmetic. Everyone tests the alpha.
So this is a rough map of how I think about testing a trading bot, borrowed from the usual software testing pyramid but bent to fit the ways trading systems actually break. The short version is that unit tests catch your math, integration tests catch your assumptions about the exchange, and chaos drills catch everything else, which is most of it.
Unit tests: the boring math is where the money leaks
The parts of a trading bot that most deserve unit tests are the least glamorous. Position sizing, fee and slippage math, order rounding to the exchange's tick and lot sizes, PnL accounting, and any conversion between quote and base currency. None of this involves the strategy. All of it can quietly move real money in the wrong direction.
My rule of thumb is that anything that turns a signal into a number the exchange will act on gets a test, and the tests are mostly about edge cases rather than the happy path. What happens when the balance is zero. What happens when a fill comes back partial. What happens at the minimum notional, where the exchange rejects orders below some floor and your sizing logic has to either round up or skip the trade. What happens when you compute a stop that lands on the wrong side of the current price, which is a surprisingly common way to accidentally market-out of a position the instant you open it.
A few cases I now write by default:
- Sizing at tiny balances, where rounding down to the lot size can produce a zero-size order or, worse, silently drop a decimal.
- Sizing at large balances, where you cross a boundary and the number does something you didn't expect.
- Fee-inclusive PnL, so a string of scratch trades that look flat actually show the fee drag they really carry.
- Sign handling on shorts, because half the sizing bugs I've seen come from a strategy that was only ever tested long.
These tests run in milliseconds and never touch a network. That's the point. You want the fast, deterministic layer to be big, because it's the only layer you'll actually run on every commit.
Integration tests: testnets are useful and also lying to you
Once the math holds, the next question is whether your bot speaks the exchange's language correctly. Auth, order placement, cancels, reading back fills, handling the specific error codes that exchange returns. This is what testnets and sandboxes are for, and they are worth using. Most major exchanges run a testnet with fake balances where you can place real orders against a real-ish matching engine.
Here's the part people skip past. A testnet is not production, and the ways it differs are exactly the ways that hurt. Testnet order books are typically thin or synthetic, so slippage and fill behavior look nothing like the real market. Rate limits are often looser, which means your bot passes on testnet and then gets throttled the moment it goes live. Latency is different. And the testnet is usually running a slightly older or newer version of the API than production, so an endpoint that works in one can behave differently in the other.
So I treat integration tests as a check on protocol correctness, not on behavior. I want to confirm that a limit order gets placed, that a cancel actually cancels, that a partial fill reads back the way I expect, and that each documented error code maps to the right branch in my code. What I don't trust the testnet to tell me is how the strategy performs, how much I'll pay in slippage, or whether my rate limiting is tuned right. Those need production data, small size, and patience.
One habit that has saved me repeatedly is recording real API responses from production, warts and all, and replaying them in tests. Exchanges return weird things. Timestamps in the wrong unit, a status field that's a string here and a number there, a fill that arrives before the order acknowledgment. Capture those responses once and you can test against them forever without hoping the testnet reproduces them.
Chaos drills: break it on purpose before the market does
The failures that actually take bots down are almost never clean. They're a websocket that drops mid-position and reconnects to a stale state. An order that gets rejected for a reason you didn't handle, so your bot thinks it has a position it doesn't. A rate limit that trips during a fast move, exactly when you most need to act. An exchange that returns a 200 with an error buried in the body. A duplicate fill message that double-counts your size.
Chaos drills are where you inject these on purpose in a controlled setting and watch what your bot does. The setup can be crude. A proxy between your bot and the exchange that you can configure to drop connections, delay responses, mangle payloads, or return canned errors. Even a set of unit tests that feed your order-handling code deliberately broken responses gets you most of the value.
The questions I'm trying to answer with each drill are always the same. Does the bot know its true position after a disconnect, or does it trust its own memory. Does a rejected order leave it in a consistent state, or does it now believe a fiction. When it reconnects, does it reconcile against the exchange as the source of truth, or assume nothing changed while it was gone. Reconnection reconciliation is the single most important behavior to drill, because the gap between what your bot thinks it holds and what the exchange says it holds is where the largest losses hide.
A short list of drills worth running before any bot touches real size:
- Kill the connection mid-order and confirm the bot re-syncs position from the exchange, not from local state.
- Reject the next order and confirm the bot doesn't proceed as if it filled.
- Return a duplicate fill and confirm size isn't double-counted.
- Delay a response past your timeout and confirm the bot doesn't fire a second order that ends up as a double entry.
- Return malformed JSON and confirm the bot fails safe rather than acting on garbage.
The pre-deploy checklist
None of this is about achieving some coverage number. It's about walking a specific set of failure paths before real money does. Before I let a bot trade live, I want green on the math, a clean pass against the testnet for protocol correctness, and every chaos drill above run at least once with the bot behaving sanely. Then I want it running in production at the smallest size the exchange allows, doing real trades, for long enough that the differences the testnet hid have a chance to show up.
The backtest tells you whether the idea has any edge. Everything above tells you whether the code will survive contact with a live exchange long enough to find out. Those are different questions, and the second one is the one that empties accounts. So write the boring tests, break your own bot on purpose, and then start small.