The scariest bot failure I ever dealt with did not crash. It sat there, process healthy, green on every uptime check, answering pings, and it was placing orders into a market feed that had quietly frozen twelve minutes earlier. The prices it was reacting to were stale. The trades were real. By the time anyone noticed, the damage was done, and every dashboard we had was showing a cheerful green dot. That is the whole problem with monitoring a trading bot. The thing you are most afraid of is not the bot being down. It is the bot being alive and confidently wrong.
Uptime tells you the process is running. It tells you almost nothing about whether the bot is doing its job correctly, and for anything that touches real money, correct is the only thing that counts. So the useful question is not is it up. It is can I prove, right now, that this bot's view of the world matches reality and that its actions are landing the way it thinks they are. Answering that takes a handful of specific metrics, and none of them are hard to compute. The hard part is deciding what counts as bad and wiring an alert to it before you need it.
The four things that actually break
After enough incidents you start to see the same shapes over and over. Most silent failures fall into one of four buckets, and each one has a metric that catches it early if you bother to emit it.
Order-ack latency. When your bot sends an order, the exchange acknowledges it. Measure the time from send to ack, per order, and watch the distribution rather than the average. A creeping p99 is usually the first sign that something upstream is congesting, an API key getting rate-limited, a venue degrading, or your own event loop backing up. Averages hide this because a few thousand fast acks will drown out the handful that took four seconds, and the four-second ones are exactly the trades that will hurt you. Track the median and the p95 and p99 separately.
Fill-rate drift. Over any reasonable window, some roughly stable fraction of your orders should fill. What that fraction is depends entirely on your strategy, a market maker sitting passively will fill far less often than something crossing the spread, so there is no universal number. What matters is your own baseline. If a bot that normally fills seven of ten orders suddenly fills two of ten, something changed, and it is usually one of three things: the market moved away from you faster than usual, your pricing logic drifted, or your orders are landing late because of the latency problem above. Fill rate is a downstream symptom that catches problems your other metrics miss.
Position reconciliation. This is the one people skip and the one that eventually bites hardest. Your bot keeps an internal model of what it holds. The exchange keeps the truth. Periodically pull the real position from the venue and diff it against your internal state. Any nonzero mismatch that is not explained by an in-flight order is a bug you want to know about immediately, because it means your bot is now reasoning from a fantasy. A dropped fill event, a partial fill you counted as full, a reconnect that missed a message, all of these show up here first.
Stale-data detection. Every input feed should carry a timestamp, and you should be checking the age of the newest tick on every decision. If the freshest price you have is older than some threshold, the bot should refuse to trade, not trade harder. The frozen-feed incident I opened with was a stale-data failure with no stale-data check. The feed's socket was still open, so the connection looked healthy, but no new data was arriving. Connection state and data freshness are different things, and you have to monitor both.
Turning metrics into alerts that fire at the right time
A metric nobody looks at is a log line. To make it monitoring you have to decide a threshold, and this is where most people either give up or set something useless. A couple of rules of thumb that have held up for me:
- Alert on rate of change, not just absolute value, wherever you can. A fill rate of 30 percent might be perfectly normal for one strategy and a five-alarm fire for another. But a fill rate that halved in an hour is worth a look no matter what the absolute number is. Baselines drift; deltas travel.
- Use two thresholds, a warn and a page. Warn goes to a channel you check when you get a chance. Page wakes someone up. Order-ack p99 crossing 500ms might be a warn. Crossing two seconds, or a position mismatch of any size, is a page. Collapsing everything into one severity trains you to ignore the channel.
- Require the condition to persist. One slow ack is noise. Ten seconds of elevated p99 is a signal. Almost every metric here benefits from a short debounce so a single blip does not cry wolf. The exception is position reconciliation, where I want to know on the first mismatch, because that one does not fix itself.
- Make stale-data a hard kill, not an alert. Some conditions are too dangerous to leave to a human noticing a page. If the data is stale past a hard limit, the bot should stop trading on its own and then tell you. The alert is the courtesy. The auto-halt is the safety.
One more thing that sounds obvious and gets skipped constantly. Alert on the absence of activity, not only its presence. A bot that suddenly places zero orders for an hour during active market hours is often broken in a way that no error will ever surface, because nothing errored, it just stopped deciding to act. A heartbeat that counts expected actions per window, and pages when that count falls to zero when it should not be zero, catches a whole class of quiet deaths.
A monitoring spec you can implement this afternoon
Here is the shape I would actually build, small enough to finish in a sitting. Emit these as metrics from the bot itself, because only the bot knows its own intent, and no external prober can tell a deliberate no-trade from a broken one.
- Per-order, log send time and ack time; compute rolling p50, p95, p99 of the gap. Warn on p99 over your normal ceiling, page on a hard multiple of it.
- Per window, track orders sent versus orders filled. Store a trailing baseline and alert on a large relative drop, say a fill rate that falls below half of its trailing average.
- Every minute or two, pull real positions from each venue and diff against internal state. Page on any unexplained mismatch. Log the diff so you can see whether it is one symbol or systemic.
- On every decision, check the age of the newest tick per feed. Warn as it ages, hard-halt trading past a strict limit, and emit the freshness value continuously so you can see it climbing before it trips.
- Count intended actions per window and alert when that count is zero during hours you expect activity. Pair it with a plain process heartbeat so you can tell a dead process from a live but idle one.
That is five signals, all of them cheap, and together they cover the great majority of ways a bot fails without falling over. The mindset behind them is the part worth keeping. Treat every number the bot believes as a claim to be checked against the exchange, and treat every feed as guilty of being stale until its timestamp proves otherwise.
When we built the execution and monitoring side of Blockcircle across a lot of venues, the reconciliation loop and the stale-data halt were the two checks that paid for themselves first, long before anything fancier. They are unglamorous and they will save you from the failure that does not announce itself. Start there, get the thresholds roughly right, and tighten them the first time something slips through.