The first time an exchange went dark on me while I had a position open, I did the worst possible thing, which was nothing, because I had no plan and did not want to make it worse by guessing. The book froze, my last fill was twenty minutes stale, and I had no idea whether the venue was down for everyone or just for my IP. By the time the API came back the position had drifted well past where I thought my stop was, and the stop had never triggered because the matching engine had been asleep. That was the day I started treating downtime as a thing my bot needs a written procedure for, the same way it has one for entries and exits.
Most bot builders wire up the happy path beautifully and then handle failure with a bare try and except that logs and moves on. That works until the venue is not throwing a clean error but a slow, ambiguous, half-broken response your code reads as success. The interesting problem is not what to do when an exchange is obviously down. It is what to do in the ten minutes before anyone admits it is down, while the venue is still answering but lying to you.
Reading the signature of a venue that is about to fail
Exchanges rarely go from healthy to hard-down in one step. They degrade first, and the degradation has a recognizable shape if you log the right things. The two signals I lean on most are error rate and latency, watched as a rolling window rather than a single sample. A single 500 means nothing. Five 500s in thirty seconds against a venue that never errors is your early warning.
Latency is the sneakier one. What matters is not the absolute number but the drift relative to that venue's own baseline. An endpoint that usually answers in under a hundred milliseconds and is suddenly taking two seconds means something upstream is queueing, and queueing is often the last thing you see before everything starts timing out. So I keep a per-venue, per-endpoint baseline and compare live latency against it, not against a fixed global threshold that treats a fast exchange and a slow one the same.
A few specific things worth watching, because they each mean something slightly different:
- Rising HTTP 5xx or timeouts on order endpoints while market data still flows. The matching engine is struggling even though the read path looks fine, so do not trust that read path.
- Order acknowledgements that arrive but never transition to filled or resting. The order went into a void and you do not know its state.
- Websocket disconnects that will not reconnect, or reconnect and then send a stale snapshot. A reconnect that hands you old data is worse than a clean disconnect, because it looks alive.
- Rate-limit responses on request volumes that were fine an hour ago. The venue has scaled something down and is shedding load, which is a soft form of downtime.
The rule of thumb I use: if I cannot confirm the state of my own orders and balances within a couple of seconds, I treat the venue as untrusted, even if it is still returning data. Untrusted does not mean panic. It means stop sending new orders and switch into a read-and-confirm loop until I regain confidence or decide the venue is gone.
Hedge somewhere else, or sit tight
Once you have flagged a venue as degraded, the real decision is whether to act elsewhere or wait it out. The core question is whether your risk is bounded or unbounded while the venue is down. If you are holding spot and the exchange freezes, your downside is just that you cannot trade for a while. Annoying, not dangerous. Sitting tight is usually correct, because opening an offsetting position elsewhere means that when the venue returns you have two positions to unwind and basis risk between two books that may have moved apart.
Leverage flips this. If you hold a perp or margin position and the venue that could liquidate you goes dark, your risk is now unbounded, because price keeps moving while your ability to react is gone. This is where hedging on a second venue earns its keep. Long a perp on a frozen exchange with the market falling, a short of similar size on a healthy venue caps your exposure until the first one wakes up. You are not trying to make money on the hedge. You are buying time and flattening delta so the reopen is not a disaster.
A short decision guide I keep pinned, roughly in order:
- Is the position leveraged or capable of forced liquidation? If no, strongly prefer sitting tight.
- Can I hedge the same or a strongly correlated instrument on a venue I trust? If no, sitting tight may be forced on me anyway.
- What does the hedge cost in fees, funding, and slippage versus the tail risk it removes? If the tail is large and the hedge is cheap, hedge.
- Do I have a clean way to unwind the hedge the moment the primary venue returns? If unwinding is messy, that is a real cost, so count it.
One failure mode to name directly: do not hedge blind when you are unsure of your position size. If the venue went dark mid-fill, you might be holding more or less than you think, and hedging the wrong quantity turns a delta problem into a bigger one in the other direction. When size is uncertain, hedge conservatively toward flat, never past it.
Resynchronizing when the API comes back
The reopen is where the quiet damage happens, because everyone is relieved and wants to resume immediately, and resuming too fast means acting on stale state. Before your bot sends a single new order after an outage, it should refuse to trust anything it remembers and rebuild state from the exchange as the source of truth.
My resync sequence looks like this. Pull open orders straight from the venue and reconcile them against what my bot thinks is open, cancelling anything orphaned and adopting anything the venue shows that I lost track of. Pull balances and positions fresh and compare to my ledger, because a fill may have landed during the blackout that I never got a confirmation for. Only after all three agree do I re-enable order submission. If any disagree beyond a tiny tolerance, I halt and flag for a human rather than let the bot trade on state it cannot verify.
Two edge cases catch people. First, duplicate orders. If you retried a submission during the flaky period, the venue may have accepted both copies, so dedupe on client order IDs and cancel the extra. Second, the mid-blackout stop. A stop-loss that never fired because the engine was down needs re-evaluating against current price the instant you reconnect, because the level it protected may already be well behind you, and re-arming it blindly is how a small loss quietly becomes a large one.
None of this is glamorous, and most of it is plumbing you write once and are grateful for later. Routing across many venues on Blockcircle, the boring parts are health checks, baselines, and a resync routine that assumes the exchange is right and the bot wrong. Write the playbook while nothing is on fire, because an outage is a bad time to invent your procedure, and it always arrives while you have a position on.