The scariest thing I have ever found in my own execution logs was a position exactly twice the size it was supposed to be. No alert fired, because every component did exactly what it was written to do. The bot sent a buy, the request timed out, the retry logic did its job and sent the buy again, and both orders filled. On spot that is an oversized bag you can trim. On perps it is unintended leverage, a liquidation price you never agreed to, and a margin call you find out about from the exchange app at 3am.
Every order you send has three possible outcomes, and most bots are only written for two of them. The exchange accepts it, the exchange rejects it, or the connection dies before you hear back. A timeout tells you nothing. The request might have died in your own network stack before it ever left the box, or it might have reached the matching engine, filled instantly, and only the acknowledgment got lost on the way home. From your side of the wire those two worlds are indistinguishable, and any code that picks one of them and assumes it is guessing with your margin.
Why retries are the wrong reflex
Retry-on-failure is such a deep habit from ordinary web work that most people carry it straight into trading code. For a GET request it is harmless, since fetching a candle twice costs nothing. Placing an order is a write with money attached, and the failure mode of a duplicate write is a doubled position, so the standard retry decorator that wraps the rest of your HTTP calls becomes quietly dangerous the moment it wraps order placement.
It gets worse when you look at what actually produces these ambiguous failures. A 504 from a load balancer in front of the exchange often means the request went through fine and the balancer simply gave up waiting for the response. Connection resets, DNS hiccups mid-request, a laptop lid closing at the wrong moment if you run locally, all of these land you in the same place. The request may or may not have been processed, and the error object in your catch block contains no information either way. I treat any timeout, any 5xx, and any transport-level error on an order call as one category, which I label unknown rather than failed.
Client order IDs do the heavy lifting
The tool that makes this solvable is the client order ID. Nearly every serious venue lets you attach your own identifier when you place an order. FIX has carried ClOrdID since the nineties, Binance calls it newClientOrderId, Coinbase and OKX have their own variants, and even Hyperliquid supports a client order ID on-chain. The idea is that you name the order before you send it, so that afterwards you can ask the exchange about the order by your name for it, whether or not you ever received the acknowledgment that carried the exchange's own ID.
Two rules govern how you generate these, and both matter more than the format. First, the ID belongs to the trading decision, so derive it from the intent. Strategy, symbol, side, and the signal timestamp hashed together works fine. Second, a retry of the same decision reuses the same ID, always. If you generate a fresh UUID inside the retry loop you have built a machine for creating duplicates with perfect traceability, which is arguably worse than no IDs at all because the logs will look healthy.
Where it gets exchange-specific is what the venue does when it sees a duplicate. In my experience venues fall into three camps. Some reject a duplicate client ID outright, which gives you real idempotency, since a blind retry of an order that already landed bounces harmlessly. Some enforce uniqueness only among open orders, which sounds like the same thing but has a nasty gap: if your first attempt filled before the retry arrives, the ID is free again and the retry is accepted as a brand new order. And some venues treat the field as a courtesy label and will happily accept the same ID twice. You have to test which camp your venue is in, on testnet, by deliberately double-sending, because documentation tends to be vague on exactly this point.
Reconcile before you retry
Because you cannot count on the venue to save you, the safe pattern puts the dedupe logic on your side. When an order call ends in the unknown state, you do not retry. You reconcile first. Query the exchange for the order by your client order ID. If it comes back, the order landed, so you adopt it, record the exchange order ID, and carry on as if the acknowledgment had arrived normally. If the venue says no such order exists, you wait a short beat and check once more, because on some venues an order can be accepted but not yet visible to the query endpoint, and only then do you resend, with the same client order ID.
The lookup itself has sharp edges worth knowing about. Query-by-client-ID endpoints on some venues only cover open or recent orders, so a fast fill can vanish from the endpoint you are checking. The fallback is to pull open orders plus recent fills for the symbol and search for your ID in both. The reconcile call can also time out itself, especially since the sort of network trouble that caused the ambiguity tends to come in bursts, so the loop needs backoff and a hard cap, after which the correct behavior is to stop trading that symbol and page a human. An execution system that cannot establish its own position has no business adding to it.
The state machine
All of this compresses into a small state machine that I think every order placement path should implement. Persist every transition to disk before acting on it, because the crash you did not plan for will otherwise happen between the send and the write.
- INTENT. The strategy decides to trade. Generate the client order ID from the decision, journal the intent locally, and only then move on. This write-ahead step is what makes crash recovery possible.
- IN_FLIGHT. The order request has been sent. Nothing else is allowed to send for this intent while it sits here.
- ACKED. The exchange confirmed receipt and returned its own order ID. Store it, then track the order as usual through LIVE, FILLED, CANCELED, or REJECTED.
- UNKNOWN. The call timed out, reset, or returned a 5xx. The only legal exit from this state is through reconciliation.
- RECONCILING. Query by client order ID with backoff. Found means promote to ACKED and adopt the exchange ID. Confirmed absent after a couple of checks means demote back to INTENT, where the normal path resends with the same client ID. Repeated reconcile failures mean HALT for that symbol and an alert to a human.
The startup path falls out of the same table. On boot, load every intent that is not in a terminal state and run it through reconciliation before the strategy is allowed to produce anything new. This turns a crash mid-send from a scary event into an ordinary one, since the process picks up exactly where the journal says it left off, checks the venue for anything ambiguous, and resumes. The first time your box reboots mid-session and comes back with the position intact and no duplicates, the state machine pays for the afternoon it took to write.
A few smaller habits round it out. Cancels are naturally more forgiving, since canceling an order that does not exist returns an error you can safely ignore, so I retry cancels freely and creates never. Idempotency at the order layer also does nothing about duplicates one layer up, so if your signal generator can emit the same decision twice, the deterministic client ID derived from the intent handles that for free, and a random one does not. And log every state transition with timestamps, because when something eventually goes wrong, the sequence of states is the difference between a five minute diagnosis and an evening of guessing.
None of this is glamorous work, and that is sort of the point. We run this exact pattern inside Blockcircle's execution layer across the 150-plus exchanges we route to, and the venues differ on almost everything except the shape of the ambiguity itself. A flaky connection is a matter of when, and the machine above is small enough to build in a day. Writing it before a timeout costs you real money is one of the cheaper pieces of insurance available in this business.