The bug that taught me to respect this was subtle in the worst way. Our local book looked fine. Bids below asks, sensible spread, sizes that moved when the market moved. But every so often a limit order would sit unfilled at a price the tape had clearly traded through, and the only tell was that our top-of-book was a few ticks off from what the exchange actually had. Nothing crashed. Nothing logged an error. The book had just quietly drifted, and it had been drifting for a while before anyone noticed.
That is the whole problem with maintaining a depth feed yourself. A book that is slightly wrong behaves almost exactly like a book that is right, and by the time you can see the difference you have usually already sent an order against bad data. So the work is less about parsing messages fast and more about being paranoid at exactly two moments: when you first build the book, and when you suspect it has gone out of sync.
Why you need both the snapshot and the stream
Most exchanges give you two things. A REST endpoint that returns the full book at a point in time, and a WebSocket stream that pushes incremental changes, usually called deltas or diffs. The snapshot is a photograph. The stream is everything that happened after the shutter clicked. Neither one alone gives you a live book. You need to glue them together, and the seam is where everything goes wrong.
The naive version looks reasonable and is broken. You subscribe to the stream, then you fetch the snapshot, then you start applying deltas on top of it. The problem is timing. Between the moment the exchange generated your snapshot and the moment your first delta arrives, there is a gap, and in that gap the book kept changing. If you just start applying whatever deltas show up, you will either replay updates that are already baked into the snapshot or skip ones that never made it in. Either way you are now off by a little, and little compounds.
The fix every serious exchange documents is sequence numbers. Every delta carries an id, and usually a range: this update covers events from sequence X to sequence Y. The snapshot carries a sequence number too, the id of the last event it includes. Your job is to make those line up.
The sequencing dance, step by step
Here is the ordering that actually holds up, and the order of operations matters more than any individual step:
- Open the WebSocket and subscribe to the depth stream first. Do not fetch the snapshot yet.
- Start buffering every delta you receive into a queue. Do not apply anything. Just hold them.
- Now fetch the REST snapshot. Note its last-update sequence number, call it S.
- Walk your buffered deltas and throw away any whose entire range ends at or before S. Those events are already inside your snapshot.
- Find the first delta whose range picks up right where the snapshot left off. Different exchanges phrase this differently, but you are looking for the delta where the previous sequence equals S, or where the range straddles S, so there is no gap and no overlap.
- If that first usable delta does not connect cleanly to S, throw the whole book away and start over from step one. A gap here means you are already desynced and no amount of clever patching fixes it.
- Apply that delta and every buffered one after it in order, then apply live deltas as they arrive, forever.
The step people skip is number six. They find a delta that is roughly in the right place, apply it, and move on. But if there is a hole between the snapshot and your first applied update, you have built the book on sand. The connection has to be exact. Subscribe before you snapshot, because if you snapshot first there is a window where updates can slip past before your subscription is live, and that is the gap you can never see.
Checksums, when you are lucky enough to have them
Some exchanges hand you a gift: a checksum. Periodically, or on every message, they send a hash computed over the top of the book, usually the first several price and size levels on each side, concatenated in a defined order and run through something like CRC32. You compute the same hash over your local book and compare.
This is the single best drift detector you will ever get, so use it if it exists. The reason it is so good is that it catches the silent failures. A book can be off by one stale level deep in the ladder, invisible to any eyeball check, and the checksum will scream about it on the next tick. When they match, you have cryptographic-ish proof your book agrees with theirs. When they do not, you know immediately, not three minutes and one bad fill later.
The fiddly part is reproducing the exact string the exchange hashed. The number of levels, the ordering of bids versus asks, whether prices are formatted as raw strings or normalized, whether zero-size levels are included, the delimiter between fields, all of it has to match byte for byte. Getting a checksum to line up is genuinely annoying the first time and then it just works. Budget an afternoon for it and read their reference implementation instead of guessing from the prose.
Detecting and recovering from a desync
Assume you will desync. Networks hiccup, you fall behind processing under load, an exchange has a bad moment. The question is not whether but how fast you notice. A few signals to watch:
- A sequence gap. The next delta's starting id is higher than one past your last applied id. Something got dropped. This is the loudest and most reliable signal, so treat any gap as fatal.
- A failed checksum, if the exchange offers one.
- A crossed book, where your best bid is at or above your best ask. Real books cross for microseconds during matching, but a persistent cross means you missed a delete.
- Negative or nonsensical sizes after applying an update, which usually means you applied a delta out of order or twice.
The recovery is boringly the same in every case, and boring is what you want here. Stop trusting the book, stop sending orders against it, and rebuild from a fresh snapshot using the exact sequencing dance above. Do not try to patch a desynced book back into agreement by fetching a snapshot and diffing. Just tear it down and rebuild. The rebuild is cheap, a stale book is expensive, and the code path you exercise on every reconnect is the same one you already trust at startup.
One practical guardrail. Gate your execution on book health explicitly. Keep a simple flag that says the book is synced, flip it to false the instant you see a gap or a failed checksum, and refuse to price or route anything while it is false. This is one boolean, and it is the difference between a bad tick and a bad fill. When we wired the depth feeds behind Blockcircle's execution layer, that flag did more to keep us out of trouble than any amount of parsing speed.
If you take one thing from this, let it be the order of operations at startup: subscribe, buffer, snapshot, align on sequence numbers, and refuse to proceed if the seam does not connect exactly. A book that drifts silently is worse than one that loudly refuses to serve, because at least the loud one tells you to go rebuild it.