The first version of every crypto data pipeline I have seen, including my own, is a Python script that opens a WebSocket to one exchange and appends JSON to a file. It works for about a day. Then the connection dies quietly at 3am, the script never notices because the socket still looks open, and you find a six hour hole in your data two weeks later when a backtest spits out numbers too good to be true. Nearly everything worth knowing about this problem lives inside that one failure, so it is worth walking through how to build the boring version that survives.
The encouraging part is that the scale is friendlier than people assume. A busy spot pair on a major venue prints somewhere between a few hundred thousand and a few million trades a day, and each trade is a tiny record. Track a few dozen pairs across three or four exchanges and you are looking at tens of millions of rows a day at the very worst, usually far fewer. One VPS with a few cores, a reasonable amount of RAM, and an SSD handles that comfortably if you write in batches. You do not need Kafka for this, and you do not need a cluster. You need a socket handler that reconnects properly and a database that likes append-heavy time-stamped data.
Ingestion is mostly a reconnection problem
The WebSocket part looks trivial in tutorials, and it is where most of the real engineering lives. Exchanges disconnect you as a matter of routine. Binance closes every connection after roughly 24 hours by design. Other venues drop you when they deploy, when you read too slowly, or for reasons you will never learn. So the consumer has to treat disconnection as the normal case. The loop is connect, subscribe, read, and on any error or any silence, tear the whole thing down and reconnect with backoff.
Silence is the subtle one. A TCP connection can die in a way that never raises an error, and your read call will happily wait forever. Every exchange gives you some heartbeat mechanism, ping frames or a periodic status message, and you should enforce it on your side rather than trusting the library to do it. My rule is that if nothing at all arrives for some multiple of the expected heartbeat interval, call it 30 to 60 seconds on a liquid pair, I kill the connection and rebuild it. A false positive costs one reconnect. A false negative costs hours of missing data, and you will not know which hours until much later.
Structurally, run one asyncio process per exchange rather than one big process for everything. Python's websockets library handles dozens of subscriptions in a single event loop without strain, and separate processes mean a parsing bug in your Kraken handler cannot take down your Binance feed. Keep each process dumb. Parse the message, normalize it, push it onto a buffer. Anything clever belongs downstream, where a crash does not lose live data.
Normalization, or why bitcoin has three names
The same asset trading against the same currency is BTCUSDT on Binance, BTC-USD on Coinbase, and XBT/USD on Kraken, which still carries the old XBT code around in parts of its API. That is the easy case, because at least it is obviously the same thing. The genuinely dangerous case is two different tokens sharing a ticker across venues, which happens with small caps more often than you would like, and which string manipulation will merge into one poisoned series without a single error being raised.
The only approach I trust anymore is an explicit instrument table that you own. Each row says this exchange plus this raw symbol maps to this canonical instrument, and the mapping was made by a human or at least reviewed by one. Any new symbol that shows up on the feed and matches nothing goes into a review queue instead of being guessed at. It feels like bureaucracy for a hobby-scale pipeline, and it is also the cheapest insurance you can buy against quietly mixing two assets for months.
While you are at the edge, normalize units too. Some venues report volume in base currency, some in quote, some both. Some tell you the aggressor side of a trade, some do not. Define one canonical trade record, something like timestamp, exchange, instrument, price, size in base units, side where known, and the venue's trade id, then force every message into that shape before it touches storage.
The database, and the two-timestamp rule
At this scale almost any serious time-series store works, so pick on operational familiarity rather than benchmarks. TimescaleDB gets you compression, retention policies, and continuous aggregates while staying plain Postgres underneath, which matters at 2am because whatever error you hit has twenty years of answers written about it. QuestDB and ClickHouse ingest faster and scan faster, and they are the right call if you outgrow the Postgres route. For a one-box setup I default to TimescaleDB and have not regretted it at retail volumes.
Whatever you choose, batch the writes. Row-at-a-time inserts will make the database your bottleneck long before the hardware runs out. Buffer ticks for a second or two, or until a few thousand rows accumulate, then flush them in one insert or COPY. That single change typically takes you from thousands of rows per second to tens of thousands on the same modest machine.
Store two timestamps on every tick, the exchange's own timestamp and the moment your process received the message. Exchange time is what you chart and backtest against. Receive time is the debugging channel, because it exposes latency spikes, clock skew between venues, and whether a quiet stretch was the market being quiet or your pipeline being broken. Together with the venue's trade id, these two fields make out-of-order data a non-problem. Insert everything as it arrives, deduplicate on exchange plus instrument plus trade id, and sort by exchange time when you read. Do not fight for ordering at ingest time, because the database genuinely does not care what order rows land in.
Gaps deserve the same explicit treatment. Detect them with sequence numbers where the venue provides them, and with your heartbeat monitor everywhere else. Then backfill over REST, since most major exchanges expose recent trade history through paginated endpoints keyed by trade id, which means a reconnect can be followed by fetching exactly the window you missed and deduplicating it against what you already stored. Keep a small table that logs every detected gap and whether it was filled. When a backtest looks suspicious, that table is the first place I check, and it has settled the argument more than once.
What I check before trusting the data
- Kill a connection by hand and watch the reconnect, the gap detection, and the REST backfill run without you touching anything.
- Compare a full day of stored trades against the exchange's own candles. The volumes will not match to the cent, but they should be close, and a large mismatch means dropped messages.
- Grep the instrument table for symbols that were auto-mapped rather than reviewed, especially anything small cap.
- Query receive timestamp minus exchange timestamp and look at the tail of the distribution. A fat tail means the consumer falls behind under load.
- Project disk usage a year out, with compression on, before subscribing to more pairs.
None of this is glamorous, and a weekend is honestly enough to get the first honest version running on hardware that costs less per month than a takeaway dinner. The pipelines that go wrong at this scale tend to fail silently rather than loudly, and every piece above, the heartbeats, the two timestamps, the gap log, exists to turn silent failure into loud failure. Start there, and add the fancy parts only when a query is actually slow.