At any given moment, Bitcoin is trading at slightly different prices on dozens of exchanges. The price on Binance might differ from Coinbase by a few dollars, which differs from Kraken by a few more. Market data aggregators like CoinGecko, CoinMarketCap, and various professional data providers need to collect all of these prices and produce a single reference price that reflects the true market value.
The data collection layer is the foundation. Aggregators maintain connections to every major exchange API, typically using WebSocket connections for real-time trade and order book data. For a comprehensive aggregator, this means maintaining hundreds of concurrent connections, each with its own data format, rate limits, and reliability characteristics. Some exchanges push data reliably. Others have frequent disconnections and require constant reconnection logic.
Normalization is the next challenge. Every exchange has its own naming conventions, timestamp formats, and data structures. One exchange might call it BTC/USDT while another calls it BTCUSDT or XBT/USD. Trade sizes might be in base currency on one exchange and quote currency on another. The normalization layer needs to translate all of these into a consistent internal format before any meaningful aggregation can happen.
Reference price calculation is more complex than a simple average. Volume-weighted average price across exchanges is a common approach, giving more weight to exchanges with higher trading volume. But volume itself can be unreliable. Wash trading inflates reported volumes on some exchanges, which would skew a naive VWAP calculation. Sophisticated aggregators apply outlier detection and volume adjustment to filter out suspected wash trading.
Latency varies significantly across the data pipeline. An exchange API might introduce 10-100 milliseconds of delay. Network transit adds more. Processing and aggregation add a few more milliseconds. By the time a reference price is published, it might be 50-500 milliseconds behind the actual market. For a retail trader looking at a price chart, this is invisible. For an algorithmic system making split-second decisions, it can be significant.
Order book aggregation is even more complex than trade aggregation. A true aggregated order book shows the total liquidity available across all exchanges at each price level. This requires maintaining a real-time view of every exchange order book and merging them. The challenge is that order books change thousands of times per second, and each exchange sends updates at different rates. The aggregated view is always a slightly stale approximation.
Data quality monitoring runs continuously behind the scenes. If an exchange reports a trade at a price that is far from the consensus, it might be a legitimate outlier or a data error. Aggregators use statistical methods to detect and handle these cases, typically flagging or excluding trades that deviate beyond a certain threshold from the reference price. This prevents a single erroneous trade on a low-volume exchange from moving the aggregated price.
Historical data storage and retrieval is the other side of the problem. Storing every trade from every exchange at full resolution generates terabytes of data per year. Different users need different resolutions. A researcher analyzing long-term trends might want daily OHLCV candles. A backtester needs minute or tick data. The data infrastructure needs to support efficient queries across all of these time scales.
The market data landscape in crypto is still maturing compared to traditional finance, where consolidated tape requirements ensure a single authoritative price feed. In crypto, there is no regulatory mandate for consolidated data, which means every aggregator makes slightly different choices about which exchanges to include, how to weight them, and how to handle edge cases. The prices you see on different platforms will always differ slightly as a result.