Every whale data vendor demos the same way. A feed scrolls, large numbers move, somebody says the phrase real time twice, and the meeting ends with everyone agreeable. Nothing in that half hour touches the questions that decide whether the feed survives contact with a research process, and almost all of those questions live in the gap between what the marketing surface claims and what the grid underneath it actually carries.
What follows is the questionnaire I would send, worked against Blockcircle's own Whale Alpha feed because it is a capable product and it still illustrates every gap worth probing. Send the same questions to whoever you are buying from. The answers belong in the file, and the ones that come back as adjectives rather than numbers are the ones that will cost you later.
Ask for fill rates, not for a feature list
The header on the Whale Alpha feed at capture said the module covers every wallet movement above your threshold across EVM, Solana, BTC and Polymarket, with tagged cohorts spanning VC, OG, smart-money and insiders. That is four chains and four label classes advertised in two sentences. Now read the table underneath. The columns are Whale, Market, Dir, Amount, Platform, Entry, Current, PnL, Win Rate, Smart Score and Time. Every row rendered in that capture carried a Platform value of PM, and no row carried a cohort tag of any kind.
That gap is not an accusation, it is the question. Which chains are populated today, which are roadmap, and where does a cohort label surface if one exists? Generalise it and you have the single most useful diligence metric available: for every claimed field, show me the column, then show me the fill rate on that column, broken out by chain, over the last ninety days. A field that exists in the schema and is null on ninety percent of rows cannot carry a rule. Fill rate costs the vendor one query and it tells you more than any capability matrix.
Latency is a distribution and you need it per chain
The engine header printed LATENCY P50 at roughly twelve seconds, alongside a claim of sub-block latency on the major chains. Those two statements are close to the same statement, since twelve seconds is about one Ethereum block, and neither of them tells you anything about the half of the events that are slower than the median.
A median is the number a vendor volunteers. Ask instead for P50, P90, P99 and the maximum, each broken out by chain and by venue, over a stated window. Then ask the question that most buyers skip, which is what the clock is anchored to. Latency measured from block timestamp to internal ingest is a different number from block timestamp to the moment the record is available on the public API, and the second one is the only one you can trade against. Get the definition in writing, because the definition is where the number comes from.
The reason the tail matters more than the median is structural. Detection systems degrade under exactly the conditions that produce the events you care about, which is chain congestion, venue load and volatility. A feed with a twelve second median and a four minute P99 is not a twelve second feed for the purposes of anything you will actually do with it.
Where labels come from and who is allowed to change them
The Smart Score column in the capture read 45 on most rows, with a 70 and a 38 among them. A score that holds one value across the bulk of the sample is behaving like a default, and it may well be a default. That is worth asking about directly rather than assuming, and the question generalises to every derived field a vendor ships.
For each label or score, ask four things. What are the inputs, stated as a list. How often does the definition change, and is it versioned. When it changes, does history get recomputed under the new definition or preserved under the old one. And what is the null convention, because the Win Rate column on that same capture read 0.0% on several rows, and a zero that means no resolved trades yet is a completely different object from a zero that means lost everything. If the vendor cannot tell you which, you cannot filter on that column.
Restatement and the as-of guarantee
Rows in this feed carry Entry, Current and PnL. Current and PnL move as the market moves, which means the row you read at one moment is not the row that sits in storage an hour later. That is reasonable behaviour for a live surface and it is fatal for a research archive if nobody has drawn the line between them.
So ask the archive questions separately from the feed questions. Which fields are immutable once written. Is there an as-of query, meaning can you reconstruct the exact row set a user would have seen at a past timestamp, or does the archive only hold current state. When a record is corrected, is the original retained with a correction record, or overwritten. What is the notification path for a restatement, and what is the historical restatement rate by field.
The practical consequence is that a backtest built on a feed with no as-of guarantee is a backtest on data that partly did not exist at the time. Nobody can tell you how much of the result that accounts for, which is precisely the problem. Note that the DISPLAYED readout on the feed tab printed a dash rather than a row count at capture, so even the size of the current result set was not established on screen. If the live surface will not tell you how many rows you are looking at, do not assume the archive knows either.
Make the aggregates reconcile before you sign
Here is the test I would run first, because it is quick and it is diagnostic. Take every headline aggregate the product publishes for the same population and check whether they agree.
On this module at capture the feed tile reported an average whale win rate of 12.1%. The Statistics tab reported an average win rate of 44.3% across 26,687 tracked wallets. The Finder header reported a filtered average win rate of 91.1%. Three win rates, one module, one afternoon. Each of those may be perfectly correct over its own population and window, and that is exactly the point: the numbers are only usable once someone writes down which population and which window each one covers.
So the request is simple and it is a fair one. For each published aggregate, give me the population definition, the time window, the filter state it reflects, and the SQL-equivalent description of how it is computed. Then reconcile two of them for me on a sample I choose. A vendor who can do that on a call has a data model. A vendor who cannot has a dashboard.
Structuring the trial so it produces evidence
Run the trial against a written scoring sheet or it will turn into a vibe. Five things to score, each with a pass threshold agreed before the trial starts.
- Coverage, measured as fill rate per claimed field per chain, against your own list of wallets you already know are active. Supplying your own known set is what turns this from their demo into your test.
- Latency, measured end to end on your infrastructure, from an independently observed on-chain event to the record being readable through the API. Record the full distribution, not the mean.
- Completeness, measured as recall against a truth set you build yourself for one chain over one week. Expensive, and the only way to find silent gaps.
- Stability, measured as the count of schema changes, definition changes and unannounced restatements over the trial window.
- Reconciliation, measured as whether the aggregates agreed with the rows underneath them, on your sample, without vendor assistance.
Then attach the answers to the contract. Coverage claims become representations, latency becomes a service level with a defined measurement point, and restatement gets a notification obligation with a stated window. This is ordinary commercial hygiene and it is skipped constantly in data buying, usually because the demo scrolled beautifully and nobody wanted to be the person asking about null conventions.
None of this requires believing a vendor is careless. Whale data is genuinely hard, chains behave differently, venues change their APIs without warning, and every provider in this space is making judgement calls under real constraints. The point of the questionnaire is not to catch anyone out. It is to make those judgement calls visible, so that when a position goes wrong and somebody asks what the data actually guaranteed, the answer is a document rather than a memory of a good meeting.