Nobody gets asked about their ingest layer until a number is challenged. Then the question arrives in a specific form. Someone points at a chart showing institutional ownership of a name rising sharply and asks whether that is real money moving or an artefact. If the answer is a shrug, the chart stops being usable, and so does everything else built on the same pipeline.
The decisions that determine the answer all get made early, in the unglamorous part of the build, and most of them are irreversible in practice because reprocessing history is expensive. What follows is the set that actually matters, in the order they bite.
The unit of analysis is the filer, not the filing
The first design error is treating a downloaded document as a record. A quarterly report from one manager is often several documents, and one document often covers several managers. Build the universe on filings and every count you produce will be a count of paperwork.
Three document types have to be handled differently before anything else happens. A holdings report carries an information table. A notice carries no holdings at all and exists to say that the manager's positions are reported by somebody else. A combination report carries some holdings directly and points elsewhere for the rest. Ingesting notices as if they were empty holdings reports is the classic way to manufacture a fleet of phantom managers who appear to have liquidated everything, and it produces a beautiful, entirely fictional wave of institutional selling.
The cover page also carries an other-included-managers table, with a sequence number per manager, and the holding rows reference those sequence numbers. That reference is the only thing telling you which entity inside a filer family actually holds a given line. If you drop it, you have collapsed a complex to a single blob and you can no longer answer the question that gets asked most often, which is whether two seemingly independent holders are in fact the same shop.
A CIK is an identity, not an entity
The registrant identifier is stable per registrant, which is not the same as stable per manager. Firms register new entities, retire old ones, reorganise, merge, and spin out. A single economic manager can therefore hold several identifiers across time, and one identifier can change hands. Keying your history on the raw identifier gives you a manager whose assets fall to zero and a new manager born the same quarter with a suspiciously similar book.
The fix is a firm key sitting above the identifier, with an effective-dated mapping. Building that mapping is manual work, and it should be. Automated name matching will merge two unrelated advisors with similar names and will fail to merge an entity that rebranded, and both errors are the kind that survive review because nobody looks at the join.
Three signals do most of the linking work without requiring judgement calls. Shared business address across filings is strong. Appearance in each other's other-included-managers table is stronger, since that is the filer telling you directly that the entities are related. Continuity of the holdings themselves across a boundary, where a new identifier's first report closely resembles an old identifier's last, is the strongest of the three and also the one that catches successor entities that share neither name nor address.
Amendments, and which row wins
Amendments are where most pipelines quietly go wrong, because handling them naively looks like it works.
An amendment cover page declares what kind of amendment it is, and the two kinds require opposite handling. A restatement replaces the original information table in full, so the original rows must be retired entirely and the amended rows stand alone. A new-holdings amendment adds rows to the original, so the original rows stay and the amended rows are unioned on. Apply restatement logic to a new-holdings amendment and you delete real positions. Apply union logic to a restatement and you double the ones that appear in both.
Two rules keep this straight. First, never overwrite. Store every document immutably with its accession identifier and derive a current view by supersession rules, so you can always reproduce what your system believed on a given day. That reproducibility is the difference between a defensible number and an anecdote, because a number that has silently changed since you published it cannot be defended even when it is now more accurate. Second, treat the deadline as a soft boundary, not a hard one. Amendments arrive months late, sometimes years late, and a nightly job that only looks at the current quarter will never see them. Reprocess a trailing window on a schedule, and record when a historical figure changed and why.
Units, and the sanity checks that catch them
The value column is the single most reliable source of silent error, because the unit convention it uses has not been constant across the whole history. Rebuild a long series without checking, and you get a level shift of three orders of magnitude at the boundary that looks, on a chart, like a structural break in institutional participation. Anyone can see the break. Almost nobody checks whether it is a units problem before writing a paragraph explaining it.
The share and principal column has the same shape of problem. A row can be denominated in shares or in principal amount, and the column that distinguishes them is a separate flag. Sum without reading the flag and a bond line contributes its face value to a share count. Option rows carry a put or call designation, and adding puts to a long share total produces an aggregate pointing in the wrong direction for exactly the positions where direction matters most.
The check that catches nearly all of this is crude and effective. Compute aggregate reported notional per quarter and look at it against the market capitalisation of the reportable universe. The relationship should be plausible and should move smoothly. When it is not plausible, stop. As an illustration of what implausible looks like in the wild, the Insider Alpha performance panel at capture time showed twenty thousand scored filings against a notional of one thousand and eighty-nine trillion dollars. Whatever that figure is measuring, it is not dollars of exposure, and it is precisely the sort of reading that a magnitude check should catch before anyone quotes it.
What defensible actually means here
Defensible is not the same as correct. You will never be fully correct, because the source data contains genuine errors that the filers themselves never amend, and because judgement calls in the firm-key mapping are judgement calls. Defensible means that for any published figure you can state the ingest date, the supersession state, the exclusion list, and the coverage rate, and that a colleague running your pipeline against the same inputs gets the same output.
The practical form of that is a per-quarter manifest, stored beside the data, recording documents seen, documents parsed, rows rejected with reasons, unmapped identifiers by reported value, and amendment supersessions applied. When somebody challenges the chart, you open the manifest. That is the whole answer, and it takes about a minute. Without it the answer takes a week and is unconvincing at the end of it, which is why this layer is worth building properly before anyone builds a signal on top of it.