The first version of my macro scorecard was a spreadsheet I updated by hand every Sunday. It worked right up until the Sunday I forgot. Then a liquidity composite I was actually leaning on was quoting a number from two weeks earlier, and I had no idea, because a stale value looks exactly like a fresh one on a dashboard. That is the whole problem in one sentence. A composite is only as current as its worst-updated series, and nothing on the screen tells you which series that is.
So the goal here is boring and specific. Pull the underlying series from their official sources on a schedule, know when each one actually updated, and refuse to trust a composite when one of its inputs has gone quiet. Most of the work is in that last part, which is the part everyone skips.
Where the data actually lives
You mostly do not need scrapers. The big sources publish clean APIs and the coverage is wide enough that you can build almost any liquidity or risk composite from three or four of them.
- FRED (the St. Louis Fed) is the workhorse. Free API key, one endpoint per series, JSON out. It also re-hosts a huge amount of international data, so a lot of what you want from other central banks is already sitting there under a FRED series ID, which is convenient but has a catch I will get to.
- ECB publishes through its Data Portal API. Series are addressed by a structured key rather than a single ID, so you specify the dataflow and the dimensions. Once you have the key format down it is very stable.
- BOJ is the awkward one. Historically a lot of its series came out as spreadsheet downloads rather than a clean REST feed, so for Japan I often lean on the FRED-hosted mirror for the daily stuff and only go to the BOJ directly for the series that FRED does not carry.
The catch with using FRED as a proxy for everyone: FRED updates on its own cadence, which can lag the primary source by a release cycle. For anything where a one-day delay matters, hit the original. For a slow monthly series, the mirror is fine. Decide that per series, not once for the whole pipeline.
The release calendar is the actual schedule
The naive approach is to refresh everything on a fixed timer, say every morning at the same hour. That mostly works and it is what I would start with. But it wastes calls on series that only move once a month, and worse, it can make you feel current when you are not. If a monthly release is late, your daily poll keeps returning the same old value and everything looks healthy.
The fix is to treat the release calendar as a first-class input. FRED exposes release dates, so for each series you can know roughly when the next print is due. Then your scheduler does two things instead of one. It refreshes on a cadence that matches the series, and it flags any series whose expected release date has passed without the value changing. That second flag is what would have saved my Sunday. A late release should be loud, not invisible.
A rule of thumb I use for cadence:
- Daily-ish market series (rates, some balance-sheet lines): poll a few times a day, off the top of the hour to be polite to the API.
- Weekly series: poll daily, because the exact release time drifts.
- Monthly and quarterly series: poll daily in the window around the expected release, then back off once the new value lands.
You are not trying to catch the data the instant it drops. You are trying to guarantee you never quietly miss it.
Revisions will embarrass you if you let them
Here is the thing that trips people up who come from price data, where a printed candle is final. Macro series get revised. A GDP or employment figure you pulled last month can be a different number today, for the same date. FRED handles this through what they call vintages, which is a fancy word for the version of the series as it existed on a given day.
Why you care: if your backtest reads today's revised value for a historical date, you are testing a strategy on data that did not exist yet at that point in time. That is lookahead bias, and it flatters your results for free. For anything historical, pull the vintage as it stood on the date you are simulating. For a live dashboard the current value is what you want, so you can use the latest. Just do not mix the two and expect the backtest to mean anything.
Practically, I store every pull with the date I pulled it, not just the series date. Storage is cheap and it lets me reconstruct what the composite would have said on any past day, which is the only honest way to check whether the thing ever actually worked.
Making the composite refuse to lie
The scheduling is the easy half. The half that earns its keep is a freshness gate on the composite itself. Before I compute a liquidity or risk score, I check each input against a staleness budget, meaning how old is too old for that particular series. A daily series more than two business days stale is a problem. A monthly series is allowed to sit for weeks. If any input blows its budget, the composite does not render a confident number. It renders a warning and shows exactly which series is holding it up.
Concretely, the checklist each series carries with it:
- Last observation date, the date on the newest data point.
- Last fetch date, when I actually pulled it, so I can tell a dead pipeline from a quiet series.
- Expected next release, from the calendar, so a late print gets flagged rather than ignored.
- Staleness budget, the max age I will tolerate before I stop trusting it.
When all four line up, the composite is live and I believe it. When one is off, I want the dashboard to be honest about it, even if that means it is uglier. An ugly honest number beats a clean stale one every time, because the clean stale one is the one that costs you.
This is the discipline we bake into the market scorecards on Blockcircle for the same reason. A macro read that silently drifts is worse than no read, because you act on it with false confidence. So the pipeline's real job is not fetching data. Plenty of tools fetch data. Its job is to know, at every moment, whether the number on the screen is one you are allowed to trust.
Start with FRED and a daily timer, get the vintages stored from day one, and add the freshness gate before you ever put a composite in front of a decision. The scheduling you can improve later. The moment where a stale input passes itself off as current is the one that actually costs money, so close that door first.