Get started

Horizon-Matched Scoring for Analysts Who Never State a Horizon

A directional call with no date attached cannot be graded, only rationalised. Fixing one house evaluation horizon per asset class in advance is what turns a wall of undated opinions into a scoreable record.

Horizon-Matched Scoring for Analysts Who Never State a Horizon

The hardest problem in scoring external analysts is not measuring accuracy. It is that most of what you want to score was never expressed in a form that admits a right or wrong answer. A direction with no date attached is not a forecast, it is a disposition, and a disposition can be defended indefinitely because there is always a longer window in which it might yet be vindicated.

Every desk that has tried to build an analyst scorecard has hit this, and the usual outcome is a spreadsheet that dies after two quarters, because arguing about which date to mark each call on costs more than the exercise returns. The fix is not a better argument. It is removing the choice.

An undated call is a free option and it is written by you

Start with why this is a scoring problem rather than a data problem.

If a call carries no horizon, then whoever grades it chooses the mark date, and that choice is being made after the outcomes are visible. Grade it generously and you pick the favourable window. Grade it harshly and you pick the unfavourable one. Either way, the score is a function of the grader's selection and the analyst's contribution is confounded beyond recovery. Two people can score the identical call set and disagree by a wide margin without either doing anything improper.

The asymmetry is worse when the analyst controls the narrative. An undated bullish view survives any drawdown, because the thesis simply has not played out yet, and it is booked as correct the first time price passes the entry. That is a free option, exercisable at the holder's convenience, and the desk that fails to demand a horizon is the party writing it. None of this requires bad faith. It is the ordinary behaviour of an unconstrained evaluation rule, which is why the constraint has to be structural rather than cultural.

The activity window is not the evaluation horizon

Analyst Leaderboard with 24H, 7 days, 30 days, 90 days and all time selectors above a table of ranked analysts with no horizon column
The selector across the top offers 24H, 7 DAYS, 30 DAYS, 90 DAYS and ALL TIME, and the columns run from analyses through the quality measures to Active Days and Last Analysis. There is no horizon column anywhere in the table: that selector filters which calls are counted, not when they are judged, and treating the two as the same thing is the most common error in analyst scoring.

The capture makes the distinction concrete. The window control governs which analyses enter the view, and the table's columns describe activity and quality, with Active Days and Last Analysis at the right. Nothing on the row states how long each call was meant to be given.

Two different clocks are hiding under one control. The activity window answers who has been contributing lately. The evaluation horizon answers over what period each call is scored. Sliding an activity window from 30 days to 90 days changes the population, not how any individual call is marked. If your internal reporting quotes an accuracy figure alongside a window selector, be certain everyone reading it knows which clock the number refers to, because both will be described as the 90 day number.

Note what the quality columns are doing here: Avg MISP% and Avg EV/$10 are dashes on every row and the directional counts are all zero, against a header showing 4,244 analyses all time and seven active analysts in the last seven days. That is activity in aggregate without enough resolved directional calls in the window to populate the quality columns. Design the scoring process for that state, because it is the one you will be in for the first several quarters.

One horizon per asset class, fixed before you look

The workable rule is to stop trying to infer horizons and to impose them.

Publish a short table: each asset class or instrument type gets one primary evaluation horizon and, at most, one secondary. Every call captured for that class is scored at the primary horizon regardless of what its author intended, and the secondary is reported alongside rather than instead. The lengths themselves matter far less than that they were set in advance and applied uniformly, which is the property that makes the resulting numbers comparable across analysts.

Three rules keep it defensible. The horizon is set before the sample is scored, and any change applies prospectively only, with the old series preserved and clearly labelled. Where a call comes with an explicit horizon from the author, honour it and mark the record as author-dated, then report author-dated and house-dated results separately, since mixing them reintroduces the selection you removed. And where the instrument carries its own expiry, which covers most prediction market contracts, the resolution date is the horizon and no house convention is needed.

That last case is the one to build first. A board covering contract markets has a large population of calls with an unarguable mark date built into the instrument. Scoring those gives you a validated pipeline and an early read on which analysts are worth the manual effort of dating the rest.

Mark against the price that existed when the call was made

A fixed horizon fixes when to mark. It does not fix what to mark against, and the benchmark choice is where analyst scorecards most often flatter their subjects.

For a contract market the correct benchmark is not fifty percent, it is the contemporaneous market price. An analyst who says yes on a contract trading at 78 cents and is proved right has told you almost nothing, because the market already had it at 78. The scoreable quantity is whether their view improved on the price available at the moment they published, which is the same standard you would apply to an internal PM. Score against the market price at call time and a great deal of apparent skill evaporates, which is the intended effect.

For directional calls on continuously traded assets, mark the return over the fixed horizon against the same benchmark for every analyst, and state whether it is absolute, benchmark-relative or risk-adjusted. Mark at the horizon endpoint only. Using the maximum favourable excursion inside the window reintroduces exactly the ex post selection the fixed horizon eliminated, and scorecards include it more often than you would expect, under the heading of giving the call a fair chance.

Restatements need a stated rule too. A repeated view is either a new call with its own horizon or an extension of the existing one, and those produce very different records for a persistent view that is eventually correct. Pick one, put it in the methodology, and never decide it case by case.

Publish the uncertainty or do not publish the rank

The output of this process is not a league table. It is a set of estimates with sample sizes attached, and the sample sizes are small.

Report n alongside every score, and set a minimum n below which no rank is published at all. On a board where individual analysts have low single-digit call counts in a window, no ordering of those analysts carries information, and publishing one manufactures a hierarchy out of noise. An interval around each score does the same work more gently: when the intervals for ranks one through six overlap completely, the picture is self-explanatory in a way that a ranked list never is.

Report coverage too, meaning how many of an analyst's calls could be scored at all. Fifty published views of which nine were specific enough to grade is a nine-call record and a forty-one call gap, and the gap is itself a finding about how that analyst communicates. Selective gradeability is the quietest bias here, because the ungradeable calls leave the numerator and the denominator together.

What a fixed horizon incentivises, including the parts you will not like

Any published metric becomes a target, and this one has three predictable distortions worth designing against from the start.

Analysts who know the house horizon will write to it, which is mostly good, since it produces dated, specific calls. Volume gaming is next: if rank responds to activity, and on a thin board it will, the cheapest way up is more calls rather than better ones. Weight by resolved calls rather than published ones and that inverts. Third is market selection, where an analyst concentrates on contracts that are cheap to be right about. Benchmarking against the price at call time is the antidote, since an obvious market offers no room to improve on the price and earns no score.

Sit with the residual honestly. Even with a fixed horizon, a clean benchmark and published intervals, distinguishing a genuinely skilled forecaster from a lucky one takes more resolved calls than most boards will accumulate in a year. The scorecard's real function in that period is not selection. It is disqualification: it identifies the analysts whose records cannot support the confidence with which they are presented, and that is a smaller claim than a ranking but a considerably more reliable one.

WRITTEN BY
Blockcircle Team

Financial market intelligence covering crypto, stocks, commodities, precious metals, and prediction markets.