I keep running into the same trap when people build composite scores. They wire up four or five components, sum them, normalize to something that lands between 0 and 100, and then treat the output like it means something. A stock reads 82. Great. Compared to what? An 82 is only useful if you can say that historically, when this thing read 82, the forward return over the next few weeks was positive more often than not, and you can put a rough number on how often. Otherwise you have built a random number generator with a confident font.
The mapping step and the validation step are two different jobs, and most people only do the first one. So this is about doing both, and doing the second one honestly enough that you would trust the score with your own money.
Getting the raw sum onto a 0 to 100 range
The first decision is how you turn a pile of component values into a single bounded number, and there are really only a few sane ways to do it. The one I reach for most is percentile ranking against your own history. You take each component, rank it against its own past distribution, and the score becomes where today sits relative to everything that came before. A reading in the 82nd percentile means roughly four out of five historical days looked worse on that component. That is already more honest than a raw sum, because it is expressed in a unit a human can reason about.
The alternative is a fixed linear map, where you decide the theoretical floor and ceiling of your raw sum and stretch it across 0 to 100. This is cleaner to explain but it lies to you the moment your components drift. If one input rarely fills its range in practice, half your scale sits empty and every real reading clusters in a narrow band. You end up with a scale that says 40 to 70 and never anything else, and now your 82 is impossible and your 55 is meaningless.
A couple of things I have learned to check before trusting either method:
- Look at the actual distribution of your scores over a long window. If ninety percent of readings fall between 45 and 65, your scale has no resolution where it counts and you should rethink the mapping, not the threshold.
- Watch for components that move together. If three of your five inputs are basically measuring the same thing, your sum is secretly weighting that thing triple, and the score will swing hard on one underlying signal while pretending to be diversified.
- Decide whether you want the score to be relative to the asset's own history or relative to a cross-section of peers. These answer different questions and the same 82 means different things under each.
The part everyone skips: does a high score actually precede the outcome
Here is where the work is. You have a score. Now you need to show that the number tracks reality. The tool for this is a reliability plot, borrowed from weather forecasting, and it is embarrassingly simple to build.
Take every historical reading, bucket the scores into bands, say 0 to 10, 10 to 20, and so on up to 100. For each band, look at what happened next. Pick a forward window that matches how you would actually trade the score, maybe the next two weeks, and measure the outcome you care about, most simply whether the forward return was positive. Now for each band you have two numbers: the average score in that band, and the fraction of times the outcome actually occurred. Plot them against each other. If your score is calibrated, the points sit near the diagonal. A band centered around 80 should see the good outcome roughly 80 percent of the time. When the curve sags below the diagonal at the high end, your high scores are overconfident and the number is promising more than it delivers.
Most first attempts sag. That is normal and it is the whole reason to do this. A raw composite almost never comes out calibrated on the first pass, because summing components has no reason to produce probabilities. What you get instead is a monotonic-ish relationship that needs to be squashed or stretched into the right shape. You can fix this by fitting a simple mapping from raw score to observed frequency, isotonic regression or a plain logistic fit both work, and then the recalibrated score reads as an honest probability.
Forward-return buckets as the sanity check
Alongside the reliability plot I always build a plain table of forward-return buckets, because it catches things the calibration curve hides. Group readings by score band and, for each band, compute the average forward return, the hit rate, and the spread of outcomes. What you are hunting for is monotonicity. As the score climbs, the average forward return should climb too, or at least not fall. If your 90-plus bucket underperforms your 70-plus bucket, something is broken, and no amount of calibration curve smoothing will save a score whose extreme readings do worse than its middle ones.
The spread column matters as much as the average. A high bucket with a great average return but enormous variance is telling you the score finds situations that pay off big or blow up, not situations that reliably drift your way. That is a different animal and you should size positions accordingly, or not at all.
One failure mode to watch for specifically. If you tune the mapping and the calibration on the same data you validate on, you will get a beautiful diagonal and it will mean nothing. Split your history. Fit the recalibration on one stretch, check the reliability plot and the return buckets on a stretch the fitting never saw. If the calibration holds out of sample, you have something. If it falls apart, your score was fitting noise and you just caught it before it cost you.
When we built the market scorecards inside Blockcircle, this out-of-sample check is what separated the readings we were willing to publish from the ones that looked good in a notebook and nowhere else. The version that ships is the one where an 80 has earned the right to be called an 80.
The payoff is small and specific. Instead of a number that feels precise, you get a number attached to a rough frequency you can actually defend. When someone asks what an 82 means, the answer is no longer a shrug and a color gradient. It is a range of forward outcomes you have measured, with an honest note about how wide the spread is and how often the pleasant case actually showed up. That is worth more than any extra component you could bolt on.