The AI Industry Has a $100 Billion Blind Spot. And It Ships With Every Model You Deploy.
Your benchmark scores are not lying to you. They are just not telling you the whole truth. There is a difference. And in production, that difference kills systems.
Every team I have ever seen ship an anomaly detection system does the same thing before they deploy.
They run the numbers. AUROC. F1. Accuracy. The dashboard turns green. Someone says “looks good.” The model ships.
And somewhere out in the real world, a system that scored perfectly on every test fires its alarm three minutes too late. The breach already happened. The regime already shifted. The window already closed.
The model was correct.
It was just late.
And here is the part that should disturb every engineer reading this: your evaluation framework had no way of knowing the difference.
The Metric You Trust Cannot See Time
AUROC measures how well your model separates positive cases from negative ones. It is elegant, interpretable, and almost universally used. It is also constitutively blind to one of the most operationally critical properties a detector can have.
When it became correct.
Consider this scenario. Two detectors. Same system. Same data.
- Both achieve AUROC = 1.0
- Detector A raises an alarm immediately after a regime change
- Detector B raises the same alarm significantly later
Under every standard evaluation framework in widespread use today, these two models are identical.
In a real industrial monitoring system, one of them saves you. The other one watches you burn and then files an accurate incident report.
This is not a hypothetical edge case. This is the default behavior of how the field evaluates temporal intelligence. And almost no one is talking about it.
I Watched It Happen
I was running a probabilistic cyber-attack regime detection project. We were building systems to detect structural shifts in network traffic before damage occurred. Switching state-space models. Variational inference. Real architecture, not toy problems.
Our models scored perfectly on every benchmark we ran.
They were still failing at the thing that mattered.
Not because they were wrong. Because the alarm was firing after the window had closed. Standard metrics could not see the gap between “correct” and “correct in time” because they were not designed to look for it. Every timestep was weighted equally. The model that detected at T+1 and the model that detected at T+60 looked identical in the evaluation report.
I searched the literature. Nothing measured this.
So I built something that did.
Introducing the HED Score
The Hiremath Early Detection (HED) Score is a measure-theoretic evaluation standard built for one purpose: to make timing a first-class property in model evaluation.
Instead of treating all timesteps equally, HED asks a different question entirely:
How early does the model place probability mass on the correct regime after a known change point?
It does this by applying exponential weighting after the onset of a structural change. Earlier correct responses receive higher weight. Delayed detections contribute less. Pre-event false positives are explicitly corrected for.
The result is a single scalar score that captures both detection quality and detection timing simultaneously.
The mathematics behind it are formally published:
- 📄 arXiv paper: 2604.04993
- 🗂️ Formal archive: DOI 10.5281/zenodo.19713081
- 💻 Interactive demo: Google Colab
What Changes When You Start Measuring Timing
1. Identical AUROC Does Not Mean Identical Performance
Two detectors. Both AUROC = 1.0. Evaluated under HED, they are not the same model. They never were. The early detector scores higher. The delayed detector scores lower. For the first time, you can see what your standard metrics were hiding.
2. Early Noisy Signals Can Be More Valuable Than Late Clean Ones
This is the finding that reordered my mental model of what “good” means in detection systems.
Get Prakul Hiremath’s stories in your inbox
Join Medium for free to get updates from this writer.
AUROC rewards clean separation. It prefers the detector that is precise and late over the detector that is noisy and early. In a lab, that makes sense. In a system where the cost of latency is operationally catastrophic, that preference is exactly backwards.
HED can prefer an early noisy signal over a late clean one. Because in real systems, acting at T+2 on 70% confidence is often worth more than acting at T+30 on 99% confidence.
3. Model Rankings Flip
This is the result that made me stop and rebuild my evaluation pipeline from scratch.
On real datasets, detectors that are statistically tied under AUROC are separated cleanly under HED. Rankings reverse. Models you would have deprioritized become the ones you should have shipped. Models you would have shipped become the ones you should have benched.
This is not a corner case. It appears consistently, across datasets, wherever detection latency carries operational weight.
This Is Not Just an Anomaly Detection Problem
Once you see the temporal blind spot, you cannot stop seeing it.
Cybersecurity: Intrusion detection systems evaluated on classification accuracy, blind to whether the alert fires before or after lateral movement completes.
Finance: Regime change detectors ranked by AUROC on historical data, with no measurement of whether the signal arrived before or after the drawdown.
Healthcare: Physiological early-warning systems evaluated on sensitivity and specificity, with zero formal accounting for detection latency against clinical intervention windows.
Infrastructure: Fault prediction models optimized for precision-recall, deployed into systems where a two-minute lag means the difference between a controlled shutdown and an uncontrolled failure.
Every field that builds systems which operate in time is using evaluation frameworks that cannot see time. The HED Score is already being extended into biological systems, physiological monitoring, and algae growth modeling. We are building toward a contributors summit to widen its application across every domain where this blind spot lives.
What HED Is Not
HED does not replace AUROC. This matters enough to state clearly.
AUROC measures ranking quality. HED measures temporal quality. They answer different questions. A system evaluated on both gives you a complete picture. A system evaluated on only one gives you half the truth.
The field has been shipping on half the truth for a long time.
The Question You Should Be Asking Before You Ship
Not: did the model get it right?
But: did the model get it right in time?
If timing matters in your system, your metric must reflect that. Right now, it almost certainly does not.
Try It
pip install hed-score- 📦 PyPI: hed-score
- 💻 GitHub + experiments: github.com/prakulhiremath/hed-score
- 📄 Paper: arXiv 2604.04993
- 🗂️ Archive: DOI 10.5281/zenodo.19713081
- 🔬 Colab demo: Run it now
The Closing Argument
Evaluation shapes what gets built. The metrics you use to measure intelligence determine what kind of intelligence you optimize for.
For decades, the field has been optimizing for correctness and ignoring timing. The systems we have built reflect that choice. They are correct. They are often late. And until now, we had no standard way of knowing the difference.
That changes today.
The HED Score is open source, formally published, and ready to use. If you work on systems where timing matters, run it alongside your existing metrics. See what it finds. The results will likely surprise you.
And if you are building in a domain where detection latency has consequences and want to contribute to extending this framework, reach out. This is the beginning of something larger.
Replace the bio and ending with this:
If your system operates in time and your evaluation framework does not, you are not measuring intelligence. You are measuring a photograph of it.
The HED Score is open source, formally published, and installable in thirty seconds. Run it alongside your existing metrics. See what it finds. If the rankings change, you will know why.
This is the beginning of something larger. We are building toward a contributors summit to extend this framework across every domain where the temporal blind spot lives. If you are working in biological systems, physiological monitoring, infrastructure, finance, or anywhere detection latency carries real consequences, this conversation belongs to you too.
Reach out. Let’s build this together.
Prakul Hiremath is a sequential ML researcher who builds evaluation standards for temporal intelligence. His work spans cybersecurity, financial regime detection, biological systems, and probabilistic modeling. He is the author of the HED Score framework and the PARD-SSM system for probabilistic cyber-attack regime detection.
📬 prakulhiremath03@gmail.com · 🌐 Medium · GitHub · arXiv
If this changed how you think about evaluation, share it with one engineer who ships models. That is how blind spots stop being blind spots.
