Inference Thermodynamics
A Framework for Reasoning About the Energy Cost of AI Inference
Taalas dropped out of stealth on February 19 with a chip that runs Llama 3.1 8B at 17,000 tokens per second. The weights of the model are hardwired into silicon. No High Bandwidth Memory (HBM), no memory bus. Single 815mm² die, TSMC 6nm, 250 watts. 10 times faster than anything else at a fraction of the power.
The catch: the chip is the model. Need a new model? You need a new chip. Two-month fabrication turnaround.
So how do you compare this against a GPU? Tokens per second doesn’t cut it. Cost per token doesn’t either. When one architecture uses 3-bit quantization and another runs FP16, you need a unit that captures speed, power, and whether the outputs are actually right.
This is my attempt at that framework. Five relationships, sourced power specs, and cross-platform comparisons. The framework is not perfect. But the core finding holds: current inference hardware operates 10¹⁰ times above the thermodynamic floor, and the biggest energy tax in the stack is the one compute-in-memory eliminates.
To put that in perspective: ENIAC, the first programmable general purpose computer in 1945, consumed ~150,000 watts to do a few thousand operations per second. A modern CPU does ~10¹⁰ operations per second per watt. That’s roughly the same gap. Current inference hardware is at the vacuum tube stage of energy efficiency.
To be absolutely transparent, I’m not a hardware specialist and used Claude heavily to explore, analyze and write about this concept. The results are outlined below.
The Inference Unit That Matters
A single token isn’t true or false. “The capital of France is” → “Paris” is four tokens where truth belongs to the completed inference, not any individual token.
So the base unit is joules per useful inference. A complete generation that actually answers the question correctly. Everything derives from that.
Three variables feed it: speed (tokens per second), power (watts), and accuracy (what fraction of inferences are useful). One more variable handles scale: how many concurrent inference streams the hardware supports.
The Five Equations
Variable List
The Equations
Equation I — Inference Energy (definition)
This is just dimensional analysis, moving variables around. Power divided by the rate of useful inference production. `N` inference streams, each producing `S/T` inferences per second, fraction `A` correct. Everything else builds off this.
Equation II — Speed Limit
Max throughput = compute density × die area, divided by operations per token. `2PQ` approximates the multiply-accumulate workload per token.
This holds for decode-dominated, short-context inference where weight multiplies dominate. It ignores attention (which scales `O(T·d)` per token) and treats `κ` as constant across architectures. Back-of-envelope, not gospel.
Equation III — Landauer Energy Floor
The thermodynamic minimum. Every irreversible bit operation costs at least:
An 8 billion parameter model at 3-bit Q generates 200 tokens: that’s about 57 nanojoules. Same `2PQ` approximation as above.
Interesting comparison for you: 160 nJ is the energy of a flying mosquito.
Equation IV — Density-Power Tradeoff
Power = total ops/sec × energy per op. `ε(D)` increases with die area. The longer wires on the chip, the more capacitance you have, the more clock distribution overhead you have. There’s no free lunch, even in AI.
Equation V — Accuracy-Capacity Bound
Accuracy bounded by model capacity (`P × Q`) and reasoning budget (`T`).
This is the weakest part of the framework. Real scaling laws describe loss as a power law, not accuracy as a log function. “Accuracy” itself is ill-defined as a single scalar: MMLU, HumanEval, and real-world task completion are different numbers. The coefficients `α` and `β` are hand-calibrated to two data points.
What matters is that accuracy generally improves sublinearly with model size. There are diminishing returns. The exact functional form matters less than what falls out of it (next section).
What Falls Out
Efficiency Ratio
Combining Relationships I and III yields η:
How far above the thermodynamic minimum a given system is. This is not Carnot efficiency — there’s no reversible cycle argument here. It’s just a useful ratio.
The 10 ¹¹ gap represents an enormous engineering opportunity. Even closing a fraction of this gap yields transformative cost reductions.
Thermodynamically Optimal Model Size
Relationship V, despite its imprecision, creates an important qualitative tension with Relationship III. Any sublinear accuracy curve + linear energy cost = a peak in accuracy-per-joule at some finite model size. You can explore that tension with:
The framework predicts the peak exists and shifts right with better hardware. Where exactly it falls depends on Relationship V’s calibration, which I already flagged as rough. The qualitative insight: there is an optimum and better hardware pushes that optimum toward larger models.
Better hardware doesn’t just make the same model cheaper. It makes bigger models the rational choice.
The Energy Ladder
The biggest gap is between “No Data Movement” and GPU baselines. That’s the memory wall tax . That’s energy spent shuttling weights from HBM to compute dwarfing the consumption of everything else.
Taalas eliminates this energy tax entirely by putting weights in mask ROM. The remaining gap between Taalas and the theoretical “no data movement” level is system overhead: clock distribution, voltage regulation, I/O, host CPU, power supply losses. You get that loss in any computational system.
Platform Analysis
Data Sources and Methodology
Power specs from official docs, datasheets, and verified reporting. Throughput from Artificial Analysis and manufacturer claims. All Llama 3.1 8B, single-user output generation.
Table 1. Data sources for platform specifications.
Methodology Notes
Groq: Each GroqChip has 230MB SRAM. Llama 8B at INT8 is ~8GB. That’s ~35 cards. Per-card power is 275W, but system power for one model instance is ~9.6kW. Charts use per-card for silicon-level comparison, but the real system cost is much higher. Groq self-reports 1–3 J/tok for their cloud service which likely reflects heavy batching.
Get James Barney’s stories in your inbox
Join Medium for free to get updates from this writer.
Cerebras: CS-3 is 23kW for a system designed to run 70B–1T+ models. Using 23kW for an 8B model comparison is like measuring a semi truck’s fuel efficiency delivering a single envelope. Unfair to Cerebras. Included for completeness, flagged here.
Taalas: The 250W per-card figure and 17K tok/s are both first-party claims verified by EE Times reporting. The entire 8B model resides on a single 815mm² chip (TSMC 6nm, 53B transistors). The “per user” qualification suggests single-stream operation. 10 cards in a 2.5kW server provide 10 concurrent user streams.
Cross-Platform Comparison
Table 2. Cross-platform comparison. Groq uses per-card power (275W); system power for full 8B model is ~9.6kW across ~35 cards. Cerebras uses full 23kW system power. See methodology notes.
Joules per Token — Alternative Metric
Joules per token provides a simpler metric that avoids assumptions about accuracy and inference length. Taalas achieves 0.015 J/tok which is dramatically lower than all competitors. Groq self-reports 1–3 J/tok for their cloud service, while NVIDIA H100-class systems typically consume 10–30 J/tok at system level.
The Real Trade Off
Taalas 8B sits in the bottom-left: cheap but lower accuracy from 3-bit quantization. B200 running 405B is top-right: high accuracy, high energy. The frontier connecting them is the real tradeoff curve — and everything off that frontier is leaving performance on the table.
The more interesting question: what happens to that frontier when hardware gets better?
Each hardware generation slides the whole curve left. GPU 405B at ~20 kJ. Taalas-class 405B at ~1.5 kJ. Future CIM at ~0.05 kJ. Same model, same accuracy (roughly), orders of magnitude less energy.
But here’s the punchline: on better hardware, the slope of the curve also changes. The marginal energy cost of jumping from 70B to 405B shrinks dramatically. On GPUs, going from 70B to 405B costs ~20 times more energy for ~10% more accuracy. On projected future CIM hardware, that same jump costs ~3 times more energy. Bigger models become rational faster on better hardware.
Hardware and model scaling aren’t independent axes. They’re coupled through energy.
Where This Breaks Down
The Taalas numbers are real for 8B models. But there’s a ceiling, and it’s physical.
An 8B model at 3-bit quantization is ~3GB of weight data. Fits on one 815mm² die — which is already pushing the reticle limit (the maximum area a lithography stepper can expose in one shot). There’s no headroom to grow on a single chip.
Scale up and the math shows how this process begins to fall apart:
The whole advantage , no data movement, degrades as you cross chip boundaries. A 70B model split across 16 dies reintroduces the interconnect penalty that Taalas exists to eliminate. It’s smaller than GPU-to-HBM bandwidth (chip-to-chip on a module is faster than off-package memory), but it’s not zero. The single-die purity is gone.
Other physical constraints:
- Yield: Larger dies have exponentially worse yield. A single defect kills the chip. Mask ROM weights can’t be remapped around defects the way DRAM uses redundant rows. Either they’ve solved this or their yield rate is a meaningful cost factor they haven’t disclosed.
- Model lock-in: Taalas CEO Ljubisa Bajic told EE Times customers should expect a one-year commitment per chip. If a better model drops six months in, your silicon is obsolete. The two-month turnaround helps but doesn’t eliminate the risk.
- Quantization quality: Gen 1 uses custom 3 or 6-bit. Gen 2 moves to standard FP4. But even FP4 has measurable quality loss on reasoning-heavy benchmarks. The accuracy axis in our framework isn’t free.
The strong counterargument is that the models worth deploying keep getting smaller. Distillation, mixture-of-experts (where individual experts are small), and better quantization methods (BitNet, QuIP#) all push useful capability into the single-die range. If a 2-bit quantized 70B model is ~17GB, that’s back to feasible on a few dies. If mixture-of-experts means you only activate 8B of parameters per token, you’re back in the sweet spot entirely.
Taalas is betting that production inference workloads converge on the 8B–70B range at aggressive quantization. I don’t think that’s a crazy bet.
Path to Theoretical Limits
On GPUs, 95% of the energy moves data. It’s not computing anything — just moves. Taalas eliminates that layer entirely, which is why it’s 20× more efficient despite running on an older process node. But look at what’s left: system overhead (yellow) is now almost the entire bar. Clocks, voltage regulation, I/O, power supply losses. That’s the next frontier.
~33 doublings between here and Landauer. Just doubling 10 of them and inference energy becomes trivial. We’re still in the ENIAC era of AI.
Implications
- Memory wall is the biggest tax. The dominant gap in the ladder. Compute-in-memory eliminates it. Taalas is the extreme case — mask ROM weights — but the principle applies to any architecture that keeps weights close to compute.
- Quantization is a thermodynamic lever. Lower `Q` improves speed (II), floor (III), and power (IV) simultaneously, at modest accuracy cost (V). 1-bit methods (BitNet) could move the whole framework an order of magnitude.
- System power swamps chip power. Groq at 275W/card looks great until you need 35 cards. Cerebras at 23kW looks terrible until you remember it’s built for 70B+ models. Always ask: power of *what*?
- `η` as a universal benchmark. Ratio of actual to Landauer energy. Architecture-independent. Track it over time the way we track transistor density. It’s just a ratio, not a deep theorem — but it’s the right ratio.
Conclusion
Five relationships. One framework. Current hardware is 10¹⁰ to 10¹¹ times above the theoretical floor. The biggest tax is data movement, and it’s eliminable. An optimal model size exists for every hardware efficiency level, and it shifts right as hardware improves. Model compression and hardware specialization are convergent, not competing.
Sources
- Taalas. “The path to ubiquitous AI.” taalas.com, Feb 2026.
- EE Times. “Taalas Specializes to Extremes for Extraordinary Token Speed.” Feb 2026.
- The Next Platform. “Taalas Etches AI Models Onto Transistors.” Feb 2026.
- Kaitchup. “Taalas HC1: Absurdly Fast Per-User Inference.” Substack, Mar 2026.
- NVIDIA. “H200 Tensor Core GPU.” nvidia.com.
- WCCFTech. “NVIDIA Blackwell B200 GPUs Consume Up To 1200W.” Mar 2024.
- Groq. “Inside the LPU: Deconstructing Groq Speed.” groq.com.
- Mouser Electronics. GroqCard product listing. TDP: 275W.
- Cerebras. “CS-3 vs NVIDIA B200.” cerebras.ai, Mar 2025.
- Artificial Analysis. Llama 3.1 8B provider benchmarks. artificialanalysis.ai.
- Zheng Li / Medium. “Groq’s Deterministic Architecture is Rewriting the Physics of AI Inference.” Dec 2025.
- Landauer, R. [“Irreversibility and Heat Generation in the Computing Process.” IBM J. Res. Dev., 1961.
Developed through collaborative exploration with Claude (Anthropic), building from first-principles analysis of the Taalas HC1 announcement. Calibration constants (α, β, ε) are approximate and should be refined with empirical measurement. Power figures sourced from official documentation and verified reporting as of March 2026.
