Member-only story
How AI Models Improve Without Simply Getting Bigger
How data, architecture, compute, feedback, context, and reasoning work together to make each generation more capable.
For a while, I had a fairly simple theory about why a newer AI model ends up better than the one that came before it. I assumed progress mostly came from giving the model more data, increasing the number of parameters and spending more compute on training. The model, in my head, was basically a bucket, and improvement depended on how much you could pour into it.
That explanation seemed reasonable when I looked back at the early scaling race. OpenAI’s GPT-3 arrived with 175 billion parameters, and Google’s PaLM later arrived with 540 billion. From the outside, the pattern seemed straightforward enough that I barely questioned it. Companies were building larger models, training them on enormous amounts of text and getting more capable systems in return, so it was easy to imagine that the same pattern would continue indefinitely.
As newer generations arrived, though, parameter count became a much less useful way to understand what had improved. Some labs stopped publishing parameter counts altogether, smaller models sometimes performed surprisingly well against much larger ones, and new models began improving in…
