Member-only story
One Token Would Take 25000 Years if an Expert Mathematician Calculated by Hand
How much do you understand about AI? Look at these unbelievable numbers.
When you type your prompt into an AI chatbot, you get the answer in less than two seconds. The answer is usually to the point, well-structured, and seems like someone intelligent wrote it.
In those two seconds, the machines do an astonishing amount of work. How much work? What did the machine do? Didn’t it just understand the question and answer as we do?
Let’s look at what happens behind the scenes:
What is a token?
A token is a piece of a word; it is three-quarters of a word on average. The AI model always deals with these tokens and never with the words you write on the screen. Yes, you read right. It is not how it seems.
Facebook, or Meta, has an open model called Llama 3.1. Since it is open-weight, they have published many details of this model. Other companies like OpenAI, Anthropic, and Google do not share the detailed workings of their models. Anyway, we can get an idea of how things work from Llama 3.1 with 405 billion parameters.
What is a parameter?
Have you ever seen the mixers in a sound-recording studio? There are rows and rows of sliders that sound engineers adjust to get the sound quality they want before starting the recording.
Similarly, inside every AI model, there are sliders you can adjust to get an acceptable quality of a written answer. The engineers call these parameters. Llama 3.1 has 405 billion sliders that are adjusted during training to create the output that we get.
What does it take to create a token?
Producing a token is a process that involves all the sliders. At the level of every slider, the text travels with two calculations altering it per token.
With 405 billion sliders, the total number of times the calculations are made is 810 billion.
810 billion calculations for one token.
If an expert mathematician calculated by hand at one calculation per second, without taking a break, 24x7, he would be at work for 25000 year to produce one token.
What does it take for one chat response?
A chat response is usually 500 words or 700 tokens. That adds up to 570 trillion calculations.
One mathematician at one calculation per second would complete 3 billion calculations in a century. To manually calculate the full chat response, you need:
180,000 mathematicians, each calculating for 100 years without a break.
That’s crazy, right?
But that is what is happening behind the scenes.
Those two seconds represent centuries of calculations. But the GPUs compress all that work into seconds.
What does it take to train a large language model?
The real work in this whole process is the training. During training, the model runs, learning to guess right on text where one part of the sentence, or a word, is hidden. Guess, check, nudge the sliders. The model adjusts its sliders until the guesses improve.
For Llama 3.1 405B, it took 31 million hours of computing time on H100 chips. In involved 3.8 ×10²⁵ calculations.
- 30.84 million chip-hours is 3,500 years of one chip.
- Done by hand at one calculation per second, it would take 1.2 quintillion years, or 87 million times the age of the universe.
Facebook engineers gave the task to sixteen thousand chips, running in parallel, and it still took months to train the model.
Things happen at the speed of light
The speed you experience while using AI is a combination of three things:
1. Each chip is super fast. A modern AI chip performs hundreds of trillions of simple calculations per second. Meta’s training run puts the calculation rate at around 400 trillion per second per chip.
2. Thousands of chips at the same time. Sixteen thousand chips at 400 trillion calculations a second each adds up to about 6.5 quintillion calculations every second.
3. The chips are designed for it. The arithmetic inside a neural network is simple multiply-and-add, repeated across enormous grids of numbers. GPUs or Chips designed for graphics are excellent at that kind of work.
A roomful of machines performs calculations that human beings would complete in hundreds of thousands of years. It takes a second or two to complete the calculations. In real chats, chips are shared among millions of users, and it takes a little longer — the seconds you see ‘thinking’ on the screen.
It is not thinking; it is completing the calculations.
What is a mixture-of-experts?
We have been talking about large dense models so far. In these models, all the sliders join in for calculating every token.
A mixture-of-experts model splits its sliders into many specialist groups. For each token, a small router picks only a few groups to work, so each token needs far less calculation
Takeaway
It shows the importance of trying again and again.
A model fails but keeps on trying. The faster it fails, the more it tries, and during that process, it adjusts the parameters that make it become artificially intelligent.
The same principle applies to you and me. Try again and again. If it doesn’t kill you, it makes you stronger. When you fail, you adjust things in your life, and you become wise. Genuinely wise.
When you ask a chatbot something, think about how many calculations it if performing. Each token is hundreds of billions of calculations. The loop keeps running until you get a complete response.
While AI is doing its thing, you do your thing. Don’t let it fool you. It knows nothing. You know things, and you must see if its output is correct. By checking its work, you become smarter. At the same time, you can delegate the repetitive tasks or tasks that you don’t like doing to your AI assistant.
An ebook: The Billion-Step Machine
How AI Really Works, One Tiny Guess at a Time
(Pay what you want, even zero.)
You can get in-depth into understanding how AI works in The Billion-Step Machine. It has 13 short chapters and explains how chatbots and image generators work, with hands-on experiments, a glossary, and further reading sources. It is available on Gumroad on a pay-what-you-want basis, with PDF and EPUB included on File Road. Click here: https://fileroad.gumroad.com/l/billion-step-machine
