Member-only story
Are You Sure You Know What MoE Is?
All you need is a fever and a hospital
Let’s suppose you walk into a hospital. You have a mild fever. You take a seat, and within a few seconds, a door opens.
It is the heart surgeon who walks in. Then the dentist comes in. And there comes the eye specialist. Now we have got the skin doctor as well as the child specialist.
All 384 doctors practicing in the hospital stand in front of you for a fever.
It gets even crazier. Every single one of them checks you. And then every single one of them sends a bill. Isn’t that stupid?
For years, this is exactly how most AI models worked.
What does that mean?
Every single time you type a word, the entire model wakes up to process it. Every single part of it. Irrespective of whether you ask for a poem or a syntax error fix.
Thankfully, the biggest models today don’t do that anymore.
A model like Kimi K2 has got 1 trillion parameters. But for every word, it uses only about 32 billion of them. That is around 3%.
But hang on. If it’s using 3% for every word, what is the other 97% doing? Who decides which part gets activated? And if only 3% has to be used, why build a 1-trillion-parameter model in…
