Running 125B Models for GPU Poor People
I am one of those GPU Poor People, instead of getting RTX 5090 when it was cheaper I decided I just wanted to get RTX 5060Ti, who needs RTX 5090? They have their melting connector issues, high power consumption, etc.
Turns out the answer was me, I NEED 5090 for running LLM models locally but I didn’t know that then.
The result of my bad decision has been, I am stuck on running 7–14B parameter models (mostly quantized models) on my GPU.
But recently that changed, now I can run the 125B model on my PC while getting ~55–60 tokens/sec. All that because I found Strata on one of those AI channels making videos about GitHub repos while scrolling YouTube shorts at 2 in the morning. I am ashamed of my doomscrolling but at least I found something good.
In this article, we are going to talk about Strata & I will introduce you guys to some new concepts, in the upcoming parts we will setup strata on my PC.
What is Strata?
Strata is an inference engine that allows users to run Qwen3.8-Flash-Next, a 125B parameter model on consumer hardware with OpenAI/Anthropic compatible API on Windows or Linux.
125B Model ≠ 125GB GPU
My assumption was if you need to run xB model then you need xGB VRAM at least. So why does Strata work on so much less VRAM?
There are 2 parts to it:
- Using quantized model
- Model being a MoE model
Using quantized model
If you look at Qwen3.8-Flash-Next hugging face model card you will see the model is 360GB at BF16 and it is a 180B parameter model (wait 180B? I thought it was 125B, yes I know, we will get to that in next part). Instead of BF16 which is 16-bit, with strata we are using 3-bit model (to be specific IQ3_XXS made by ISTA-DASLab). With quantization, our model shrinks from 360GB to 75GB.
Get Nikhil’s stories in your inbox
Join Medium for free to get updates from this writer.
Downside of quantization is degradation in quality. I was assuming the Strata dev, Niko1221, has compared full precision model & quantized ones but turns out ISTA-DASLab has already measured it. On their benchmarks (AIME25, GPQA-Diamond, LiveCodeBench) IQ3_XXS gets 99.4% of the score of full BF16 model while being 4.7x smaller, so the loss of quality is negligible.
Using MoE model
When anyone says a model is 120B or 350B, doesn’t really mean the model is using that many parameters for every token. Model only produce output with active parameters, in dense model all the parameters are active but in case of Qwen3.8-Flash-Next active parameters are around 6B. So only 6B parameters are doing the work for each token, but it is not the same 6B every time.
Note: If you don’t know what MoE models are I recommend reading this blog post from Hugging Face
Every LLM model has multiple layers and every token goes through all of them, this is same for dense and MoE models. The difference is inside the layer, in a dense model the whole layer is used but in MoE models each layer has lots of small experts and only few of them are used for a token.
Qwen3.8-Flash-Next has 48 layers with 512 experts and each layer uses 10 experts for every token, that’s around 2% of the layer.
Think of it like a hospital with 512 doctors, each one is a specialist in something. When a patient comes in, the receptionist (in MoE it is called router) sends them to 10 doctors who are best for that problem, the other 502 don’t do anything for that patient. Next patient comes in and receptionist again picks 10, some might be the same as last time and some different.
But here is the catch, we don’t know which 10 experts are needed until the token comes in. So we can’t keep only 10 experts in VRAM, all 512 has to be somewhere close so the right 10 are available when needed. This is the thing that Strata solves.
Conclusion
So 125B model doesn’t need 125GB of VRAM because of 2 things. First, quantization shrinks the model from 360GB to 75GB without losing much quality. Second, the model is MoE so for each token only around 6B parameters are doing the work.
So turns out my bad decision of getting 5060Ti wasn’t that bad after all, I still want 5090 though.
Next part is how strata stores things and how we went from 180B parameters model to 125B while using strata.
