Strata, Explained: Running 125B-Parameter Models on a Gaming PC

A hands-on tutorial covering installation, the exact model compatibility list, the API, and an honest feature-by-feature comparison with llama.cpp

Paul Hoke

12 min read

·

Oct 26, 2026

--

1

Listen

Share

Press enter or click to view image in full size

A year ago, running a 125-billion-parameter model at home meant multiple datacenter GPUs. Today a free, MIT-licensed engine called Strata says it can run Qwen3.8-Flash-Next, a 125B mixture-of-experts model, on a 12 GB graphics card and 64 GB of system RAM. Its own measurements show up to 94 tokens per second.

That claim has attracted attention fast. The official repository shows over 32,000 stars. It has also attracted confusion, unofficial “documentation” sites and a lot of half-correct comparisons. This tutorial explains what Strata actually is, how to install it, which models it really supports, how to call it from code and where llama.cpp is still the better tool.

1. First, which Strata?

Several unrelated projects share the name, so start by getting this right.

• Niko1221/Strata is the subject of this tutorial: a consumer inference engine built around giant MoE models, with an official GitHub repository.
• A separate “Strata” by Mihai Farcas is a from-scratch Rust and HIP engine for Qwen 3.5 on AMD cards, described in his blog post. It is a different codebase.
• Forks such as XeStrata (Intel Arc and SYCL) and strata-nvfp4 extend the original, each with its own model list.

One more warning: stratallm.org calls itself an “Unofficial Community Documentation” hub. Its figures and model list do not match the official repository. For example, it lists Mixtral 8x22B and DeepSeek-V2-Lite, and its benchmark numbers differ from the README. Use the GitHub docs as the only source of truth.

2. The core idea: think in experts, not layers

To see why Strata works, start with the model it was built for. According to the Developers Digest summary of the model card, Qwen3.8-Flash-Next has:

• 125B parameters, with about 6B active per token
• 48 layers, each with 512 experts, of which 10 routed plus 1 shared are active per token
• A separate 51B n-gram embedding table and a multi-token-prediction (MTP) head
• A native 262,144-token context

The important number is 6B active out of 125B. Every token needs only a small slice of the model. A traditional engine splits the model by layers, so whatever does not fit on the GPU gets pushed to the CPU. Strata splits it by experts and treats the whole PC as one memory hierarchy:

VRAM (the counter)    = attention, routers, shared experts, MTP draft layer,
                        hot KV cache, plus a cache of the most-used experts
RAM (the pantry)      = every expert, pinned in memory; the CPU computes
                        experts the GPU does not hold, at the same time
SSD (the warehouse)   = the 29 GB n-gram lookup table, read a few rows per token

The VRAM expert cache keeps adapting to the conversation. According to the engine reference, “every extra GB holds about 700 more experts, and every expert on the GPU is one the CPU does not have to compute.” That leads to a practical rule that goes against intuition: for Strata, more VRAM matters more than a faster GPU.

The second speed lever is speculative decoding with MTP. The model’s built-in draft layer guesses several tokens ahead, and the full model checks all the guesses in one pass. Every token is still verified by the big model. There is also a prompt-lookup drafter that speeds up code edits, where the reply often repeats text from the context.

3. Hardware requirements

The README lists this hardware as supported.

Press enter or click to view image in full size

For the flagship model, plan on a GPU with 12 GB or more, 32 to 64 GB of RAM (RAM decides which quantization fits), an NVMe SSD and roughly 70 to 80 GB of free disk for the smaller packs.

Support is not the same for every model. Section 5 shows which models run on which hardware.

4. Installation

Strata now ships two official installation paths.

Option A: the desktop app

Download the installer for your system from the repository releases. The files for version 0.2.6 are:

• Strata-0.2.6-windows-x64-setup.exe
• Strata-0.2.6-linux-x86_64.AppImage
• Strata-0.2.6-macos-arm64.dmg

Install it, pick a model and start chatting. On first launch, the app downloads the engine build that matches your graphics card.

Option B: the terminal app

On Linux or macOS:

curl -fsSL https://github.com/Niko1221/Strata/releases/latest/download/install.sh -o install.sh && sh install.sh

On Windows PowerShell:

irm https://github.com/Niko1221/Strata/releases/latest/download/install.ps1 | iex

On Windows, open a new terminal afterward. Then run the guided launcher:

strata launch

Good habit: preview before downloading. The --plan flag prints what Strata would download and how it would configure the run, without changing anything:

strata launch --plan --family qwen --model iq2_xs

When the plan looks right, run the same command without --plan. Add --yes --no-tui to run it unattended.

Before running piped install scripts, read them first. That advice applies to any curl | sh or irm | iex command. Developers Digest also points out that the README offers a prompt asking a coding agent to run the setup for you. Review those docs before giving an agent that much control.

Expect a rough first start. The model loads 35 to 55 GB into RAM, and the PC can stop responding for one to three minutes. That is normal.

Source builds (START-HERE.bat, ./setup.sh) and Docker are still available for contributors. The docs make clear that those scripts exist only in a source checkout, not in the installed app.

5. The exact model compatibility list

This list comes from MODELS.md and the README as of 2026–10–25. It is the official list. Forks support other models, but those do not count here.

Family 1: Qwen3.8-Flash-Next (125B MoE), the flagship

ISTA-DASLab packs (the recommended provider):

Press enter or click to view image in full size

Variants and other providers of the same model:

• Coder (IQ1_M): keeps 256 of 512 experts per layer, chosen on code. Its authors report 91% of the full model’s SWE-bench Verified score. It runs with about 32 GB of RAM, but it is weaker outside code and outside English, including Chinese and other CJK text. Install with --family coder.
• Swift 1.5: a fine-tune by UkisAI that thinks for fewer tokens at about the same quality. Install with --family swift. It uses its own license.
• Unsloth (--family unsloth): UD-IQ4_XS, UD-Q4_K_XL, UD-Q5_K_XL, UD-Q6_K_XL and Q8_0. Keeping every expert in RAM takes about 72 to 136 GB.
• Gyro-S and Gyro-M (agentionai): use the least memory and run slower. Gyro-S needs 27.6 GB of VRAM and RAM combined.
• NVFP4 (NVIDIA cards only): three versions (NVFP4, NVFP4-NVIDIA, NVFP4-QAD). Each needs about 80 GB of RAM.
• OrcaRouter Uncensored IQ3_XXS: manual conversion only. It is not in the installer menu.

Family 2: Qwen3.6–35B-A3B (smaller MoE, Apache 2.0)

Unsloth GGUFs only: UD-Q4_K_XL, UD-Q6_K_XL and Q8_0, needing 32 to 52 GB of RAM. It runs on NVIDIA and AMD cards. The docs say the Intel Arc engine and the Mac engine do not run it. Unlike the flagship, it can serve several requests at once.

Family 3: DeepSeek V4.1-Flash-1017 (very large; separate engine)

IST-DAS sets at 1.75, 2.00 and 2.25 bits per weight (239 to 274 GB downloads). It needs a 12 GB or larger card, at least 46 GB of RAM and a fast NVMe drive, because much of the model is read from the SSD. It runs on NVIDIA RTX 30, 40 and 50 series (Windows and Linux) and on the AMD Radeon AI PRO R9700 (Linux). The docs say Mac and Strix Halo do not run it yet. Expect single-digit tokens per second on a 12 GB card.

What is not on the list

Strata does not load any GGUF you point it at. strata models import <folder or .gguf> --plan checks whether a set of files maps onto a supported pack. Llama, Mistral, Gemma and other architectures are not officially supported. If you need them, use llama.cpp.

6. Practical examples: using the API

Once a model is running, Strata serves an HTTP API at http://127.0.0.1:8080.

Example 1: health check and first request

curl http://127.0.0.1:8080/health

curl http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "strata", "messages": [{"role": "user", "content": "Explain MoE routing in two sentences."}], "max_tokens": 512}'

Any API key and any model name work. The server runs one model, so it ignores the model field.

Example 2: controlling thinking

The model thinks before answering by default (effort level high). For quick tasks, turn it down per request:

{
  "model": "strata",
  "reasoning_effort": "none",
  "messages": [{"role": "user", "content": "Classify this ticket: refund request for order 1182."}]
}

Valid levels are none, low, medium and high. On the Anthropic endpoint, use "thinking": {"type": "disabled"} or a budget_tokens value.

Example 3: connecting a coding agent

Strata serves three API formats: OpenAI Chat Completions (/v1/chat/completions), Anthropic Messages (/v1/messages) and OpenAI Responses (/v1/responses). That covers most coding agents. For Claude Code, the README as quoted by Developers Digest gives:

export ANTHROPIC_BASE_URL=http://127.0.0.1:8080

For everything else, run strata connect. It prints setup steps for Claude Code, Codex CLI, OpenCode and other supported tools.

Example 4: structured JSON output

{
  "model": "strata",
  "response_format": {"type": "json_object"},
  "messages": [{"role": "user", "content": "Return the invoice date and total as JSON."}]
}

One detail matters here. The engine reference states this is “schema prompting followed by server validation, not grammar-constrained decoding.” Malformed output returns a 502 error with structured_output_failed. Client code should catch that and retry.

Example 5: sharing the GPU with other work

By default the model stays loaded until Strata closes. To free VRAM automatically, add this key to strata-<model>.json:

{ "idle_unload_s": 600 }

After ten idle minutes the model unloads, and the next request reloads it. You can also unload it manually with curl -X POST -H "Content-Type: application/json" localhost:8080/unload.

Example 6: monitoring

GET /metrics returns JSON by default. Send Accept: text/plain and it returns Prometheus format using vLLM's metric names, so existing vLLM

Grafana dashboards work with Strata unchanged. Ready-made files are in docs/monitoring/.

Example 7: measuring Strata against llama.cpp yourself

Published benchmarks are a starting point. Your own measurements on your own hardware matter more. The script below sends the same prompt to two OpenAI-compatible endpoints and reports time to first token and approximate output speed.

"""
Side-by-side latency check for two OpenAI-compatible local endpoints.

Streams one prompt to each server and reports time to first token and
approximate decode speed, so engines can be compared on the same machine.
"""
import time
import typer

from dataclasses import dataclass
from datetime import datetime
from loguru import logger
from openai import OpenAI

# ============================================================================
# Configuration
# ============================================================================
DEFAULT_PROMPT = "Write a Python function that merges two sorted lists, with tests."
DEFAULT_MAX_TOKENS = 512

app = typer.Typer()

# ============================================================================
# Data Models
# ============================================================================
@dataclass
class StreamResult:
    """Timing results for one streamed completion."""

    label: str
    ttft_seconds: float
    chunks: int
    decode_seconds: float

    @property
    def chunks_per_second(self) -> float:
        """Approximate decode speed in streamed chunks per second."""

        if self.decode_seconds <= 0:
            return 0.0

        return self.chunks / self.decode_seconds

# ============================================================================
# Measurement
# ============================================================================
def measure(label: str, base_url: str, prompt: str, max_tokens: int) -> StreamResult:
    """Stream one completion and time it.
    Args:
        label: Display name for the endpoint.
        base_url: OpenAI-compatible base URL, ending in /v1.
        prompt: User prompt sent to the model.
        max_tokens: Output token cap.
    Returns:
        A StreamResult with first-token latency and decode timing.
    """

    client = OpenAI(base_url=base_url, api_key="local")
    started = time.perf_counter()
    first_token_at: float | None = None
    chunks = 0

    # Thinking is disabled so both engines spend tokens on the answer only
    stream = client.chat.completions.create(
        model="local",
        messages=[{"role": "user", "content": prompt}],
        max_tokens=max_tokens,
        temperature=0,
        stream=True,
        extra_body={"reasoning_effort": "none"},
    )

    for event in stream:
        if not event.choices:
            continue

        delta = event.choices[0].delta

        if not (delta.content or getattr(delta, "reasoning_content", None)):
            continue

        if first_token_at is None:
            first_token_at = time.perf_counter()

        chunks += 1

    finished = time.perf_counter()
    first = first_token_at or finished

    return StreamResult(
        label=label,
        ttft_seconds=round(first - started, 3),
        chunks=chunks,
        decode_seconds=round(finished - first, 3),
    )

@app.command()
def main(
    strata_url: str = typer.Option("http://127.0.0.1:8080/v1", help="Strata base URL"),
    llamacpp_url: str = typer.Option("http://127.0.0.1:8081/v1", help="llama-server base URL"),
    prompt: str = typer.Option(DEFAULT_PROMPT, help="Prompt to send"),
    max_tokens: int = typer.Option(DEFAULT_MAX_TOKENS, help="Output token cap"),
) -> None:
    """Compare first-token latency and decode speed across two engines."""

    run_at = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
    logger.info(f"Run started {run_at}")
    targets = {"strata": strata_url, "llama.cpp": llamacpp_url}

    for label, url in targets.items():
        result = measure(label, url, prompt, max_tokens)

        logger.info(
            f"{result.label}: TTFT {result.ttft_seconds}s, "
            f"{result.chunks} chunks, ~{result.chunks_per_second:.1f} chunks/s"
        )

if __name__ == "__main__":
    app()

Streamed chunks usually match tokens one to one, but not always, so treat chunks per second as an estimate. Run each engine separately so they do not compete for VRAM. Repeat each test several times, and use the same GGUF file, context size and prompt for both.

For the llama.cpp side, a typical MoE offload command looks like the following.

llama-server -m model.gguf -ngl 99 --n-cpu-moe 30 -c 32768 --port 8081

--n-cpu-moe keeps the expert weights of the first N layers on the CPU. Tune N until the model fits in VRAM.

7. Strata vs llama.cpp: the full feature comparison

Strata is not a rival built from nothing. The README credits parts of llama.cpp and ggml, and its API deliberately reuses llama.cpp names such as /slots, n_ctx and chat_template_caps. The two tools are built for different jobs.

Press enter or click to view image in full size

What the speed comparisons actually show

There are two independent tests are worth knowing.

• Same file, both engines: SOTAAZ ran the same IQ2_XS file in both. Quality matched: GSM8K 93.3% vs 93.0%, and HumanEval 157 of 164 for each. Held to 12 GiB of VRAM, Strata decoded at 62.9 tokens/s vs 23.4 for llama.cpp.
• Long context: Startup Fortune reports that developer Carteakey measured Strata at 53.2 tokens/s at 60,000 tokens of context, against 20.8 for upstream llama.cpp. His later write-up puts the gap closer to 2x against his best-tuned llama.cpp setup.

The lesson: same model and same quality, but Strata is clearly faster on its target models. The gap is real, but it only applies to those models.

8. Tuning and gotchas

• Long first prompts are slow. Time to first token equals prompt length divided by prompt speed. With Q2_0 the docs estimate about 25 seconds at 32K tokens. After the first message, Strata keeps the conversation cached and reads only the new text, so stay in one thread.
• Calibrate for your PC. --tune quick or --tune thorough measures settings for your machine. The CPU worker count, PCIe share and cache adaptation rate all depend on hardware.
• Keep the learned cache across restarts. Set "expert_profile_save" in the config. Keep one profile per kind of work, because a profile learned on code can slow down prose.
• Windows Task Scheduler trap. The docs measured expert loading fall from about 35 seconds to over 800 seconds under default Task Scheduler settings. Set the task’s priority to Normal and enable “Run with highest privileges.”
• Output is not byte-reproducible by default. The adaptive cache changes from run to run. For strict A/B tests, the docs list the flags that make output repeatable.
• Model licensing. Strata’s MIT license covers the engine, not the model weights. Qwen3.8-Flash-Next uses the Qwen Community License 1.0, which requires a separate license for “Model as a Service” or “AI Work Assistant” businesses. Read it before building a product on it.

9. Common misconceptions

• “Strata runs any GGUF faster than llama.cpp.” It runs a short, curated list. Outside that list it does not run the model at all.
• “Q2 on Strata equals the full model.” The impressive model-card scores are for full precision. IQ3_S is the pack the docs say matches the full model on published tests. Smaller packs trade some quality for speed and memory.
• “A faster GPU is the best upgrade.” For Strata, VRAM capacity usually matters more, because every extra gigabyte holds more experts.
• “Strata replaces a serving stack.” It is a personal engine. For many users at once, tools like vLLM and SGLang are built for that job.

10. A practical starting path

1. Check your RAM and VRAM against the table in Section 5.
2. Install the terminal app and run strata launch --plan to preview the setup.
3. Start with IQ2_XS on 48 to 64 GB of RAM, or the Coder on 32 GB for code-only work.
4. Send the health check and Example 1 request.
5. Connect one coding agent with strata connect.
6. Run the comparison script against llama.cpp using the same file, and keep the numbers.
7. Run --tune quick, then measure again.

Bottom line

llama.cpp = the universal engine: every model, every machine
Strata    = the specialist: a few giant MoE models, as fast as consumer hardware allows
Experts   = the unit Strata manages, across VRAM, RAM and SSD
VRAM      = the upgrade that matters most for Strata
Evidence  = same quality on the same file, roughly 2x or more faster on its targets
Rule      = use Strata for its flagship models, llama.cpp for everything else

Most people will end up using both: llama.cpp (directly or through Ollama) for everyday experiments with many models, and Strata as a dedicated local engine when one very large MoE model needs to run fast on hardware they already own.