Why Discord Moved From Cassandra to ScyllaDB to Store Trillions of Messages
A system design deep-dive into the architecture that keeps a billion-message-a-day platform from breaking a sweat
Every time you fire off a message in Discord, it lands in a database that holds trillions of its siblings. No spinner. No lag. Just an instant little “delivered.”
That smoothness is not an accident, and it is not the result of one magic database. It is the product of nearly a decade of painful migrations, two databases thrown away, a custom disk topology, and a thin layer of Rust doing some very clever bookkeeping.
This article walks through how Discord got here — and more importantly, why each decision was made. If you design distributed systems, the interesting part isn’t the database they picked. It’s the reasoning.
- Discord went through three storage generations: MongoDB → Cassandra → ScyllaDB.
- Cassandra did not fail because it was a bad database. It failed because Discord’s workload had outgrown the operational shape Cassandra gave them: hot partitions and garbage collection pauses.
- The fix wasn’t just a better database. It was moving load off the database: a Rust “data services” layer that collapses duplicate reads, a GC-free C++ storage engine, and a custom superdisk for predictable latency.
- The payoff: 177 nodes → 72 nodes, and p99 read latency from 40–125ms → 15ms.
The three ages of Discord’s database
Discord’s storage story is a story of outgrowing things.
Age 1 — MongoDB (2015)
In the beginning there was a single MongoDB replica. It worked until it didn’t. By late 2015, around the 100-million-message mark, the working set — data plus indexes — no longer fit in RAM. Once you spill to disk on every read, latency stops being predictable. Time to move.
Age 2 — Cassandra (2017–2022)
Cassandra was the obvious next step: a distributed, horizontally scalable, fault-tolerant store with no single point of failure. Discord started with a modest 12-node cluster.
It scaled — for a while. By early 2022, the message count had crossed into the trillions and the cluster had swollen to 177 nodes. Latency grew unpredictable, and the operational cost of keeping it healthy became brutal. The database meant to scale with Discord had become the thing holding it back.
Age 3 — ScyllaDB (2022–today)
The current home for Discord’s messages. Cassandra-compatible on the outside, radically different on the inside. We’ll get there — but first, why Cassandra had to go.
Why Cassandra broke
Two failure modes did the damage. Both are worth internalizing because they show up in almost every large stateful system.
1. Hot partitions
In a partitioned database, your data is sharded by a partition key. The implicit assumption is that load spreads evenly across those partitions. It usually doesn’t.
A popular channel — a celebrity announcement, a game launch, a viral moment — drives a flood of traffic at a single partition. That one partition’s replica nodes get hammered while the rest of the cluster sits idle. Worse, the slowdown doesn’t stay contained: requests queue, threads block, and the latency spike cascades across the system.
2. Garbage collection
This one is for everyone who has ever tuned a JVM at 2 a.m.
Cassandra is written in Java. At Discord’s scale, GC stopped being a background nuisance and became an operational hazard. Stop-the-world pauses spiked tail latency. Occasionally the pauses chained together so badly that a node became unresponsive and an engineer had to manually reboot it and babysit it back to health.
GC pauses were a leading source of on-call pain — and a database you have to hand-hold through garbage collection is not a database you can grow on.
The lesson Discord took away wasn’t “Java bad.” It was that an unpredictable runtime under your storage engine becomes your unpredictability, multiplied by every node.
The new architecture
The redesign has three parts, and the most important insight is that only one of them is “a new database.” The rest is about keeping work away from the database in the first place.
Piece 1: ScyllaDB — the same interface, a calmer engine
ScyllaDB speaks Cassandra’s protocol and data model, so the migration didn’t require rewriting application logic. But under the hood it’s written in C++, which means no garbage collector — the single biggest source of Cassandra’s tail-latency chaos simply disappears.
Get Amit’s Engineering Log’s stories in your inbox
Join Medium for free to get updates from this writer.
Its other structural advantage is a shard-per-core architecture. Each CPU core owns its own slice of the data and processes requests independently, with its own scheduler. Compare that to a multi-threaded JVM model where threads contend for shared structures and a single GC cycle freezes everything. Shard-per-core gives you better workload isolation and far less coordination overhead.
Piece 2: Data Services — a Rust layer that refuses to do the same work twice
This is the cleverest part of the design, and it’s the direct antidote to hot partitions.
Discord inserted a thin intermediate layer — written in Rust — between the API monolith and the database. Its signature trick is request coalescing.
Imagine a celebrity posts in a channel and a million clients try to read the same recent messages within the same instant. Naively, that’s a million identical queries pointed at one already-stressed partition. The data services layer instead recognizes that these are all the same request, fires one query to ScyllaDB, and fans the single result back out to all million waiters.
Rust matters here specifically because this layer is a concurrency problem: you’re juggling thousands of in-flight requests and waiters safely, without GC pauses of your own. Rust’s ownership model lets you do “fearless concurrency” without a runtime that stutters.
Piece 3: The Superdisk — speed and durability
Discord runs on GCP, where you face a classic tradeoff: local SSDs are blazing fast but ephemeral (lose the VM, lose the disk), while persistent disks are durable but slower.
Their answer was a two-layer RAID they call a superdisk:
- RAID 0 stripes several local SSDs together into one low-latency virtual disk.
- RAID 1 mirrors that fast array onto a persistent disk.
You read at local-SSD speed; you survive failures at persistent-disk durability. The hardware stops being a source of latency surprises — which, when you’re chasing single-digit-millisecond p99s, is the whole game.
The results
The efficiency win is not hand-wavy. It’s measurable, and it’s large.
Half the nodes, denser storage, and tail latencies that stopped depending on whether a compaction or a GC cycle happened to be running. The system got quieter — which, for an on-call team, is the real metric.
The migration itself
Migrating trillions of live messages with zero downtime deserves its own footnote.
The plan was the textbook approach: dual-write new data to both databases, then backfill history behind a cutover point. Verification used random sampled reads to confirm the two stores agreed.
The fun part: the off-the-shelf migrator projected a three-month backfill. The team decided three months was three months too long, rewrote the migrator in Rust in about a day, and pushed 3.2 million records per second. Three months collapsed into nine days — no downtime.
It’s a nice reminder that sometimes the bottleneck isn’t your architecture; it’s the tool you grabbed first.
Takeaways for your own systems
You will probably never store trillions of messages. The principles still transfer:
- Your database’s worst day is a hot partition. Design keys to spread load, and put a coalescing/caching layer in front of inevitable hotspots rather than praying they don’t happen.
- An unpredictable runtime under your storage is your problem. GC pauses, noisy neighbors, variable disks — anything that makes per-request latency nondeterministic will surface as tail latency at scale.
- The best database optimization is often less load on the database. Discord’s biggest wins came from the Rust layer that prevented queries, not from the query engine itself.
- Boring, measurable wins beat clever ones. “Half the nodes, steady 15ms p99” is the kind of result that ends firefighting.
The next time Discord delivers your message instantly during a match-day traffic spike, you’ll know what’s underneath: a C++ engine with no GC, a Rust layer quietly deduplicating a million reads into one, and a hand-built disk that refuses to flinch.
Sources
This article synthesizes Discord’s own publicly published engineering material. All figures and architectural details are drawn from the following:
- Discord Engineering Blog — How Discord Stores Trillions of Messages: discord.com/blog/how-discord-stores-trillions-of-messages
- ScyllaDB — How Discord Migrated Trillions of Messages from Cassandra to ScyllaDB (talk by Bo Ingram, Senior Software Engineer at Discord, ScyllaDB Summit 2023): scylladb.com/tech-talk/how-discord-migrated-trillions-of-messages-from-cassandra-to-scylladb
- The New Stack — How Discord Migrated Trillions of Messages to ScyllaDB: thenewstack.io/how-discord-migrated-trillions-of-messages-to-scylladb
If you enjoyed this breakdown, it’s part of an ongoing system design series. Next up: how to design the message data model itself — bucketing strategies that keep partitions from growing unbounded.
