Why One Post From a Celebrity Can Break a Database
Picture this. Someone with a hundred million followers posts something right now. Not a big deal for them, they just tapped a button. But somewhere, a server has to figure out how to get that one post in front of a hundred million different people, fast enough that it still feels instant. That single tap just turned into one of the hardest problems in software design.
That’s the case study day three of Designing Data-Intensive Applications opens with, and it’s the best explanation of “nonfunctional requirements” I’ve read, because it never once uses the phrase without showing you exactly what it costs in practice.
The chapter opens with a line from Alan Kay that stuck with me.
The internet works so well that people think of it like a natural resource, something that was just always there, instead of something humans built and keep from breaking
That’s what good engineering looks like from the outside. Invisible. This chapter is about everything that has to happen behind the scenes for that invisibility to hold up.
Two Kinds of Requirements: What It Does, and How Well It Does It
Every app has two different lists of requirements, even if nobody wrote the second list down.
The first list is what the app is supposed to do. A food delivery app needs a button that places an order. A messaging app needs a screen that sends a text. These are functional requirements, the actual features, the actual buttons.
The second list is how well the app needs to do those things. The order button needs to work even when ten thousand people are placing orders at the same time on a Friday night. The message needs to actually arrive, not vanish because a server hiccuped. Nobody writes “the app should not crash” on a feature list, because it feels too obvious to say out loud. But an app that crashes under load might as well not have that feature at all.
That second list is what this chapter calls nonfunctional requirements, mainly performance, reliability, scalability, and how easy the system is to maintain over time. And instead of just defining those words, the book walks through building a small version of a social network like X, formerly Twitter, and shows you exactly where each of those requirements starts to bite.
Building a Tiny Twitter
Here’s the setup. Users post messages and follow other users. Some simple numbers to work with, 500 million posts a day. Do the math on that and it comes out to about 5,800 posts every single second, on average. Not average during a busy moment either, average across the whole day, every day. During something big happening in the world, that number can spike to 150,000 posts a second.
Now think about who’s reading those posts. The average person follows around 200 people and has around 200 followers. But average hides the real story here. Most people have a small handful of followers. A small number of accounts, celebrities, athletes, public figures, have followers in the tens or hundreds of millions. Someone like Barack Obama has over 100 million people following his account. That gap between “average” and “extreme” is exactly where this whole problem lives.
The One Query Everything Depends On
The main thing this app needs to do is show you your home timeline, the recent posts from everyone you follow, newest first. In plain terms, the database has to look up everyone you follow, grab their recent posts, mix all those posts together, sort them by time, and hand you the newest thousand.
That sounds simple enough for one person checking their feed once. The trouble starts with how often people check, and how many people are checking at once.
Say your app wants new posts to show up within five seconds of being posted. The obvious way to do that is to have your phone ask the server “anything new?” every five seconds, over and over, the whole time you have the app open. That’s called polling, and it’s exactly what it sounds like, repeatedly tapping someone on the shoulder to ask if anything changed yet.
Get Sheraz Hussain’s stories in your inbox
Join Medium for free to get updates from this writer.
Now imagine 10 million people have the app open at the same time, all polling every five seconds. That’s 2 million of those timeline queries running every single second. And each one of those queries isn’t cheap, since it has to check up to 200 different people’s recent posts and merge them together. Multiply it out, 2 million queries times 200 followed accounts, and you land on 400 million lookups a second. That’s not a typo. Four hundred million. Every second. For a system built this way, that number alone is enough to bring the whole thing down.
Flip It: Deliver the Mail Instead of Making People Check the Mailbox Constantly
Here’s the fix, and it’s a genuinely clever flip once you see it. Instead of making every reader go fetch fresh results every time they check, deliver the post directly to every follower’s personal timeline the instant it’s posted.
Think of it like mail delivery instead of everyone walking to the post office to ask if anything arrived. The moment someone posts, the system looks up their followers and drops a copy of that post straight into each follower’s own personal mailbox. When you open the app, you’re not running a big search anymore. You’re just opening your mailbox and reading what’s already sitting there.
This is where a specific word comes in, fan-out. One person posts, and that single action turns into hundreds of downstream actions, one delivery per follower. One event fans out into many.
Here’s the picture from the book that makes it click instantly. A user makes one post. That post then fans out and gets delivered separately into three different recipients’ personal timelines, ready and waiting the moment each of them checks their feed.
Run the new math and it’s a completely different story. At 5,800 posts a second, with the average post reaching 200 followers, that’s about 1 million timeline deliveries a second. Compare that to the 400 million lookups a second from before. Still a big number, but a fraction of the original cost, and this time the work happens once, at posting time, instead of getting repeated over and over every time someone checks their feed.
There’s a nice side effect too. During a huge traffic spike, the system doesn’t have to deliver every post the instant it’s posted. It can queue the deliveries and let them arrive a little late, seconds behind instead of instant. Meanwhile, everyone’s timeline still loads fast, because reading it is just opening a mailbox, not running an expensive search.
The Celebrity Problem
Here’s where it gets genuinely hard, and it’s the part of this chapter that stuck with me the most.
Picture someone with 100 million followers posting something. The naive version of fan-out says, the instant they post, go deliver a copy into all 100 million of those followers’ mailboxes, right now. That’s not a big number anymore. That’s a wall.
Compare that to a completely different edge case the book brings up first, someone who follows an enormous number of accounts. If that person’s feed gets flooded with writes because they follow thousands of very active accounts, it’s actually fine to quietly drop a few of those writes. That person almost certainly isn’t reading every single post anyway, so showing them a solid sample instead of every last post barely matters to them.
You cannot do that with a celebrity’s post. Their followers are actively expecting to see it. Dropping writes there isn’t a shortcut, it’s a broken product.
So the fix has to be different. Instead of physically delivering a celebrity’s post into a hundred million separate mailboxes, the system keeps celebrity posts in one shared place, almost like a public noticeboard instead of a hundred million individual letters. When one of their followers opens their own timeline, the system quietly checks that noticeboard for anyone they follow who posts this way, and blends those posts in with their regular precomputed timeline, right at the moment they’re reading it.
That means two different follower groups get two different treatments behind the scenes, but the exact same experience up front. A normal person’s post gets hand delivered ahead of time. A celebrity’s post gets picked up and merged in on the spot when someone checks their feed. The reader never knows the difference. That’s the whole point.
Why This Case Study Actually Matters
None of this is really about social networks specifically. It’s a live demonstration of what those four nonfunctional requirements actually mean once real numbers get involved.
Performance is the difference between a query that costs one lookup and a query that costs 400 million. Reliability is the system staying usable during a massive spike by queueing deliveries instead of collapsing. Scalability is fan-out working fine for someone with 200 followers and completely differently for someone with 100 million, without either of them noticing a difference on their end. And maintainability shows up in that celebrity special case itself, since now there are two different code paths doing two different things, and someone has to keep both of them working correctly, forever, as the system grows.
That last part is worth sitting with. The clever fix for celebrities didn’t remove complexity from the system. It added a second, more complicated path and hid it behind a simple experience. That’s most of what real system design actually is, not making complexity disappear, but deciding exactly where to put it so the person using the product never has to think about it.
That’s day three. Next time I want to get into how this chapter formally defines performance and reliability, the actual language engineers use to measure “fast enough” and “won’t fall over” instead of just feeling it out.
If you’ve ever built something that broke the moment one user got unusually popular, I’d like to hear what that looked like on your end.
