}); Episode 0-An Introduction
Beep, beep…
+ Hello?
-Design Twitter!
+ But how… click
Yes, but how?
Let’s look at Twitter. (It’s X now, but I prefer to call it Twitter.)
When we open it from the application, lots of posts show up on our homepage. This is our timeline. When we scroll down the page, new posts are recommended to us. Some of them are from profiles followed by millions of people. They have thousands, sometimes millions, of likes and comments below them.
And we have a + button. When we click it, an empty, text-editable box appears on the page, along with our famous POST button. We can write something and post it from this section. We can also attach media such as photos and videos.
Okay, so we have recommended posts from different profiles, text posts that we can create, media files that can be included in our posts, and like counts and comments for each post.
You are designing these mechanisms in our system. Let’s just forget about signing in and other Twitter mechanisms like AI, blocking, or following profiles. Consider that we are part of a team, and we have been assigned these parts.
Go on. Design the System of Twitter.
Oh, sorry. I forgot that you don’t have any information about System Design.
Don’t worry. I’m here to design it with you.
Follow me!
What a System Is? How Can We Design One??
The average internet user spends around 6 hours every day interacting with networked systems — more than 100 days per year. And more than half of that time is spent through a device sitting in our pockets. So when a company asks you to design a system, remember: you have already been a user of thousands of them. You just need to learn how to see what happens behind the screen.
But what exactly do we mean by “system”?
Every time you send a message, scroll through your feed, watch a video, order something online, or check your bank account, you are interacting with a system.
When you click the “POST” button on Twitter, a few things happen. You hear some clicks, something starts moving in the background, and moments later, your post is shared.
Is this simply the universal power of a button, or is there something more happening behind it? With a single click, you trigger an API request that travels through your internet connection.
“I request this. I want this…”
Suppose we are a hungry guy in a restaurant, and we order some food from the waiter. Our order needs to have a specific structure for the waiter — or the restaurant — to understand what we want. Once they understand what they need to serve, they can start the processes required to make it happen.
We, the hungry guys, are actually the clients, and the restaurant is the server. The specific format our order follows is actually a protocol.
HTTP Structure
In the world of the Internet, requests follow protocols as well. We are going to use the most common and appropriate one: HTTP, the Hypertext Transfer Protocol.
An HTTP request is actually structured data. It contains different pieces of information in its headers and body.
Yes, it has parts.
Let’s take a look at them:
If you already know these terms, good for you. But if you are seeing them for the first time, don’t be scared. We are just going to scratch the surface of them.
As the data structured with HTTP travels through Wi-Fi waves and cables, something is there to catch it. This is the server I mentioned before.
The server looks at the request and sees from its headers that a client at a specific IP address is sending this request. So, it processes the request and sends a response back to that client.
All the routers along the way help carry this request and response to the right addresses, much like postmen delivering a letter. But they are not really our concern here.
Get Yusuf Baran Doğan’s stories in your inbox
Join Medium for free to get updates from this writer.
Our goal is to understand what data should be inside the requests and responses, what the server should do when it receives a request, and what happens when the server decides not to serve it and returns an error instead.
This request-and-response process can happen within milliseconds. An HTTP request may contain only a small amount of data. But when a server receives millions of these requests, each one triggers software and hardware processes that require computational power.
So, how does a server handle millions of requests on its own?
Think about your phone. When you open an application, software and hardware processes start running to make it work. When you open more applications, your phone has more processes to handle, and it may start to slow down.
A server is not so different. It has RAM, storage, a processor. The difference is that instead of serving one person using a few applications, a server may have to handle millions of requests from users at the same time.
And this is exactly what we are here to do: design the system so that it can handle this workload without crashing and process requests as efficiently as possible.
That is a part of system design.
But for now we are looking protocol itself. If you have a knowledge, you can pass to the Needs part.
If we look at the image again, there is a section in the header called METHOD. Every request is divided according to its method. Each method asks the server to do something different. So, let’s take a look at what some of these methods do:
As you see in the picture, each method sent by the client (blue) has a different impact on the server (red). These methods are the actual skeleton of our architecture. Let’s go deeper and look at the structure of the requests then.
GET /car HTTP/1.1
Host: api.fundamantals-system-design.com
Accept: application/json
User-Agent: Blue-Robot-Client/1.0POST /plane HTTP/1.1
Host: api.fundemantals-system-design.com
Content-Type: application/json
Content-Length: 65
{
"name": "Lego Plane",
"color": "colorful",
"propeller": true
}If you look closer, in the POST method, we have a body part inside curly brackets. Thus, we can say that different methods may have different structures.
If the information or the structure of the request is incorrect, the server definitely must not accept it.
We have a decision-making mechanism on servers, as they are computers on a network. They must decide what response should be sent back to the client. There are many different possible responses:
- 200 OK: The request was successful. Everything is fine.
- 301 Moved Permanently: A redirection. The page has been permanently moved to another address.
- 400 Bad Request: The client sent an invalid request.
- 404 Not Found: The requested page or resource could not be found.
- 500 Internal Server Error: Something went wrong on the server side.
System design is about implementing the technologies and strategies needed to handle every problem a server may encounter, while making sure that its responses do not break the rest of the system.
Secure HTTP
How does the client know that it has successfully connected to the server? And how does it know that the communication is still working?
For this, we have a process called a handshake. Before the blue robot sends the famous GET /car request to the red robot, it first needs to establish a connection with it. This is done through a TCP process called the 3-Way Handshake.
SYN: “Hello, are you available?
SYN-ACK: “Yes, I’m here. Are you ready?”
ACK: “Great, let’s start!”
Now the connection is established.
If you want to dive deeper into this process, I recommend you to research Two Generals’ Problem to see where 3-Way Handshake comes from.
But there is a problem. Let’s imagine that there is a malicious Pirate Robot somewhere between the blue robot and the red robot. Since they are using plain HTTP, the Pirate Robot can intercept their communication. It can read the GET /car message, and even worse, if the blue robot sends credit card information or a secret Lego password in a POST request, the Pirate Robot can see and steal that data. It can even modify the message and send something completely different to the red robot.
How do we prevent this?
We use HTTPS (S is coming from Secure). After the basic TCP handshake, but before the GET /car request is sent, another handshake takes place. This time, it is the TLS Handshake.
Client Hello: “Hello, let’s talk securely.”
Server Hello & Certificate: “Here is my identity.” The Red Robot chooses an encryption method and sends a digital certificate proving its identity.
Authentication: “Are you really the Red Robot?” The Blue Robot checks the certificate to make sure that the server it is talking to is actually the Red Robot and not the Pirate Robot.
Key Exchange: “Let’s create a secret.” The Blue and Red Robots perform a key exchange to establish the secret key that will be used to encrypt their communication.
Secure Connection Established. Now both sides have what they need to communicate securely.
Now, even if the Pirate Robot intercepts the packet, instead of seeing: GET /car HTTP/1.1 it sees only encrypted data that looks like meaningless characters, such as: x9$kP2@!q
System Design includes network security layers as we see.
So, what happens when we put all of these pieces together?
A system is not simply a server waiting for requests. It is a carefully designed structure of machines, protocols, and processes, all working together to handle communication between clients and servers.
Every request has to reach the right place, every response has to make its way back, and the system has to keep working even when millions of requests arrive, something goes wrong, or someone tries to interfere with the communication.
System design is about strategically putting all of these pieces together, choosing the right technologies and strategies, and building a system that can handle real-world problems while serving its users as efficiently, reliably, and securely as possible.
In the next episode, we will go through the needs of our system. See you soon!
