Building a BitTorrent Client from Scratch (part 1): Understanding the architecture
“Sometimes it is the people no one imagines anything of who do the things that no one can imagine.” — Alan Turing
Starting off with a quote of Alan Mathison Turing, let’s dive in this series of building a BitTorrent client from scratch. This first blog will be covering the outlines of what we’ll be building, some key components of BitTorrent. We’ll look into Bencoding, DHTs, Trackers, Peer Discovery Protocols. Potentially we’ll try exploring a few additional algorithms such as the Choking-Unchoking algorithms, End Game algorithm etc.
(The reason I’ve mentioned “We’ll try exploring…” even though this is supposed to be a full-fledged torrent implementation, is because covering everything in a single blog might be difficult, as we progress further, things will start becoming clearer)
Introduction
When most developers think about distributed systems, technologies such as Kubernetes, Apache Kafka, Cassandra, or distributed databases often come to mind. Yet long before many of these systems became popular, BitTorrent was already solving large-scale distribution problems across millions of machines connected through the public Internet.
At its core, BitTorrent is a distributed file transfer protocol. That description, however, barely scratches the surface of what makes it interesting. Behind every torrent download lies a collection of independent systems working together: metadata serialization formats, peer discovery mechanisms, decentralized routing tables, network protocols, integrity verification schemes, and scheduling algorithms.
Over the next few articles, we’ll build a BitTorrent client from scratch and understand each of these components in detail. The objective isn’t merely to download files successfully. The objective is to understand why the protocol is designed the way it is and how seemingly simple design decisions allow BitTorrent to scale to enormous numbers of participants.
Before writing any code, however, we need to understand the architecture we’re about to build.
The Problem BitTorrent Was Designed To Solve
Imagine distributing a 4 GB Linux ISO image to 10,000 users.
In a traditional client-server architecture, every download request must be served directly by the hosting server.
The problem becomes obvious almost immediately.
If every client downloads the entire file, the server must upload:
4 GB × 10,000 = 40 TBThe server becomes the bottleneck.
As demand increases:
- Bandwidth costs increase
- Infrastructure requirements increase
- Network congestion increases
- Failure of the central server affects everyone
This architecture scales poorly because the burden of distribution grows linearly with the number of users.
BitTorrent introduces a radically different idea.
Instead of treating downloaders as passive consumers, every downloader becomes a distributor.
The moment a client downloads a piece of data, it can immediately begin sharing that piece with other peers.
Rather than one server serving thousands of clients, thousands of clients cooperate to distribute the workload among themselves.
The fascinating consequence of this design is that increased demand often improves the network instead of harming it. More users frequently means more available bandwidth, more redundancy, and better resilience.
This simple idea forms the foundation of the entire BitTorrent ecosystem.
A High-Level View of BitTorrent
Before diving into individual components, it’s useful to understand the lifecycle of a torrent download.
When a user starts downloading a torrent, several things happen:
Torrent File
│
▼
Metadata Parsing
│
▼
Peer Discovery
│
▼
Peer Connections
│
▼
Piece Exchange
│
▼
Integrity Verification
│
▼
File ReconstructionEach stage introduces its own set of challenges. How does a client know which files belong to a torrent? How does it discover peers? How does it verify downloaded data? How does it coordinate thousands of simultaneous participants?
The rest of BitTorrent’s architecture exists to answer these questions.
Torrent Files: The Blueprint Of The Swarm
The first misconception many people have is that a torrent file contains the actual data being downloaded.
It doesn’t.
A torrent file is purely metadata.
Think of it as a blueprint describing a distributed dataset.
A torrent file typically contains:
- Names of files
- File sizes
- Piece lengths
- Piece hashes
- Tracker information
Conceptually, it looks something like:
Torrent Metadata
├── File Information
├── Piece Information
├── Tracker URLs
└── Integrity DataHere’s a snippet of a sample file (Source: Free Torrents — Public Domain, Creative Commons — WebTorrent)
d8:announce40:udp://tracker.leechers-paradise.org:6969
13:announce-list
ll
40:udp://tracker.leechers-paradise.org:6969
el
34:udp://tracker.coppersurfer.tk:6969
el
33:udp://tracker.opentrackr.org:1337
el
23:udp://explodie.org:6969
el
31:udp://tracker.empire-js.us:1337
el
26:wss://tracker.btorrent.xyz
el
32:wss://tracker.openwebtorrent.com
el
25:wss://tracker.fastcast.nz
e
7:comment
34:WebTorrent <https://webtorrent.io>
10:created by
34:WebTorrent <https://webtorrent.io>
13:creation date
i1490916601e
8:encoding
5:UTF-8
4:info
d5:fil...Notice what’s missing.
The actual file contents. A torrent file contains enough information to locate and verify data, but none of the data itself.
This separation is important because it allows a tiny metadata file to describe datasets that may be several gigabytes — or even terabytes — in size.
One of the most important values stored inside a torrent file is the Info Hash.
The Info Hash acts as a unique identifier for the torrent. Nearly every operation in the BitTorrent ecosystem eventually revolves around this value.
Peers use it to identify swarms.
Trackers use it to organize participants.
DHT nodes use it to locate peers.
Without the Info Hash, there is effectively no torrent.
Bencoding: The Language of Torrent Metadata
Since torrent files contain metadata, a question naturally arises:
How is that metadata represented?
BitTorrent uses a serialization format called Bencoding.
At first glance, Bencoding appears surprisingly primitive compared to formats such as JSON or Protocol Buffers.
This simplicity is intentional.
The format supports only four data types:
- Integers
- Strings
- Lists
- Dictionaries
That’s it.
No booleans.
No floating-point values.
No schemas.
No optional fields.
The result is a format that is extremely easy to parse and deterministic to encode.
Deterministic encoding is particularly important because the Info Hash is generated by hashing a Bencoded structure. Two clients encoding the same data differently would generate different hashes and therefore join different swarms.
By restricting the format and eliminating ambiguity, BitTorrent guarantees consistent hash generation across implementations.
In many ways, Bencoding represents one of the protocol’s earliest design philosophies:
Simplicity is often more valuable than flexibility.
Our first implementation milestone in this series will be building a Bencoding parser from scratch.
Trackers: The Original Peer Discovery Mechanism
After reading a torrent file, a client knows what it wants to download.
What it doesn’t know is where the data exists.
This introduces a new problem.
How do peers find one another?
The original solution was the tracker.
A tracker is a service responsible for maintaining information about participants in a swarm.
When a client starts a torrent, it contacts the tracker and sends an announce request containing information such as:
- The torrent’s Info Hash
- Peer identifier
- Download statistics
- Upload statistics
- Listening port
The tracker responds with a list of peers currently participating in the same torrent.
Client ─────► Tracker
Client ◄───── Peer ListAn important detail is that trackers never transfer actual file data.
Get ACHINTYA SHARMA’s stories in your inbox
Join Medium for free to get updates from this writer.
They merely introduce participants.
Once peer addresses are exchanged, all communication occurs directly between peers.
Trackers solved the peer discovery problem effectively, but they introduced a new weakness.
Centralization.
If the tracker disappears, peer discovery becomes difficult.
This limitation eventually led to one of BitTorrent’s most interesting innovations.
Distributed Hash Tables (DHT): Removing the Central Authority
The tracker model works well until the tracker fails.
What if peer discovery could happen without any central server at all?
This is precisely the problem solved by Distributed Hash Tables.
DHT transforms peer discovery into a decentralized operation.
Instead of asking a tracker:
“Who has this torrent?”
A client asks the network itself.
Most BitTorrent clients implement a DHT system based on the Kademlia algorithm.
Kademlia organizes participating nodes into a structured overlay network where each node maintains information about a subset of other nodes.
Node A
│
├── Node B
├── Node C
└── Node DWhen searching for peers associated with a particular Info Hash, nodes progressively route queries toward participants that are numerically closer to the target.
This process resembles searching through a distributed directory rather than broadcasting requests across the entire network.
The result is a system that remains:
- Scalable
- Fault tolerant
- Decentralized
- Efficient
DHT is arguably one of the most fascinating components of BitTorrent because it demonstrates how large-scale discovery can emerge without centralized infrastructure.
Peer Discovery: Finding The Swarm
Peer discovery is often treated as a single feature, but in reality it is an umbrella term encompassing multiple mechanisms.
A modern BitTorrent client may discover peers through:
- Trackers
- DHT
- Peer Exchange (PEX)
Each mechanism contributes additional peers to the swarm.
The objective is simple:
Increase connectivity.
A peer connected to only three other participants has limited download opportunities.
A peer connected to hundreds of participants has significantly more options when requesting pieces.
The effectiveness of peer discovery directly influences download speed, resilience, and overall swarm health.
Without peer discovery, BitTorrent cannot function.
Without effective peer discovery, BitTorrent performs poorly.
The Peer Wire Protocol
Finding peers is only the beginning.
Once a connection is established, peers must communicate.
This communication occurs through the Peer Wire Protocol.
The protocol defines every message exchanged between participants.
The interaction begins with a handshake.
The handshake serves two purposes:
- Verify that both peers are participating in the same torrent.
- Exchange identifying information.
Once the handshake succeeds, peers begin exchanging protocol messages.
These messages coordinate:
- Available pieces
- Piece requests
- Upload permissions
- Download permissions
- Transfer cancellations
In many ways, the Peer Wire Protocol represents the nervous system of BitTorrent.
Every download, upload, verification, and scheduling decision ultimately flows through these messages.
When we eventually implement this component, we’ll be dealing directly with binary network protocols, message framing, and connection management.
Piece-Based Distribution
Perhaps the most important idea in BitTorrent is that files are never transferred as complete units.
Instead, files are divided into smaller pieces.
File
├── Piece 0
├── Piece 1
├── Piece 2
├── Piece 3
└── Piece NThis design creates several advantages.
First, different pieces can be downloaded simultaneously from different peers.
Peer A → Piece 1
Peer B → Piece 5
Peer C → Piece 9Second, peers can begin uploading immediately after downloading a piece.
They don’t need to wait for the entire file.
Third, corruption becomes easier to detect.
Each piece has an associated SHA-1 hash stored inside the torrent metadata.
After downloading a piece, the client computes its hash and compares it with the expected value.
If the hashes differ, the piece is discarded.
This verification mechanism guarantees data integrity even when data is exchanged between thousands of untrusted participants.
Algorithms That Make BitTorrent Efficient
BitTorrent’s success isn’t solely due to its architecture.
Several scheduling and optimization algorithms contribute significantly to its performance.
One such strategy is Rarest First, where clients prioritize downloading pieces that are least common within the swarm.
This prevents rare pieces from disappearing and improves overall availability.
Another important mechanism is Choking and Unchoking.
BitTorrent rewards peers that actively contribute bandwidth. Clients preferentially exchange data with peers that are reciprocating, creating a simple but highly effective incentive structure.
Finally, there is End-Game Mode.
Near the completion of a download, a few remaining blocks often become bottlenecks. To avoid waiting for slow peers, the same block is requested from multiple sources simultaneously. Once one copy arrives, the remaining requests are cancelled.
These algorithms may appear simple individually, but together they dramatically improve swarm performance.
We’ll explore them in much greater detail later in the series.
Where We’re Headed Next
At this point, we have a high-level understanding of the major components involved in a BitTorrent client.
We’ve seen how torrent metadata describes files, how Bencoding represents that metadata, how trackers and DHT enable peer discovery, how peers communicate, and how data is exchanged in pieces across a distributed swarm.
The next step is to begin building.
We’ll start with the lowest layer of the system: Bencoding.
By the end of the next article, we’ll have implemented a parser capable of decoding real torrent metadata, giving us our first tangible step toward a fully functioning BitTorrent client.
Here’s a thought: BitTorrent was built to develop a decentralized file sharing system, but if we have a “TRACKER” that shares peer info, aren’t we still depending on a central authority?
