Building a Distributed Object Storage Engine

Abdelkader Ould hennia

3 min read

·

1 day ago

--

Listen

Share

High-Level Architecture Overview

At its core, a distributed object storage system splits responsibilities into two separate layers:

1. The Control Plane (The Brain / Coordinator): Manages requests, tracks file blueprints, coordinates replication, and ensures data integrity.
2. The Data Plane (The Warehouses / Storage Nodes): A cluster of physical servers whose sole job is to safely store raw binary data on disk.

Core Components & Their Responsibilities

The API Gateway / Coordinator

The entry point for all client traffic. When a client initiates an upload, the coordinator acts as the master orchestrator:

• Chunking: Splits large files (e.g., a 150MB video) into manageable, uniform pieces (e.g., 50MB chunks).
• SHA-256 Hashing: Generates a cryptographic digital fingerprint for every chunk. This hash acts as both an integrity check and the physical filename on the storage node’s disk.
• Replication & Distribution: Sends copies of each chunk to different physical storage nodes to ensure fault tolerance.
• Metadata Registration: Writes the file manifest and mapping details into the metadata database.

The Storage Nodes

These are worker servers that hold the actual file bytes.

• Content-Addressable Storage: Storage nodes do not care about file names like video.mp4. They save raw binary chunk data directly onto their disk file systems using the SHA-256 hash as the actual filename (e.g., /data/chunks/e3b0c44298fc1c14…)
• Blind Storage: They act purely as secure binary storage units without needing to know how chunks fit together into a broader file.

The Metadata Store (The System Ledger)

Implemented via an RDBMS or Key-Value store, this database holds the system’s blueprints and maps. It stores zero heavy file data; instead, it tracks:

watermarked_video_2979282996657630717.mp4

• File Registry: The user-facing filename (e.g., video.mp4) and its unique file ID.
• Chunk Locations & Indices: Which sequence index maps to which chunk, along with its expected SHA-256 hash.
• Replica Count: Rules ensuring each chunk has multiple backups across distinct nodes.
• Node IP Addresses: The exact network locations of the storage nodes holding each chunk.

3. Step-by-Step Workflows

The Upload Workflow

1. Request Intake: The client sends an upload request containing the file to the API Gateway / Coordinator.
2. Chunking & Hashing: The coordinator splits the file into pieces and passes each piece through the SHA-256 hashing algorithm to generate its unique ID.
3. Replication & Storing: The coordinator distributes the binary chunks across multiple physical storage nodes. The nodes save the raw bytes to disk using the SHA-256 hash as the filename.
4. Metadata Recording: Once the chunks are safely written, the coordinator writes the complete mapping (file name, chunk hashes, replica counts, and node IP addresses) into the metadata database.

The Download & Verification Workflow

1. Metadata Lookup: When a client requests a file, the coordinator queries its metadata store to retrieve the file’s chunk layout, expected SHA-256 hashes, and hosting node IP addresses.
2. Streaming & Real-Time Hashing: The coordinator requests the hash-named chunks from the respective storage nodes. As the raw binary bytes stream back over the network, the coordinator runs them through the SHA-256 hashing algorithm in real-time.
3. Integrity Verification:

• The newly computed hash is compared against the expected text hash stored in the metadata database.
• If a match occurs, the chunk is verified as 100% uncorrupted and safe from silent disk “bit-rot”.
• If a mismatch occurs due to disk corruption, the system discards the bad chunk and fetches a clean replica from an alternate node.

1. Reconstruction: Once all chunks are verified and gathered, the coordinator stitches them back together in the correct sequence and streams the complete file back to the client.

4. Why This Architecture Matters

By decoupling metadata tracking from binary physical storage, this architecture provides:

• High Availability & Fault Tolerance: If a storage node goes offline, backup replicas hosted on other nodes prevent data loss.
• Built-In Data Integrity: Content-addressable hashing ensures that silent hardware degradation or file corruption is caught immediately on read.
• Scalability: Storage nodes can be horizontally scaled infinitely just by adding more machines to the cluster pool, while the metadata coordinator manages the global routing table.

Written By Engineer Abdelkader Ould Hennia

LinkedIn: https://www.linkedin.com/in/abdeloh/

Twitter (X): https://x.com/abdeloh72

Github: https://github.com/Kader1680

Youtube: jeolardo

Press enter or click to view image in full size