Object Storage (BLOBs) Explained for System Design
What is Object Storage (BLOB)?
Object Storage is a special type of storage designed to handle massive amounts of unstructured data.
If you’re designing for the cloud or dealing with massive datasets, then you’ll need to use some sort of object storage there.
Press enter or click to view image in full size
These are the services like:
- Amazon S3 — Probably the most popular one, S3 is a Simple Storage Service that is used for storing large files and retrieving them quickly.
- Google Cloud Storage — Google Cloud offers its object storage service called Google Cloud Storage.
- Azure Blob Storage — Azure also provides a service called Azure Blob Storage.
Press enter or click to view image in full size
As you can see, some services, like Azure, also call it BLOB storage, which stands for Binary Large Object.
But What is an Object?
So, what’s an object, exactly? It’s just a chunk of data. It could be an image, a video, or even an entire virtual machine.
Press enter or click to view image in full size
And each object comes with a few essential bits of information:
- Data: The actual binary content that you’re storing.
Press enter or click to view image in full size
- Unique Identifier: The object’s address within the storage system.
Press enter or click to view image in full size
- Metadata: Information about the object (filename, type, size, creation date, etc.)
Press enter or click to view image in full size
Object Storage vs. Databases vs. File Systems
But why don’t we use our database to store these large files?
Well, we still use our database to store the URI and some more metadata about these image or video files, but the actual file is stored in Object Storage.
Press enter or click to view image in full size
If you throw a bunch of huge files into your database, that will raise performance issues, and our queries will become slower to execute.
Press enter or click to view image in full size
And what about File Systems? Can we store these images and videos in some file storage like Google Drive or OneDrive, or what if we store them in our server’s file system and read them from there?
Press enter or click to view image in full size
File systems are great for local storage and organization, but they don’t scale as easily as object storage.
Get Hayk Simonyan’s stories in your inbox
Join Medium for free to get updates from this writer.
Why? — Because file systems have a hierarchical structure, meaning they are structured like trees with folders, whereas object storage is flat and unstructured — which gives them the advantage of being highly scalable and fast for file retrievals.
Press enter or click to view image in full size
Folders in BLOB Storage?
So, does this mean that there are no folders in object storage?
Many platforms like AWS S3 offer a “folder-like” experience by using delimiters in object names (e.g., “photos/vacation/beach.jpg”) — meaning they split the path to object with forward slashes and represent them as folders.
Press enter or click to view image in full size
But this is just an illusion to make it easier to use — technically, there are no folders in object storage.
Unique Identifiers
As we saw, each object gets a unique identifier (think of a bucket as a container for objects). It’s usually unique within the bucket (your object’s container) and the specific storage service, like in S3.
However, since bucket names must be unique across all AWS accounts, your object names are unique globally, not just inside your bucket.
Press enter or click to view image in full size
Interacting with Object Storage
Most object stores offer a RESTful API for interacting with objects. You send HTTP requests (PUT, GET, DELETE) to manipulate data.
Uploading an object
For instance, you can upload objects to S3 storage using a POST request or by using the AWS platform directly.
Press enter or click to view image in full size
Retrieving an object
Similarly, you can read your uploaded objects with a GET request or by using the AWS console.
Press enter or click to view image in full size
Updating an object
However, editing is typically done by replacing the entire object with a new version. This is different from file systems, where you can modify parts of a file in place.
To edit a file in object storage, you typically remove it with a DELETE request and then create a new updated one with a POST request.
Press enter or click to view image in full size
In addition, there are many libraries and SDKs that make this process easier in different programming languages.
Press enter or click to view image in full size
Summary
To conclude, object storage is designed for speed — it quickly locates objects based on their identifiers.
In addition to that, many object storages use a distributed architecture to distribute traffic across multiple storage nodes.
Frequently accessed objects are stored in faster memory (cache) for even quicker retrieval.
Press enter or click to view image in full size
Many object storage platforms offer security features such as signed URLs and signed cookies to control access to your objects. If you want to learn more about how these features work, check out my upcoming posts.
