Uber’s Move from Postgres to Schema less Datastore
When your scale is too high sometimes the design decisions you took may not work out. The same happened with Uber, their Postgres Master-Slave is working fine but they know that it doesn’t scale as Postgres can handle greater no of reads but not the writes and it’s handled at the Master instance. Uber decided to re-think their data storage system; their main focus was to solve these problems and find answers for them
- If any issue occurs, my team should be able to resolve the issue in no time
- System should Linearly scale, if 1 server handles 100 req/sec then 2 servers should handle 200 req/sec and so on
- If one node fails other nodes should pick up write
- Same index support as Postgres
- Triggers should be present, if one event happens other downstream services should be able to pick that change
Core
Uber team just don’t randomly picked Mongodb or Cassandra for solving their issues, because they need a team who can resolve those issues when they are flagged at 3:00 am. So they did something opposite they used MySQL for having Schema less datastore. Well you may think that MySQL is a Schema related database, let’s understand bit by bit. The team can have multiple interests on using fancy databases but architecting the same with the tools which team can manage is how a great decision would look like.
Cell
This is core part of the Uber system, smallest data unit they defined in their system. This entirely defines how the data is stored in the database. This cell contains these data fields.
- added_id; int, auto_increment: This just resembles uniqueness of the row
- row_key;uuid: This is the original data uuid, for now consider it is same as id for another schema table, example: trip table or user table
- column_name; string: This is the column name in schema table, let’s say there is a table with trip(id, distance, status, etc) these are column names
- ref_key; int: This resembles the version number with the combination of (row_key, column_name), if there is any update to that particular row_key row present in schema table then here it is not updated another version is created
- body; blob: The data which we want to actually store is present here. Compression algorithms are used inorder to reduce the size of the storage.
- created_at; timestamp: This is just the row creation time
This is what a cell means, collection of different data types.
Dry Run:
Let’s consider there is a trip table, here BASE, STATUS, NOTES, FARE ADJS is of type Cell. There are multiple versions of individual columns.
Get Pavaneeshwar’s stories in your inbox
Join Medium for free to get updates from this writer.
Let’s take an example of one column which is STATUS
step-1: When a new ride is confirmed then STATUS value will be PENDING,
cell structure:
added_id: 1
row_key: trip_uuid_1
column_name: STATUS
ref_key: 1
blob: {
"status":"PENDING"
}
created_at: 12:00 amstep-2: When the rider confirms with a otp if goes into PROCESSING
cell structure:
added_id: 2
row_key: trip_uuid_1
column_name: STATUS
ref_key: 2
blob: {
"status": "PROCESSING"
}
created_at: 12:05 amWe can see that ref_key is incremented and blob is populated with a new value, this is how it works in case of updates for the row instead of updating it creates a new row this also acts as a log
Indexes
They also mentioned that indexes should be supported, but making blob indexes can be overwhelming on Json structures, they also mentioned that most of the queries are on row_key but if they want to query on blob then they can de-normalize the blob and store separately and index those particular columns
blob: {
"name": "pavan",
"phone_number": "222222",
}
Instead of something like this, they store in different table
Meta-data-users-blob:
id: uuid
name: "pavan"
phone_number: "222222"
Index on name and phone_number
Users:
blob: meta-uuid
In this way they can store to improve index queries on those tables.
Architecture
Worker node, Responsible for handling client requests and aggregating results over multiple storage nodes for results, worker nodes can scale as they are stateless.
Storage node, Total nodes shards are 4096, the data divided on these shards. This is done via row_key as a shard key. Here every shard contains one Master and multiple Slave nodes.
Requests
Read and write request is handled by worker nodes and collect aggregated results from storage nodes.
Read Requests: Reads can be served from any node it can be master or the slave. All the data is replicated to the slave nodes from the master node within 200ms as mentioned. There is a chance of stale data while replicating or if master dies instantly after taking the write this is not sent to other slave nodes. This uses circuit breaker pattern if a parictular slave node is not responding for reads then it marks as unhealthy and proceeds with another node.
Write Requests: Writes are always done to the master, if slave is down this doesn’t effect but if master is down still it will be able to accept writes but now another slave will become master and then writes go through it. What if primary goes down just after accepting the write.
Buffer Writes: There is something called buffer writes. Before a write request is written to a master it saves in secondary master. Then writes to primary master and then slave. Then a background worker searches for the records whether they are present in slave from secondary master if present delete from secondary master. If primary master goes down then it tells the worker node to wait until master is live or catch up by another slave.
That’s it about uber storage decision how to migrate from Postgres to MySQL making it a Schemaless in a full relationship based database.
