Member-only story
8.2 minutes, and you will understand how most data systems execute joins.
From Spark and Snowflake to BigQuery, here is how joins are built
I only publish 1/4 of my deep-dive data engineering articles on Medium. Access the full repository here.
Intro
If you need to write SQL for the paycheck (:D), you might be familiar with the join operations. From the left join to the full outer join, data is normalized into separate tables to streamline the ingest, store, and manage process. When insight needs to be gathered from multiple tables, we join them.
The from clause, the join kind (e.g., LEFT JOIN), the join conditions, and 10 minutes later, the result is returned to us.
The beauty of SQL is that you don’t need to care much about the physical implementation of how SQL is executed. The optimizer will do it.
However, understanding how things work behind the scenes will help us, especially data engineers, to debug and optimize our data workload more efficiently.
And I think understanding joins is worth our time, given that we encounter them almost daily.
