Medallion Architecture Revisited: What Most People Get Wrong
A technical deep-dive for data engineers and architects who’ve moved beyond the happy-path tutorial
There’s a version of Medallion Architecture that lives in conference slides and vendor documentation. It has three neat layers — Bronze, Silver, Gold — each one a polished step up from the last. Data flows in one direction. Pipelines are clean. Stakeholders are happy.
Then there’s what actually happens in production.
After working through real-world Lakehouse implementations, I’ve seen the same class of mistakes repeat: teams that over-layer, under-model, or misunderstand what each zone is actually supposed to guarantee.Creating three folders or schemas named Bronze, Silver, and Gold does not magically create a scalable data platform. In many organizations, these layers slowly become cluttered, expensive, and difficult to maintain because teams misunderstand the responsibility of each layer.
The real Medallion Architecture is not about storage hierarchy.
It is about progressive data trust.
In this blog, we’ll revisit Medallion Architecture from a practical engineering perspective and explore what most teams still get wrong.
Mistake #1: Treating Bronze as a Staging Area, Not a System of Record
The most foundational error is conceptual. Most tutorials describe Bronze as “raw data” — which is true but incomplete. What they fail to emphasize is that Bronze is your immutable audit log and your recovery point of last resort.
The common mistake looks like this:
# Wrong: overwriting or truncating Bronze on each load
def load_bronze(source_df):
source_df.write.mode("overwrite").parquet("s3://lake/bronze/orders/")This destroys your ability to replay. If a transformation bug corrupts Silver two weeks from now, you cannot reprocess from source without re-pulling from the origin system — which may have changed, been decommissioned, or rate-limited you out.
The correct mental model is: Bronze is append-only by default, and every record that ever entered the lake must be recoverable from it. This means:
# Right: partition by ingestion date, never overwrite
def load_bronze(source_df, batch_id: str):
(
source_df
.withColumn("_ingested_at", current_timestamp())
.withColumn("_batch_id", lit(batch_id))
.withColumn("_source_system", lit("orders_api"))
.write
.mode("append")
.partitionBy("_ingested_at_date")
.parquet("s3://lake/bronze/orders/")
)Notice the metadata columns. Bronze records must carry provenance: when they arrived, from which system, in which batch. Also you can include filename to trace it. Without these, debugging downstream anomalies becomes archaeology. To handle schema drift from source systems, the bronze layer should maintain a consistent schema that can flexibly accommodate changes over time. This ensures stability and simplifies troubleshooting.
The deeper issue: If your Bronze layer doesn’t let you answer “exactly what bytes did we receive from system X at time T?”, it isn’t a Medallion architecture — it’s just a landing zone with extra steps.
Mistake #2: Silver Is Not a Cleaned Copy of Bronze — It’s a Validated, Conformed Domain Model
This is where architectural understanding fractures most severely. Teams build Silver as “Bronze but with nulls dropped and types cast.” That’s not Silver. That’s Bronze with a light rinse.
Silver’s actual job is semantic conformance. It answers the question: “Across all our source systems, what does a canonical customer, order, or product actually mean?"
Consider a company ingesting orders from three systems: a legacy ERP, a Shopify store, and a mobile app. Each has a different representation of “order status”:
Source Status Values ERP 1, 2, 3, 4 Shopify pending, fulfilled, cancelled Mobile CREATED, SHIPPED, DELIVERED, REFUNDED
Bronze faithfully stores all three representations. Silver resolves them:
from pyspark.sql import functions as F
STATUS_MAP = {
# ERP
"1": "pending", "2": "processing", "3": "shipped", "4": "cancelled",
# Shopify (pass-through after normalization)
"pending": "pending", "fulfilled": "delivered", "cancelled": "cancelled",
# Mobile
"CREATED": "pending", "SHIPPED": "shipped",
"DELIVERED": "delivered", "REFUNDED": "refunded"
}
status_mapping = F.create_map(*[F.lit(k) for pair in STATUS_MAP.items() for k in pair]
silver_orders = (
bronze_orders_union
.withColumn("status_normalized", status_mapping[F.col("raw_status")])
.withColumn("status_normalized",
F.when(F.col("status_normalized").isNull(), F.lit("unknown"))
.otherwise(F.col("status_normalized")))
.filter(F.col("order_id").isNotNull()) # Only hard-reject truly invalid records
.withColumn("_silver_processed_at", F.current_timestamp())
.withColumn("_source_system", F.col("_source_system")) # Preserve lineage
)Notice what Silver does not do: it does not drop records because a status is unmapped. It surfaces unknown and lets downstream layers or data quality jobs handle the exception. Silently dropping records in Silver is one of the most dangerous antipatterns in Lakehouse engineering because data loss becomes invisible.
Silver should also enforce schema contracts using tools like Delta Lake’s schema enforcement or Great Expectations, but it should handle violations gracefully — routing bad records to a quarantine table, not silently swallowing them.
# Quarantine pattern: don't drop, redirect
valid_records = silver_df.filter(F.col("status_normalized") != "unknown")
quarantine_records = silver_df.filter(F.col("status_normalized") == "unknown")
valid_records.write.format("delta").mode("append").save("s3://lake/silver/orders/")
quarantine_records.write.format("delta").mode("append").save("s3://lake/quarantine/orders/")Mistake #3: Gold Is a Serving Layer, Not an Aggregation Layer
Teams build Gold tables that look like this: pre-aggregated daily revenue, pre-rolled-up product metrics, materialized pivot tables. Then they hit the first inevitable stakeholder request for a slightly different grain, and suddenly they’re maintaining 14 Gold tables for 12 different BI reports.
Gold is not a cache for pre-computed aggregations. Gold is a domain-specific, access-optimized serving layer.
The distinction matters enormously. Gold tables should be modeled around business entities and access patterns, not around specific queries. This usually means one of two approaches:
Option A — Dimensional Modeling (Kimball): Build fact and dimension tables that serve an entire analytical domain. A fct_orders and dim_customers can serve hundreds of different BI queries without needing schema changes.
-- Gold: fct_orders (wide fact table, grain = one row per order line)
CREATE TABLE gold.fct_orders AS
SELECT
o.order_id,
o.order_line_id,
o.order_date,
o.customer_id,
o.product_id,
o.quantity,
o.unit_price,
o.quantity * o.unit_price AS gross_revenue,
o.status_normalized,
o.source_system,
-- Degenerate dimensions
o.promo_code,
o.channel
FROM silver.orders o
WHERE o.status_normalized != 'cancelled';Now any aggregation — daily revenue, revenue by channel, revenue by promo — is a simple GROUP BY away. You didn't need a Gold table per question.
Option B — One Big Table (OBT) for high-cardinality analytics: For OLAP tools like Superset or Metabase, a denormalized wide table often outperforms joins at query time.
The wrong instinct is to build Gold tables reactively — one per analyst request. The right instinct is to model Gold proactively around stable business concepts, then let BI tools do the aggregation work they were designed for.
Mistake #4: Unidirectional Pipelines Are a Myth in Practice
The canonical diagram shows data flowing Bronze → Silver → Gold. In reality, this breaks down the moment you introduce:
- Late-arriving data: A transaction from three days ago arrives today. Your Silver and Gold tables may have already published metrics that didn’t include it.
- Corrections and restatements: The source system pushes a corrected batch. You need to reprocess Silver and propagate to Gold.
- Reference data enrichment: Gold tables often need to join against slowly-changing dimension data (SCD Type 2) that itself lives in Silver.
Handling these properly requires event-driven reprocessing with change data capture (CDC), not just scheduled batch appends.
With Delta Lake, you can implement this cleanly using the MERGE pattern:
from delta.tables import DeltaTable
silver_table = DeltaTable.forPath(spark, "s3://lake/silver/orders/")
# Upsert: handles both new records and corrections from source
(
silver_table.alias("existing")
.merge(
new_bronze_data.alias("incoming"),
"existing.order_id = incoming.order_id AND existing.source_system = incoming.source_system"
)
.whenMatchedUpdate(set={
"status_normalized": "incoming.status_normalized",
"unit_price": "incoming.unit_price",
"_silver_updated_at": "current_timestamp()"
})
.whenNotMatchedInsertAll()
.execute()
)After a Silver merge, you need to propagate changes downstream. Delta Lake’s Change Data Feed (CDF) makes this tractable:
# Read only changed Silver records since last Gold refresh
silver_changes = (
spark.read.format("delta")
.option("readChangeFeed", "true")
.option("startingVersion", last_gold_version)
.load("s3://lake/silver/orders/")
.filter(F.col("_change_type").isin(["insert", "update_postimage"]))
)Without CDF or equivalent, teams resort to full-table rewrites of Gold on every pipeline run — which is expensive, slow, and operationally fragile at scale.
Mistake #5: No Explicit Data Contract Between Layers
Perhaps the most insidious mistake: layers that are coupled through implicit schema assumptions rather than explicit contracts.
When a Silver pipeline just reads whatever Bronze produces, you have invisible coupling. A source system changes a field name, Bronze faithfully ingests the new schema, and Silver breaks silently — or worse, produces wrong data without failing.
The fix is explicit schema contracts enforced at layer boundaries. With Delta Lake, this is mergeSchema: false combined with schema evolution policies. But the real discipline is defining the contract in code:
from pyspark.sql.types import StructType, StructField, StringType, TimestampType, DoubleType
SILVER_ORDERS_SCHEMA = StructType([
StructField("order_id", StringType(), nullable=False),
StructField("customer_id", StringType(), nullable=False),
StructField("order_date", TimestampType(), nullable=False),
StructField("status_normalized", StringType(), nullable=False),
StructField("gross_revenue", DoubleType(), nullable=True),
StructField("source_system", StringType(), nullable=False),
StructField("_silver_processed_at", TimestampType(), nullable=False),
])
def validate_silver_schema(df):
incoming_fields = {f.name: f.dataType for f in df.schema.fields}
for field in SILVER_ORDERS_SCHEMA.fields:
if field.name not in incoming_fields:
raise ValueError(f"Missing required field: {field.name}")
if incoming_fields[field.name] != field.dataType:
raise TypeError(
f"Field {field.name}: expected {field.dataType}, got {incoming_fields[field.name]}"
)Teams using dbt can enforce this with dbt-expectations or by defining contract: enforced: true in their model YAML — which will fail CI before a bad schema ever reaches production.
Best Practices That Production-Grade Implementations Actually Use
Correcting the five mistakes above gets you to a functional Medallion architecture. The following practices get you to a maintainable one — the kind that survives team turnover, source system surprises, and scale.
BP1: Keep All Bronze Columns as Strings
This is counterintuitive to engineers who want clean typed schemas from day one, but it is one of the highest-leverage decisions you can make for ingestion stability.
The reason: Source systems lie about their types. An API that documents a field as integer will eventually send "N/A". A timestamp field will drift between ISO 8601 and epoch milliseconds across versions. If Bronze casts eagerly, type mismatches cause ingestion failures — and you lose data.
python
# Bronze: cast everything to string on ingest
from pyspark.sql import functions as F
from pyspark.sql.types import StringType
def cast_all_to_string(df):
return df.select([
F.col(c).cast(StringType()).alias(c)
for c in df.columns
])
bronze_df = cast_all_to_string(raw_source_df)
bronze_df.write.format("delta").mode("append").save("s3://lake/bronze/orders/")Type casting is Silver’s responsibility. Bronze’s responsibility is to never lose a record. This separation of concerns is worth encoding in your team’s engineering norms document — because the instinct to “clean up as early as possible” is strong and consistently wrong at the Bronze layer.
Get Divyansh Goyal’s stories in your inbox
Join Medium for free to get updates from this writer.
The only exception: binary payloads (raw JSON blobs, Avro bytes, Protobuf) that you want to store unparsed — handled in the next practice.
BP2: Use the Variant Column Type for Semi-Structured Bronze Data
When your source is a JSON API, a webhook payload, or a Kafka topic with evolving schemas, storing data as typed columns in Bronze is a trap. The schema will change. Your ingestion job will break.
The modern answer is the Variant column type — available natively in Databricks (Delta Lake) and Snowflake — which stores semi-structured JSON in a self-describing binary format optimized for selective field extraction without requiring a full schema at write time.
-- Databricks: create a Bronze table with a Variant column
CREATE TABLE bronze.events (
_ingested_at TIMESTAMP,
_batch_id STRING,
_source_system STRING,
raw_payload VARIANT -- entire JSON blob stored here, schema-free
);
-- Insert raw JSON without any schema binding
INSERT INTO bronze.events
SELECT
current_timestamp() AS _ingested_at,
'2024-01-15-batch-001' AS _batch_id,
'mobile_app' AS _source_system,
PARSE_JSON(raw_json_string) AS raw_payload
FROM raw_staging;In Silver, you extract only the fields you need — and that extraction logic is versioned in your transformation code, not baked into the Bronze schema:
sql
-- Silver: selectively extract fields from Variant as needs evolve
SELECT
raw_payload:event_id::STRING AS event_id,
raw_payload:user_id::STRING AS user_id,
raw_payload:event_type::STRING AS event_type,
raw_payload:properties.amount::DOUBLE AS amount,
raw_payload:occurred_at::TIMESTAMP AS occurred_at
FROM bronze.events
WHERE raw_payload:event_id IS NOT NULL;The Variant approach means your Bronze layer genuinely never breaks on schema changes. A new field from the source? It lands in the Variant and sits there until Silver decides to use it. A field gets renamed? Old records still have the old name; Silver handles the mapping. You absorb schema drift without pipeline failures.
BP3: Use Delta Tables for Versioning and Time Travel — Not Plain Parquet
Plain Parquet gives you files. Delta Lake gives you a table with a transaction log — and that transaction log is what makes Medallion architecture recoverable in practice, not just conceptually.
The three Delta capabilities that matter most across Medallion layers:
Time Travel: Query any layer at any historical version. When a stakeholder reports “the numbers looked different last Tuesday,” you reproduce last Tuesday’s state exactly — without any manual backup infrastructure.
-- Reproduce Silver as it existed at a specific point in time
SELECT * FROM silver.orders VERSION AS OF 42;
SELECT * FROM silver.orders TIMESTAMP AS OF '2024-01-10 09:00:00';
-- Audit: what changed between two versions?
SELECT * FROM silver.orders VERSION AS OF 50
EXCEPT
SELECT * FROM silver.orders VERSION AS OF 42;OPTIMIZE + Z-Ordering: Small files are the silent killer of Lakehouse query performance. Delta’s OPTIMIZE compacts them; ZORDER co-locates related data physically on disk, making predicate pushdown dramatically more effective for high-cardinality filter columns.
-- Run after significant batch loads on Silver and Gold
OPTIMIZE silver.orders ZORDER BY (customer_id, order_date);
OPTIMIZE gold.fct_orders ZORDER BY (order_date, channel);Vacuum with Deliberate Retention Windows: Storage costs compound silently. VACUUM removes files no longer referenced by the transaction log, but the retention window controls how far back Time Travel works. Set it deliberately — not the 7-day default.
-- Bronze: keep 90 days (long recovery window)
ALTER TABLE bronze.orders SET TBLPROPERTIES ('delta.deletedFileRetentionDuration' = 'interval 90 days');
VACUUM bronze.orders RETAIN 2160 HOURS;
-- Gold serving tables: 14 days is usually enough
ALTER TABLE gold.fct_orders SET TBLPROPERTIES ('delta.deletedFileRetentionDuration' = 'interval 14 days');
VACUUM gold.fct_orders RETAIN 336 HOURS;Make retention a first-class configuration parameter per table. The decisions you make here determine your recovery capabilities during the next incident.
BP4: Handle SCDs in Silver, Not Gold
Slowly Changing Dimensions are almost always implemented in the wrong layer. Teams either flatten them into Gold (losing history) or handle them at BI query time (making dashboards do transformation work they shouldn’t).
SCD Type 2 belongs in Silver. Silver’s job is to produce a temporally accurate representation of business entities — which means tracking when a customer changed their address, when a product changed its category, or when an employee moved to a different cost center.
Here is a clean SCD Type 2 implementation using Delta Lake MERGE:
python
from delta.tables import DeltaTable
from pyspark.sql import functions as F
silver_customers = DeltaTable.forPath(spark, "s3://lake/silver/dim_customers/")
# Incoming: new snapshot of customer records from Bronze
incoming = (
spark.read.format("delta").load("s3://lake/bronze/customers/")
.filter(F.col("_ingested_at_date") == today)
)
# Step 1: Close out current records that have changed
(
silver_customers.alias("existing")
.merge(
incoming.alias("incoming"),
"""
existing.customer_id = incoming.customer_id
AND existing.is_current = true
AND (
existing.email != incoming.email OR
existing.address != incoming.address OR
existing.segment != incoming.segment
)
"""
)
.whenMatchedUpdate(set={
"is_current": "false",
"valid_to": "current_timestamp()"
})
.execute()
)
# Step 2: Insert new versions of changed records + net new customers
new_versions = (
incoming
.join(
silver_customers.toDF().filter(F.col("is_current") == True),
on="customer_id",
how="left_anti" # records with no current match = new or just-expired
)
.select(
F.col("customer_id"),
F.col("email"),
F.col("address"),
F.col("segment"),
F.current_timestamp().alias("valid_from"),
F.lit(None).cast("timestamp").alias("valid_to"),
F.lit(True).alias("is_current"),
)
)
new_versions.write.format("delta").mode("append").save("s3://lake/silver/dim_customers/")Gold then joins against WHERE is_current = true for current-state reporting, or against the full SCD table with valid_from / valid_to filtering for point-in-time historical analysis. No history is lost, and all temporal complexity lives in exactly one Silver model.
BP5: Use Materialized Views for Gold — Sparingly and Deliberately
Materialized views (available in Databricks, Snowflake, and BigQuery) compute and persist query results, refreshing automatically when upstream data changes. They are powerful for Gold — and a maintenance trap if used without discipline.
Where they genuinely earn their place:
Aggregations that are expensive to recompute but queried frequently (daily revenue rollups, 30-day rolling averages), large fact-dimension joins where the result is stable, and BI dashboards with strict latency requirements.
sql
-- Databricks: materialized view for Gold revenue summary
CREATE MATERIALIZED VIEW gold.mv_daily_revenue
AS
SELECT
DATE_TRUNC('day', order_date) AS revenue_date,
channel,
SUM(gross_revenue) AS total_revenue,
COUNT(DISTINCT order_id) AS order_count,
AVG(gross_revenue) AS avg_order_value
FROM silver.orders
WHERE status_normalized NOT IN ('cancelled', 'refunded')
GROUP BY 1, 2;Where they become a problem:
Materialized views create an implicit dependency graph that’s easy to lose track of. If mv_daily_revenue references silver.orders and someone alters that table's schema, the MV can silently become stale or break without obvious errors surfacing. Always document MV dependencies explicitly — in dbt lineage, a data catalog, or at minimum a README that lives with the DDL.
Also: don’t materialize at the wrong grain. A materialized daily summary that an analyst re-aggregates to monthly is doing double work and restricts query flexibility. Define the grain of each MV by the access pattern, not by what’s convenient to compute.
A good rule of thumb: if an analyst could get the answer they need by writing a GROUP BY against a well-modeled Gold fact table in under 5 seconds, you don't need a materialized view yet.
BP6: Implement Idempotent Pipeline Runs Across All Layers
Every pipeline in every layer should be safely re-runnable without producing duplicate records, corrupted state, or side effects. This sounds obvious. It is widely ignored — and the gaps show up exactly during incidents when you most need to re-run pipelines quickly.
The pattern: identify a natural idempotency key for each layer and enforce it explicitly.
python
# Bronze: idempotency via batch_id — delete-then-reinsert pattern
spark.sql(f"DELETE FROM bronze.orders WHERE _batch_id = '{batch_id}'")
source_df.write.format("delta").mode("append").save("s3://lake/bronze/orders/")
# Silver: idempotency via MERGE on business key
# (already demonstrated in the MERGE pattern above)
# Gold: idempotency via partition overwrite — touches only the affected date
(
gold_df
.write
.format("delta")
.mode("overwrite")
.option("replaceWhere", f"revenue_date = '{process_date}'")
.save("s3://lake/gold/fct_daily_revenue/")
)The Gold replaceWhere pattern is particularly important. It overwrites only the partition relevant to the current run, leaving all other partitions untouched. This is both idempotent and efficient — and it's meaningfully different from a full table overwrite, which reprocesses everything even when only one day changed.
BP7: Instrument Every Layer Boundary with Data Quality Metrics
Most teams add data quality checks that fire alerts when something is wrong. That’s necessary but insufficient. You also need continuous DQ metrics that let you observe degradation trends before they become incidents.
Three metrics worth tracking at every layer boundary:
python
from datetime import datetime
def compute_layer_dq_metrics(df, layer: str, table: str, batch_id: str) -> dict:
total = df.count()
return {
"layer": layer,
"table": table,
"batch_id": batch_id,
"row_count": total,
"null_rate_primary_key": df.filter(F.col("order_id").isNull()).count() / total,
"duplicate_rate": 1 - (df.dropDuplicates(["order_id"]).count() / total),
"measured_at": datetime.utcnow().isoformat()
}
# Persist to a dedicated DQ metrics table
metrics_df = spark.createDataFrame([compute_layer_dq_metrics(silver_df, "silver", "orders", batch_id)])
metrics_df.write.format("delta").mode("append").save("s3://lake/_meta/dq_metrics/")Persist these to a _dq_metrics Delta table and build a simple dashboard on top. A spike in null_rate_primary_key on Bronze points to a source system issue. The same spike appearing only on Silver is a transformation bug. Layer-aware metrics tell you exactly where to look without manual log archaeology.
What Good Looks Like: A Revised Mental Model
After correcting the mistakes and applying these practices, a rigorous Medallion implementation looks less like a pipeline and more like a layered set of guarantees:
The architecture earns its name only when these guarantees hold under failure conditions — late data, schema drift, source corrections, and pipeline restarts. If any layer can silently lose, corrupt, or misrepresent records, you have a data lake with good naming conventions, not a Medallion architecture.
Final Takeaway
The most successful Medallion Architectures are not necessarily the most complex.
They are the ones that:
- Preserve raw fidelity in Bronze
- Create reliable entities in Silver
- Deliver trusted analytics in Gold
- Optimize for scalability and governance from day one
Modern lakehouse engineering is no longer just about moving data.
It is about building resilient, trustworthy, and scalable data products
Have a different take on where Medallion implementations break down? I’d love to hear about the edge cases you’ve run into in production
