My Data Pipeline Worked Until the Data Changed

The code was correct. The pipeline was healthy. The data had other plans.

Maximilian Oliver

8 min read

·

Oct 6, 2026

--

1

Listen

Share

Press enter or click to view image in full size

When I first built data pipelines, reliability meant one thing: the job completed successfully.

Extract → Transform → Load → Schedule → Monitor

If the pipeline was green, I considered it healthy.

Then the data changed.

Column disappeared
New value appeared
Timestamp format changed
Duplicates arrived
Old data was resent

The Python code was fine. The scheduler was fine. The database was fine.

The pipeline was still green.

The data was simply no longer what the pipeline expected.

That changed how I think about reliability.

A pipeline isn’t reliable because the code runs.

It’s reliable when it can detect, understand, and safely handle changes in the data.

1. My First Pipeline Assumed the World Was Stable

The first version of a pipeline is usually straightforward.

Something like:

Source
  |
  v
Extract
  |
  v
Transform
  |
  v
Load
  |
  v
Warehouse