Member-only story
How I Cut a Data Pipeline’s Cost with DuckDB and Batching
A practical migration story about reducing unnecessary infrastructure, improving throughput, and measuring the trade-offs.
The most expensive part of a data pipeline is not always the part doing the most work.
Sometimes, it is the database sitting behind a perfectly ordinary transformation job. Sometimes, it is a cloud warehouse waking up to process a few files, an application making thousands of tiny queries, or a pipeline moving data between systems simply because that is how the original architecture was designed.
I ran into an engineering problem that looked like a performance issue but turned out to be an architectural one.
The pipeline was processing files, cleaning records, joining datasets, and producing analytical outputs. Nothing about the workload seemed unusual. Yet the infrastructure supporting it was doing more work than the actual transformation required.
The obvious response was to optimize the queries.
That was not the first thing I changed.
I started by questioning whether the workload needed that infrastructure at all.
