Stop Building Database Pipelines. Start Building Metadata Driven Database Ingestion Frameworks.
How we turned repeated variations into reusable platform capabilities using a metadata-driven architecture.
The Problem Nobody Talks About
Our database ingestion pipelines worked.
Until they didn’t.
The first few pipelines were straightforward. Connect to the source, extract the required tables, land the data in ADLS, and move on. As new systems were onboarded, we simply copied an existing pipeline and adjusted it for the new source.
Then came the requests.
“This table is a little different…”
A source system only wanted to expose stored procedures instead of direct table access. Another required a custom filter to limit the extraction. Some tables supported incremental loads, while others could only be extracted as full loads. Individually, none of these requests seemed unreasonable. But each one nudged the orchestration in a slightly different direction.
The real pain wasn’t building another pipeline.
It came later.
When a consumer reported inconsistent dashboard numbers, debugging became an investigation. Which pipeline ran? Which query was executed? Which filter was applied? What changed between environments? After every complaint or production incident, tracing the execution path and performing RCA took far longer than it should have because every pipeline had evolved differently, and we lacked a consistent way to audit what had actually happened.
One experience stayed with us.
Everything worked perfectly through Development, SIT, and UAT. Then, just before Production, the source team informed us that direct table access wouldn’t be allowed. We would need to extract through stored procedures instead. What should have been a deployment activity forced us back into implementation.
That was the point where we realised the problem wasn’t the source systems.
It was the way we were building pipelines.
From File Ingestion to Database Ingestion
The challenges we were seeing with database ingestion felt familiar.
A few months earlier, we had built a metadata-driven framework for file ingestion. Looking back, the biggest outcome wasn’t the framework itself. It was the way it changed how we approached platform engineering.
We weren’t building ingestion pipelines anymore.
We were building ingestion capabilities.
Every new source no longer started with a blank pipeline. Instead, we asked a different question:
Can the platform already do this?
If the answer was yes, it became another metadata configuration. If the answer was no, we didn’t immediately create another pipeline — we asked whether it was a capability the platform should support for every future source.
That mindset naturally carried over to database ingestion.
The objective wasn’t to build another generic pipeline. It was to understand what capabilities a database ingestion platform should provide and which of those belonged in metadata instead of orchestration.
Before writing a single pipeline, we decided to study the ones we already had.
Looking for the Differences
Before starting the pilot, we spent a couple of weeks reviewing the database ingestion pipelines already running in our platform. We expected to find very different implementations. Instead, we found that almost every pipeline was doing the same thing: connect to a source system, execute a query or stored procedure, and land the extracted data into ADLS as Parquet. Whether the source was SQL Server, Oracle, DB2, or Teradata made surprisingly little difference to the orchestration.
The differences appeared only after we looked at the extraction itself. Some operations required full loads, others incremental. A few relied on stored procedures, while others queried tables directly. Some needed custom filters or selected only a subset of columns. Individually, these were perfectly reasonable requirements. Collectively, they explained why pipelines that looked identical on the surface had gradually started diverging.
Turning Variations into Capabilities
The next question almost asked itself.
If the orchestration wasn’t really changing, where should these differences live?
Get Divyansh Goyal’s stories in your inbox
Join Medium for free to get updates from this writer.
We already had part of the answer. Our file ingestion framework had taught us that pipelines shouldn’t own configuration — they should consume it. So instead of adding another branch into the orchestration every time a source behaved differently, we started asking whether that difference could simply be described through metadata.
We didn’t try to redesign the framework overnight. Instead, every variation became an opportunity to simplify the orchestration. Whenever we found logic that was specific to a source rather than common to every extraction, we asked a simple question:
Can this become a capability instead?
If the answer was yes, we removed it from the pipeline and described it through metadata. Full versus incremental loads, custom filters, generated queries, stored procedures, destination paths — one by one, they stopped being branches in the orchestration and became capabilities the platform could execute.
Note: The metadata model used throughout this framework — covering Jobs, Operations, Connections, and Objects — was introduced in my previous article on metadata-driven file ingestion. Rather than repeating that design here, I’ll focus on how the same foundation enables database ingestion.Designing a Metadata-Driven Lakehouse Architecture | by Divyansh Goyal | Medium
One example was query generation.
Rather than storing complete SQL statements in metadata, the metadata simply described what needed to be extracted. The framework then invoked a query generator, which interpreted those settings and produced the SQL to execute. For operations that required complete control, it simply returned the configured stored procedure instead. The orchestration never needed to know the difference.
Over time, the pipeline became smaller, while the platform became smarter.
The Design Principles We Chose
As the framework evolved, we found ourselves coming back to the same principles whenever a new requirement appeared.
- Rule #1 — No hardcoding. If it can change between two operations, it belongs in metadata, not the orchestration.
- Rule #2 — The orchestrator coordinates, metadata decides. Business logic and execution behavior come from metadata. The orchestrator simply executes them.
- Rule #3 — Repeated variations become capabilities. If the same exception appears repeatedly, it deserves to become a reusable platform capability.
- Rule #4 — Audit everything. Every execution should be traceable. Given a Job ID and Operation ID, we should be able to reconstruct exactly what happened, even years later.
A Single Execution
By now, the framework had all the pieces it needed. The only thing left was execution.
Every pipeline execution begins with two identifiers: a Job ID and an Operation ID. From there, the orchestrator simply follows the metadata.
- Using the Job ID and Operation ID, it retrieves the source and target connection details, source object, destination path, and execution configuration from the metadata repository.
- If the operation is configured for incremental extraction, the framework first retrieves the latest processed watermark (CDC value). That value is then passed to the query generator so the extraction boundary is determined consistently for every execution.
- The framework invokes the query generator, which reads the operation metadata and either constructs the SQL dynamically or returns the configured stored procedure.
- The generated query is passed to the extraction activity, which connects to the source database and lands the result directly into ADLS as Parquet files.
- Throughout the execution, audit information is continuously recorded — from the moment an operation starts until it completes. Execution status, rows copied, throughput, duration, and any errors are all captured against the Job ID and Operation ID.
- Once the files reach Landing, the responsibility of the database ingestion framework ends. From there, Autoloader takes over, and the rest of the Lakehouse journey continues exactly as it does for every other source.
By separating description, execution, and auditing, the orchestration remains remarkably small. It doesn’t need to know whether it’s extracting from SQL Server, Oracle, or DB2, whether the operation is full or incremental, or whether the source prefers tables or stored procedures. It simply executes the operation described by the metadata.
Looking Back
Looking back, the framework wasn’t really about reducing the number of pipelines.
It was about changing where change happened.
In the beginning, every new requirement pulled us back into the orchestration. A different extraction strategy, another custom filter, a production constraint, or a stored procedure all meant opening the pipeline again. Over time, we realised we weren’t solving those problems — we were repeatedly accommodating them.
The metadata-driven approach changed that perspective.
Instead of asking, “Which pipeline needs to change?”, we started asking, “Is this another capability the platform should provide?” That single question influenced almost every design decision that followed.
Today, onboarding a new source is rarely a development exercise. Connection details are updated in one place. Extraction behavior is described through metadata. Repeated variations become reusable capabilities. The orchestration remains focused on one responsibility: coordinating execution.
The framework isn’t perfect, and it continues to evolve. New requirements still challenge our assumptions. The difference is that we’re no longer building another pipeline every time.
We’re improving the platform that builds them.
