How Metadata Drives a Database Ingestion Framework
A walkthrough of one execution — and what fifty tables in production taught us along the way.
Behind Every Metadata-Driven Framework
In our previous article, we introduced a metadata-driven ingestion framework and explained how we moved away from building a dedicated pipeline for every database table. We covered the architecture and the metadata model — what tables held what fields, how they fit together. What we left unanswered was how any of it actually did anything.
How does metadata actually drive the framework?
On its own, a row in a metadata table does nothing. Whether it contains source_system = SQL_SERVER, load_type = Delta, or watermark_column = updated_at, it simply sits there as configuration. It doesn't generate SQL, execute pipelines, or move data. Those values only become meaningful when a framework reads them, understands what they represent, and uses them to make execution decisions.If you have not read my previous article here is the link (Stop Building Database Pipelines. Start Building Metadata Driven Database Ingestion Frameworks. | by Divyansh Goyal | Jul, 2026 | Medium).
That’s the part we intentionally left out in the previous article, and it’s the part that matters most. The real capability isn’t the metadata itself, but the framework that interprets those configurations, combines them with the execution flow, and turns them into a working ingestion process. To see how that happens, let’s walk through a real onboarding scenario and follow a requirement from its first metadata entry to a successful execution.
Onboarding a New Requirement
We received a requirement from the client team to onboard 50 Oracle tables into the lake. As intended, the onboarding didn’t begin with development because the framework already supported Oracle Database support from previous integrations, leaving very little to change. Apart from configuring the new source and defining the objects to ingest, the framework was already capable of handling the rest. The next step was understanding what the client expected from this integration.
As we walked through the source system together, building a checklist of items and objects as we went, the questions started coming naturally. Can some tables be loaded incrementally? A few tables can only be extracted through stored procedures. Some require additional filter conditions, while others need overlap days to safely capture late-arriving updates. None of these requirements were new. They were challenges the framework had already evolved to handle through previous integrations.
The client eventually asked the question we were expecting.
“So, what needs to be developed?”
Our answer was simple.
“You just need to create the metadata entries.”
That response almost always led to the next question.
“How can metadata solve all of these different extraction requirements?”
The answer was almost embedded inside the question.
From Metadata to Meaning
We opened the metadata tables that were already part of the framework. Almost every item from our checklist already had a natural place to go, and the conversation quickly shifted from what information needed to be captured to why the framework had been modeled this way.
The connection details were the easiest to understand. Since every ingestion needs both a source and a destination, the Oracle database and ADLS were represented as separate Connection entries. The Job then linked those two connections together, defining what needed to move from the source system to its target. At this point, the framework knew where to read the data from and where it needed to be written.
The next question was more interesting.
“Why not just store the table name directly in the Operation?”
The answer became clear once we looked beyond the table name. An Object is more than just the name of a table. It carries its own characteristics — its schema, optional source query, filter clause, select clause, object type, and other properties that describe the source itself. These attributes belong to the object regardless of how it is eventually ingested. Treating every table as an Object kept those characteristics together and made them available whenever the framework needed them.
That naturally brought us to the Operation entity. If the Object answered what was being extracted, the Operation answered how it should be extracted. Was this a full load or an incremental load? Which CDC column should be used? Was an overlap day required? Should the framework generate a SELECT statement or execute a stored procedure instead? These weren't properties of the table. They were properties of the extraction. Separating those responsibilities kept the metadata model simple while allowing the same object to support different ingestion behaviors.
By the end of the walkthrough, the client was no longer looking at Connection, Job, Object, and Operation as individual metadata tables. They had started seeing the framework as a model where each entity had a clear responsibility. With the metadata in place, the onboarding was complete. The next step wasn’t to write new code — it was simply to let the framework execute what the metadata described.
A Single Execution
By this point, the metadata was in place. The framework knew everything about the ingestion requirement. The next step was to see job in motion.
Every execution begins with two identifiers: a Job ID and an Operation ID.
The Job ID pointed to two connections: an Oracle source and an ADLS destination. Connections returns — host, port, and credentials on one side; a storage account url , credentials to connect on the other. Before a single row was read, the framework already knew where it would connect, where the data would come from, and where it would eventually be written. The execution then created its first audit entry against the same Job ID and Operation ID, giving every event that followed a place to be recorded.
Get Divyansh Goyal’s stories in your inbox
Join Medium for free to get updates from this writer.
The next step was GetQualifiedOperation. It checked that the requested operation was active, confirmed the same was not already Running, marked it as Running, and joining the source object, target object, and operation details into one JSON and pass it into the execution context. Information that had been spread across the Job, Object, and Operation metadata was now assembled into a single execution context: the source table, destination path, container, current CDC values, overlap days, filter clauses, operation script, and the rest of the operation's configuration.
From this point onwards, the execution never needed to read the metadata repository again. Every component that followed worked from the same execution context.
The Execution Takes Shape
Everything the execution needed was already sitting in the context GetQualifiedOperation had assembled. But until this point, we still hadn't answered one question: what query should actually be sent to Oracle? That responsibility belonged to a single function — GetObjectSourceQuery,which pulled the table name, load type, filters, CDC values, overlap days, and every other parameter it needed and generated the query.
For this operation, the metadata described a full load. Nothing more was needed, so the function simply returned:
SELECT *
FROM SALES.CUSTOMERThe query was then handed to the Copy Activity. It connected to Oracle using the connection details already available in the execution context, executed the query, and began reading. Rows started moving. The extracted data landed at the destination path. When the copy finished, the audit entry created at the start of the execution was updated with the rows copied, duration, throughput, and execution status before the operation was marked complete.
The next operation followed the very same execution path. This time, however, the metadata described an incremental load. Before GetObjectSourceQuery was called, the execution retrieved the last processed watermark and the current maximum CDC value from the source. Those values were passed into the function, and the query that came back looked different.
SELECT *
FROM SALES.CUSTOMER
WHERE LAST_UPDATED > :LastProcessedWatermark
AND LAST_UPDATED <= :CurrentMaxCDCThe Copy Activity didn’t know or care that the query had changed. It executed the query exactly as it had before. Once the extraction completed, the latest processed watermark was written back to the operation metadata, ensuring the next execution would resume from exactly where this one finished.
A few tables later, another operation introduced a filter clause — perhaps a regional restriction or a status flag the source team wanted excluded. GetObjectSourceQuery simply folded it into the WHERE condition before handing the query to the Copy Activity. Another operation wanted to execute a stored procedure, so GetObjectSourceQuery returned a stored procedure instead of a generated query. The execution didn't branch into a different pipeline or follow a different path. The Copy Activity executed what it received, and the rest of the execution continued exactly as before.
Fifty tables. Fifty different operations. One execution path.
When the Framework Met Reality
Everything we discussed so far worked exactly as expected during development and testing. The metadata was configured, the framework generated the extraction queries, and the ingestion completed successfully. From our perspective, the onboarding was complete.
The real test came when we moved to production.
One of the first tables to be onboarded was an EDI history table. We configured the metadata, triggered the execution, and expected the load to complete like every other table. Instead, the pipeline kept running for hours without making meaningful progress.
After speaking with the source team, we found out why. The table held nearly 1.2 billion rows — almost twelve years of history. We’d asked for all of it in one go.
SELECT *
FROM EDI_HISTORYThe problem wasn’t the framework. The problem was the extraction strategy — twelve years in one shot was never going to work. So we broke it apart, one year at a time, updating the extraction window through the Operation Script each run.
SELECT *
FROM EDI_HISTORY
WHERE TRANSACTION_DATE >= DATE '2013-01-01'
AND TRANSACTION_DATE < DATE '2014-01-01'Twelve runs, twelve windows, same execution flow every time. Nothing about the pipeline changed. Only the SQL did.
We thought the extraction was solved. Then business checked the Bronze layer against the source, and the data types didn’t match.
Once we understood why the types were drifting — the connector wasn’t preserving precision the way the source database did — the fix was simple: cast every column explicitly before it left Oracle, instead of trusting the connector to get it right. We set that up through a custom Select Clause on the affected objects.
SELECT
CAST(CUSTOMER_ID AS NUMBER(18)) AS CUSTOMER_ID,
CAST(CUSTOMER_NAME AS VARCHAR2(100)) AS CUSTOMER_NAME,
CAST(ORDER_AMOUNT AS NUMBER(18,2)) AS ORDER_AMOUNT,
CAST(LAST_UPDATED AS TIMESTAMP) AS LAST_UPDATED
FROM SALES.CUSTOMEROther tables brought smaller surprises. One had an index on a date column that was not being utilised and giving performance issues, because the default query was casting it to a timestamp before comparison — we did not see that coming, but a subtle change to the object_date_format column saved us that day.
Where This Leaves Us
Everything we had built worked exactly as expected. The onboarding took shape as designed, the metadata described each extraction, the execution followed the same path every time, and the data landed where it was supposed to. From our perspective, the work was done.
Then the first production workloads arrived.
Looking back, none of the situations that followed felt extraordinary. They weren’t failures in the execution or flaws in the design. They were simply the kinds of questions that only appear once a system starts doing real work. A table that carried twelve years of history. Data types that didn’t quite survive the journey unchanged. An index that quietly stopped helping because of a small detail in a generated query.
Every one of those moments gave us two choices: solve the problem in front of us, or solve it in a way that made the framework better for the next problem. We tried to choose the second whenever we could.
That’s the part worth carrying forward more than the entities or the SQL functions. Solving a problem quickly, in the moment, and in isolation will always feel like progress. It’s only later — table fifty, or table two hundred — that you find out whether you were solving the problem, or simply postponing it.
