Your SOX Control Passed. The Code Changed Anyway.
A field-tested approach for re-testing SOX controls when the AWS pipeline code underneath them keeps changing. I call the habit “No Silent Drift.”
Most of what’s written about SOX on AWS sits at two extremes. On one end: platform tooling — AWS Config, CloudTrail, Glue Data Quality. On the other: generic SOX 404 testing templates that could apply to any system on earth.
The middle is where I actually spent my days, and it’s barely written down:
How do you keep automated financial controls valid when the code underneath them won’t hold still?
It’s a practical habit I worked out on a live engagement — because the obvious approach kept letting things slip past me. I’m sharing it because I couldn’t find it written down anywhere and if you test controls over a modern financial general ledger data pipeline, you will hit the same problem.
I started calling it No Silent Drift.
The pipeline, and its three controls
At a regulated financial-services firm, financial data flowed through a pipeline on AWS. Three control points sat along it, in the order the data moves:
- An ingestion Lambda. When a source file lands in Amazon S3, this Lambda validates and prepares it before anything else touches it. It checks the file name, normalizes several source formats into one uniform shape, validates the file’s format and required fields, and the important part balances the file against its own trailer - the record count and the summed debit/credit totals. It also stamps every row with a unique record ID and the file with a batch ID.
This is the completeness anchor of the whole flow - the file is proven balanced before a single value is transformed.
2. A Glue job. An AWS Glue (PySpark) job then converts each chart-of-accounts segment for example — company, natural account, cost center, product group, intercompany, geography, and the rest — from the source coding into the corporate Chart of Accounts.
3. A Glue Data Catalog table. Finally, a Glue table defines how the downstream reporting system reads the curated output like column names, data types, order, and partition key.
Each link is a different kind of control. The Lambda gates what gets in. The Glue job transforms it correctly. The Glue table defines how it’s read. If any one of them drifts, the numbers downstream are wrong — quietly, with nothing obviously broken.
How I tested them the first time
Most data-control testing samples a batch of rows and eyeballs the values. I didn’t. I tested at the source-code and schema-definition level, because for an ingestion-and-conversion pipeline, that’s where completeness and accuracy actually live.
The ingestion Lambda. I read the code and confirmed each validation fails closed — a bad file name is rejected, the format check stops a malformed file, and the record-count and summed-total checks reconcile the file against its trailer and halt on a mismatch. Nothing invalid gets to proceed.
The Glue job. I went segment by segment through the PySpark code and, for each chart-of-accounts segment, confirmed the code did exactly what the approved mapping specification said: the right source key, the right reference lookup, the right fallback when a value didn’t map (assign the defined suspense value and preserve the original — never drop the row, never leave it blank), and a hard-fail where the spec demanded one. The question wasn’t “did these 25 rows look right?” It was “is the code written to be right for every row?”
The Glue table. This is the piece I’d most point other AWS folks to. The table schema wasn’t hand-maintained — it was declared as code in cdk.json, shared with the writing job, with no crawler-inferred definitions. So the test became a code-to-code comparison: parse the columns and data types the Glue job emits and diff them, name-by-name and type-by-type, against the schema declared in cdk.json.
On a lakehouse, your Data Catalog schema isn’t just config — it’s a financial-reporting control, and because it’s defined as code, you can test it like code.
All three passed. Evidence captured. Signed off.
Then the code was updated — and the obvious re-test wasn’t enough
Some time later, the engineering team shipped updated code. Not a rebuild — refactors, fixes, enhancements. The normal churn of a pipeline that’s maintained like software.
The obvious move is to re-run each control and check the output. I learned not to rely on that, because of one trap:
“Passed” is not the same as “unchanged.”
A clean re-run tells you the control still runs. It does not tell you the behavior is still the one you signed off. Code changes in three ways, and only one is safe to wave through:
- Unchanged — a pure refactor; line numbers moved, behavior identical.
- Changed, now matches the spec better — a genuine fix.
- Changed, drifted off the spec — still runs, still produces plausible output, still “passes” a naive re-test — but no longer does what the approved design says.
The third case is the entire reason to be careful. Nothing errors. The output looks fine. A results-only re-test sails straight through.
The habit: No Silent Drift
Three moves. The value is doing them per control, every time the code changes and holding one line.
1. Diff the control, not the file. For each control — each Lambda check, each segment mapping, each schema column — find the specific code in the old and new versions and compare just that. A 300-line file diff is noise. “The natural-account mapping changed, here are the exact lines” is signal.
2. Classify every change — and keep “changed” and “passed” in separate columns. Every change is unchanged, changed-matches-spec, or changed-off-spec. Whether a control changed is a fact about the code. Whether it passes is an audit conclusion. Collapsing those two is exactly how an unauthorized change gets rubber-stamped.
3. Anchor to the spec — not the old code, not your memory. “It matches what it did last quarter” is not a control objective. “It matches the approved design” is.
Get Gurucharan V S’s stories in your inbox
Join Medium for free to get updates from this writer.
Here’s roughly what a single catch looked like, sanitized — a company-derivation segment:
# Approved spec:
# Company = Department→Company, else BU→Company, else Segment→Company, else FAIL# OLD (signed off):
CO_ID = coalesce(dept_company, bu_company, seg_company)
# Dept → BU → Segment# NEW (after the update):
CO_ID = coalesce(bu_company, seg_company)
# BU → Segment — Department step gone
It still ran. It still produced company values. A results-only re-test would have passed it. The per-control diff against the spec is what flagged that the Department step had quietly dropped out — so we marked it changed, off-spec: do not sign off until authorized.
What the re-test actually found
Same update, and the per-control, spec-anchored diff produced genuinely different verdicts across the chain:
- Unchanged. Most segments — and the Lambda’s core validations — were pure refactors: identical behavior, shifted line numbers. Prior conclusions carried forward; no effort wasted re-testing settled logic.
- Changed, and now more correct. One segment used to keep the raw source value when a code didn’t map. The update changed it to assign the defined suspense value — which is what the spec said all along. The change made the control match the spec better than before. Pass, with a refreshed rationale.
- Changed, cosmetic but consequential. One job’s output column names moved from lowercase to UPPERCASE. Trivial-looking — except the Glue table (in
cdk.json) used UPPERCASE, so the old output was a latent mismatch and the change actually resolved it. This only surfaced on the schema-as-code comparison; a data sample would never reveal column names drifting against the contract. - Changed, and drifted off-spec — the catches. The company-derivation drop above, plus a second job that stopped populating a set of fields the spec said to populate. Both ran clean. Both were invisible to a results-only check. Both were held as pending authorization.
- A scary-looking change that was actually fine. The jobs added an aggregation step — summing and rolling up output. A new step that changes row counts sounds like it must break a schema control. The schema-as-code diff proved it didn’t: the aggregation preserved the output column set and routed its one new identifier column only to the table that already declared it. Re-verified against
cdk.json: no new column gap. The control held — and we could show it held, rather than assume it.
Same codebase, same update: one control strengthened, one real fix, two silent drifts caught, several unchanged, and one alarming-looking step proven harmless. A single “re-run and check the output” pass would have reported every one of them as fine.
What’s standard here, and what isn’t
I’d rather be precise than impressive.
Standard practice: code diffing, change management, IT general controls over deployments, mapping controls to code, schema validation, fail-closed testing. None of that is my invention.
What I found under-written is the combination — treating SOX automated-control testing as a regression problem across a continuously-changing AWS pipeline (ingestion Lambda + Glue job + schema-as-code table), with the plain discipline that “changed” is not “passed.” It’s not a new algorithm. It’s a field-tested way of working that keeps automated financial controls honest when the code won’t sit still.
Operationalizing it with AWS AI agents
Everything above, it was did by hand. It works — but it doesn’t scale and a pipeline that changes every sprint deserves a check that runs on every change, not once a quarter.
Here’s how I’d operationalize it on AWS. To be clear, this is a proposed reference architecture, not something I have shipped — but every piece is an existing AWS building block.
A code change (a pull request, or a new version) triggers an agent that runs the regression sweep and drafts the findings for a human to adjudicate:
- Amazon Bedrock Agents orchestrate the workflow and write the plain-language “what changed” narrative.
- A Bedrock Knowledge Base holds the approved mapping specs and
cdk.json, so the agent anchors to the approved design via retrieval — the automated version of “anchor to the spec, not the old code.” - Lambda action groups do the exact work: a per-control code diff, a schema extractor/comparator (job output vs
cdk.json), and the three-way classifier. - EventBridge + CodePipeline make it run on every change, not on a calendar.
- Findings and evidence are drafted to Amazon S3 for a reviewer.
Three guardrails that keep it SOX-defensible
1. Deterministic comparisons stay in code, not in the model. The schema diff, the column/type match, any totals reconciliation — these run as plain Lambda logic. A control test must be exact and reproducible; you don’t let a language model do the arithmetic or decide whether two column lists match. The agent orchestrates and explains; the tools compute.
2. Retrieval surfaces drift; it doesn’t adjudicate it. The agent can flag “this changed and no longer matches the retrieved spec” — but if the spec in the Knowledge Base is stale, the agent will confidently mis-classify (which is the very problem this whole approach exists to catch). So the agent raises the flag; it never decides whether a deviation is authorized.
3. Human-in-the-loop by design. The agent drafts findings and classifies changed vs unchanged. It must never set the audit conclusion.
Handling an approved requirement change
The Knowledge Base is only as trustworthy as the spec inside it — so when a requirement changes and the approved mapping has to change with it, the spec update must be a controlled, versioned event, not a quiet file swap. The habit:
- Version the spec like code. Keep mapping specs in a repo (or a versioned S3 prefix), each with a version and an approval record — never a loose document.
- Approve first, then sync. A qualified person approves the spec change; then you trigger a Bedrock Knowledge Base ingestion to that version (KBs don’t auto-update) and log which spec version is now live, who approved it, and when.
- Pin the agent to a version. The agent cites which spec version it evaluated against (“evaluated against Spec v3.2”), so any finding is reproducible.
- Mind the sequence. A code change is judged against the spec version that authorizes it. If code lands before its spec is approved, the agent should flag it off-spec and a human holds it — that’s the control working, not a false positive. Approve the spec → sync the KB → re-run the agent; only then does the matching code change classify as changed-matches-spec.
An agent can tell you, on every commit, which controls changed and whether they still match the approved design — turning a quarterly manual sweep into a continuous check. But a qualified person still signs the control. The agent scales the detection, the human keeps the attestation.
The takeaway
A control that still runs is not the same as a control that still complies.
On AWS, where the Glue and Lambda code keep moving, test the change, not just the result — diff per control, anchor to the approved spec and never let “changed” quietly become “passed.”
That’s No Silent Drift.
