Your Logs Are Claims, Not Evidence — LLM Audit Trails — EU AI Act
Why traditional logs won’t survive Article 12 of the EU AI Act and what you need to implement across three layers: Capture, Store, Use.
March 3, 2027. Six months after Article 12 of the EU AI Act went into effect. An architect spreads a dashboard across the table — fourteen million LLM events. The auditor looks on without expression: “Logs from the party being audited. We need independent verification.”
The architect has no answer — because until now, no one required one. Between observability (“I can see what happened”) and auditability (“I can prove to a skeptic that it happened”) lie ninety percent of current LLM observability deployments.
“Falsifiability is the criterion of the scientific status of a theory.” — Karl Popper, Philosopher
Popper was talking about scientific theories. The EU AI Act auditor is talking about LLM logs. The logic is identical: without independent verification, a record stops being evidence and becomes a hypothesis from an interested party. Most platform teams will discover this distinction at the audit table — and that’s the wrong place for the discovery.
What You’ll Learn in This Article
- Why “logs are claims, not evidence” — the distinction between observability and auditability is epistemological, not configurational.
- Where LLM structural non-determinism comes from — it’s not temperature, it’s floating-point non-associativity and batch-invariance failure.
- How the three-layer Capture → Store → Use architecture works — what the gateway contracts, what the hash chain stores, what the SIEM consumes.
- How to resolve the GDPR Art. 17 vs EU AI Act Art. 12 paradox — deterministic tokenization at the edge and a dual-key architecture.
- What can’t be done retroactively — five day-zero decisions and a list of 12 fields to complete before August 2, 2026.
Logs Are Claims, Not Evidence — The LLM Auditability Paradox
After fifteen years of building systems for regulated industries, I’ve seen dozens of audits where the platform team was deeply convinced their observability stack “would be enough.” That confidence was justified in the world of deterministic microservices. It stops being justified in the world of probabilistic agents. An architect who carries that same pattern of trust into the EU AI Act territory is making a category error — and doesn’t find out until they’re sitting across from an auditor at the same table.
The distinction is simple but ruthless. Observability answers the question “what happened?” — that’s the operational layer. Auditability answers the question “how do you prove to a skeptic that it happened, without relying on the word of the party that acted?” — that’s the epistemological layer.
An audit trail is a chronological record of system events designed so a third party can reconstruct and verify every decision — as opposed to an application log, which records only what the system reports about itself. Conflating these two layers is the most common reason why “we have everything” turns into “you have nothing” over the course of a single audit afternoon.
There’s already a common law precedent that grounds this category. On February 14, 2024, the British Columbia Civil Resolution Tribunal (BCCRT 149) ruled in Moffatt v. Air Canada that the airline was liable for its chatbot’s hallucination. Tribunal Member Christopher Rivers put it plainly: “It should be obvious to Air Canada that it is responsible for all the information on its website. It makes no difference whether the information comes from a static page or a chatbot.” The penalty was symbolic — CAD 812.02.
The legal mechanism is not. It’s exactly the same mechanism Article 12 of the EU AI Act introduces in August 2026 — with a penalty up to €15 million or 3% of global revenue.
What the Skeptical Auditor Is Looking For
The mechanics of the audit trail paradox are straightforward. A traditional record of a “successful execute_script tool call” is assembled by the same system that executed it. The skeptical auditor is looking for the third element of the chain of custody — the independent witness — and in a traditional log, that witness simply doesn’t exist.
Article 12(2) itself repeatedly invokes the word traceability in a context where the auditor is a third party. That’s not an implicit regulatory preference. That’s the text of the statute.
None of the popular LLM observability stacks — LangSmith, Langfuse, Arize Phoenix — is a tamper-evident store in the sense of Article 12. Their own authors say so publicly. These are excellent operational tools, but an audit trail is a SEPARATE layer, not an alternative configuration of an existing stack.
The simplest change this section implies is also the hardest to execute organizationally: a separate “audit trail” column appears in the roadmap next to the “observability” column, with separate SLAs and a separate owner. Since logs aren’t evidence — the question remains: what exactly causes an LLM to break the assumptions underlying SOX, PCI-DSS, and HIPAA?
Structural Non-Determinism — Why LLMs Break the Assumptions Behind SOX, PCI, and HIPAA
“What we observe is not nature itself, but nature exposed to our method of questioning.” — Werner Heisenberg, Physicist
Model non-determinism isn’t a configuration bug — it’s a property of a system where the act of observation itself (runtime parameters, batch size, hardware) is part of the mechanism generating the output. I’d argue the biggest gap in current LLM deployments isn’t a lack of retention or a missing schema — it’s the failure to understand that the model is the source of non-determinism at the atomic level, not at the application layer. Every compliance architecture that minimizes this fact will prove to be paper-thin on audit day.
The Empirical Proof: 1,000 Identical Prompts
The strongest available evidence for this thesis comes from 2025. The Thinking Machines Lab team ran the same prompt one thousand times with temperature=0 on the Qwen3-235B model. The first 102 output tokens were identical across all one thousand runs.
At token 103, reality forks: 992 times “Queens, New York,” 8 times “New York City.” This is a published measurement, not a thought experiment.
The underlying mechanism settles the question: floating-point arithmetic on GPUs isn’t associative at the bit level. The same operations in a different order (different batch size, different hardware load) produce marginally different results — and after enough layers, those differences surface as a different token choice.
Anthropic’s public documentation admits it directly: “even with a temperature of 0.0, the results will not be fully deterministic.” OpenAI describes the seed parameter as "best effort." There's no deterministic LLM on default infrastructure — only a model whose variance you control better or worse.
Scale Amplifies Non-Determinism
IBM Research provided a second independent validation in a controlled experiment: 480 runs, 5 models, 3 tasks, 2 temperatures, 3 concurrency levels. Small 7–8B models achieved 100% output consistency at temperature=0.
The 118-billion-parameter model achieved 12.5% consistency — Fisher’s exact test, p<0.0001. Scale directly amplifies non-determinism. Trusting a “production” LLM stack that relies on a frontier model is mathematically unjustified if your traceability amounts to “we log temperature.”
An agent with fourteen steps and a five-percent divergence probability per step has a fifty-percent chance that at least one step will behave differently than expected. Boundary tracing via eBPF (AgentSight) sees 521 raw events in an attack, which the correlation engine reduces to 37 security-relevant events. An application log would see zero alerts. What “your dashboard” doesn’t show is precisely what the auditor will ask about.
The practical implication fits on a Post-it note: minimum traceability is NOT prompt content — it’s model_id + model_version + system_fingerprint + the full sampling parameter set (temperature, top_p, top_k, max_tokens, seed, stop_sequences). OpenAI's system fingerprint (e.g., fp_44709d6fcb) changes with every silent backend migration — it's the minimal independent signal that the underlying model was swapped without your knowledge. Without these fields, you can't tell the auditor what the system could have returned at a given moment.
The Three-Layer Architecture — Capture → Store → Use
Audit-ready AI consists of three orthogonal layers. Capture enforces the schema and masks PII at the edge. Store guarantees tamper-evidence and WORM. Use exposes the log to the auditor, the SOC, and the board in their own query language.
I’d argue the biggest design mistake in current enterprise LLM deployments is treating the Use layer (a Datadog dashboard) as the complete answer to compliance. Datadog is an excellent operational tool — but it isn’t tamper-evident, which means the Store layer in that architecture is simply empty.
The Logic of the Chain of Custody
The forensic chain of custody explains this better than any diagram. Who secured the evidence (Capture)? Who seals and stores it (Store)? Who presents it to the judge (Use)? These three roles are played by different people — because if one person played all of them, the court would trust none of them.
Architecturally, it looks like this: all LLM calls pass through a gateway that, after PII sanitization, emits a structured OpenTelemetry GenAI event. The event lands in an append-only ledger with a hash chain, whose Merkle root is periodically anchored to an external ledger. The stream is mapped to OCSF and forwarded to the SIEM. One flow, three orthogonal contracts.
Capture — The Gateway as the Single Source of Truth
The LLM gateway is the physical single point of capture. It enforces that no LLM call can bypass the audit trail — eliminating shadow AI structurally, not by policy. The schema isn’t optional: the OpenTelemetry GenAI Semantic Conventions (v1.40.0, February 2025) establish a shared vocabulary of gen_ai.* attributes — gen_ai.operation.name, gen_ai.request.model, gen_ai.usage.input_tokens, gen_ai.conversation.id, and optionally gen_ai.input.messages with opt-in OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true. Without this vocabulary, logs from LangChain, LlamaIndex, and custom microservices will never be comparable for the auditor.
An open debate worth knowing: Liudmila Molkova (OTel GenAI SIG co-lead) publicly warned in issue #2010 that “prompts contain health concerns, SSNs, addresses — applications compliant with different regulators will have trouble sharing this data with a broad group of DevOps humans.”
Compliance practitioners counter that Article 12 requires automatic recording — not sampling. The emerging consensus: store the hash + verbatim content in an access-controlled store. The audit log holds the hash and a pointer; the verbatim content lives in a separate, highly restricted store.
The latency budget is real: a gateway in the critical path can’t add more than a few dozen milliseconds. In my deployments, structured logging at the gateway level fits within that budget together with schema validation and sanitization.
Store — Hash Chains, Merkle Trees, WORM
A hash chain is a record where each new entry begins with a hash of the previous one. Changing anything in the middle breaks the continuity — and does so in a way that’s mathematically detectable, without needing to trust the operator.
Get Jaroslaw Wasowski’s stories in your inbox
Join Medium for free to get updates from this writer.
The technical mechanism: each event e_i contains SHA-256(e_{i-1} || payload_i || seq_i). Modification propagates through all subsequent hashes (avalanche). The academic foundation is Efficient Data Structures for Tamper-Evident Logging (Crosby et al., USENIX Security 2009), which introduced history trees giving O(log n) lookup. When you see "hash-chained append-only ledger" in a specification, you're looking at a derivative of that one paper.
The Merkle tree ensures scalability. Leaves are event hashes, nodes are SHA-256(left || right), and the root is a single signature of the entire log for a given period. You publish the root to an external ledger (transparency log RFC 9162, blockchain) every N minutes. That's exactly the "independent witness" from the first section — a cryptographic anchor that extends beyond your trust boundary.
WORM is the physical guarantee: AWS S3 Object Lock in Compliance mode physically blocks deletions even for the root user, Azure Immutable Blob and GCP Bucket Lock provide analogous guarantees. A key precedent: SEC Rule 17a-4 (amended October 2022) introduced an audit-trail alternative to classical WORM that cryptographically hash-chained records can satisfy — effectively recognizing them as a legally compliant mechanism. A hash-chained ledger in place of physical WORM is a legally permissible alternative.
The triple bind that can’t be technically circumvented: S3 Object Lock Compliance satisfies 17a-4(f), conflicts with GDPR Art. 17, and remains exposed to the US CLOUD Act. Microsoft, before a French Senate committee in 2025, acknowledged that it “cannot guarantee that French customers’ data will never be disclosed under an American court order.” A day-zero decision you can’t delegate to the vendor: does your ledger’s jurisdiction accept this triple bind, or does it require a sovereignty layer?
Use — SIEM, KPI Governance, Auditor-Facing Query
The Use layer starts with translation. OCSF (Open Cybersecurity Schema Framework) takes the nested OTel GenAI JSON and normalizes it to a vendor-agnostic event format. The result: a prompt injection event from the gateway is read by Microsoft Sentinel (KQL) or Splunk (SPL) in the same structure as an AWS CloudTrail anomaly. That’s the key that turns LLM telemetry from a curiosity for the AI team into a full-grade signal source for the SOC.
Concrete detection patterns are well-documented today. KQL aggregates completion calls in a 10-minute window and triggers an alert when a single user iterates prompts with abnormal token consumption — that’s a proxy for prompt injection. Sentinel’s series_decompose_anomalies() detects when an agent that normally runs 50 queries/hour suddenly spikes to 4,000.
From the same Use layer come four governance KPIs: policy compliance rate, incident frequency + MTTD, time-to-approval for LLM use requests, audit findings closure rate. Without this layer, these KPIs are unmeasurable — because there’s no unified query interface.
Cost economics closes the argument. For 1M LLM calls per day (~100 GB metadata + content per month, US East pricing, early 2026): Sentinel PAYG ~$430/mo, Sentinel Auxiliary ~$15/mo, S3 Standard ~$2.30/mo, Glacier Deep Archive ~$0.10/mo. The pattern I recommend: 7 days hot in the SIEM for security signals, 90 days in IA for incident response, 6 months to 7 years in Glacier for compliance retention. That segmentation reduces cost by one to two orders of magnitude.
GDPR Art. 17 vs EU AI Act Art. 12 — The Paradox and Its One Clean Solution
I’d argue this section can’t be left to the lawyers. The lawyers won’t converge — because the problem is technical, not legal. The architect needs to propose an engineering solution before walking into the room with the compliance officer, not after.
The Nature of the Conflict
The conflict is literal. GDPR Art. 17(1) guarantees the right to erasure “without undue delay.” EU AI Act Art. 12 + Art. 19 require automatic logging plus a minimum of six months’ retention plus tamper evidence implied by the word traceability.
A hash chain by definition avalanches on the deletion of a single record — breaking the integrity of ALL subsequent events. The most common pseudo-solution is to lean on Art. 17(3)(b) as an exemption — legally fragile.
Recent CJEU case law on pseudonymization from 2025 (EDPS v. SRB, September 2025) shifted the interpretation: pseudonymized data may NOT constitute personal data for a recipient incapable of re-identification. That doesn’t rescue immutability — but it opens a space where the technical solution carries legal weight.
The Solution: Deterministic Tokenization at the Edge
The architectural solution has one name: deterministic tokenization at the edge, in the gateway, before any logging occurs. An NER model in the gateway detects entities (“Jane Smith”), replaces them with a deterministic session-consistent token (“Person_hxg3”), and holds the entity↔token mapping in a separate vault with its own KMS key. The audit log contains ONLY the token.
Pseudonymization keeps data as personal data under GDPR (the key can restore it); anonymization renders it irreversibly non-personal. When a customer exercises their Art. 17 right, you delete the specific mapping in the vault. The token in the hash chain remains mathematically valid but becomes mathematically untraceable to the person — and that’s precisely the condition CJEU SRB is looking for.
Two keys, two retention policies — this is the core of the separation and it isn’t optional. The vault key falls under GDPR (deletable on request, audited access). The hash in the ledger falls under the EU AI Act (immutable, ≥6 months, tamper-evident). These two systems live separately, with separate lifecycles, separate trust boundaries, and separate owners.
Conflating them — letting raw PII into the hash chain “just temporarily” — defeats the entire architecture and creates a compliance liability that retroactive masking doesn’t remove.
Concrete Tools
Concrete tools are well-documented today. Microsoft Presidio serves as a baseline (open source, F1 ~0.85 on clinical data), Azure PII service as production-grade with high precision on PHI detection tasks. NIST FF1 FPE remains the only approved format-preserving cipher (FF3 was withdrawn on February 3, 2025) — important because many existing business deployments relied on FF3 and require migration.
On Monday you do one concrete thing: insert a lightweight NER in front of the gateway, hold the vault on a separate KMS with a separate lifecycle. That requires coordination across Platform, Security, and Compliance — which is exactly why you start now, not a month before the audit.
Day-Zero Discipline — 12 Fields on the Whiteboard Before August 2, 2026
“It is not that we have so little time, but that we waste so much of it.” — Lucius Annaeus Seneca, Philosopher
I’d argue that for a retrofit you’ll pay not just a higher engineering cost, but also a loss of trust. An audit that catches you off guard redefines AI’s position in the organization — and rebuilding that position takes years, not quarters. Day-zero discipline isn’t a purist ideal — it’s an economic calculation.
The Deadline Math
The deadline math is brutally simple. Article 12 for Annex III high-risk systems takes full effect on August 2, 2026. Article 19 requires ≥6 months of retention. Compliance practitioners warn: if you start a logging program on enforcement day, you’re already late — because your log from August 2, 2026 must reach back to February 1, 2026.
The professional deadline for most organizations isn’t August 2026 — it’s February 2026. No exceptions, no negotiations, no phase-in.
Five Decisions Beyond the Reach of Retrofit
Five design decisions are beyond the reach of retrofit because each one breaks integrity when inserted retroactively:
- Identifier scheme (
model_id,deployment_id,dataset_idas stable IDs) — inserting them retroactively breaks the hash chain. - Append-only backend — migrating from a mutable store to an append-only one leaves a gap for the migration window.
- PII masking pipeline at the gateway — once raw PII enters storage, it creates new GDPR liabilities.
- Hash chain genesis event — inserting a genesis event retroactively defeats the integrity guarantees of the entire chain.
- Schema version field — without it, forward-compatible queries are impossible.
12 Fields on the Whiteboard
12 fields every LLM call must account for: user_identity, agent_identity, system_prompt (verbatim or hash+pointer), user_prompt, retrieved_chunks (id + content_hash), model_id + version + system_fingerprint, sampling parameters, output (or hash+pointer), tool_calls, guardrail_decisions, cost_metadata, correlation_id propagated from the HTTP edge.
Each of these fields answers a specific EU AI Act auditor’s question. A missing field = a question you can’t answer at the audit table. The 12 fields aren’t a checklist — they’re a mental model the architect carries in their head and uses to spot gaps on any review immediately.
This episode extends component 5 (Incident Response) from aigov01 Governance OS — the audit trail is its operational mechanism. The LLM gateway from aigov02 Acceptable Use Policy is the physical Capture point in this architecture. The focus here is exclusively on Art. 12 + 19 as a concrete implementation of the broader EU AI Act from aigov04.
Summary
Back to the opening scene. The auditor demands independent verification — and this time the architect doesn’t go silent. They walk through the three-layer architecture, point to the Merkle root anchoring points in the external ledger, and demonstrate that the PII mapping vault lives separately from the immutable hash chain. The auditor checks the box.
Five things you’re leaving with:
- Logs are claims, not evidence — the difference between observability and auditability is epistemological, not configurational.
- LLM non-determinism is structural (token 103, 12.5% consistency) — minimum traceability includes the full distribution parameter set.
- The three layers are orthogonal: Capture enforces the schema and masks PII, Store guarantees tamper-evidence + WORM, Use exposes the log in the auditor’s and SOC’s query language.
- The GDPR vs EU AI Act paradox has one clean engineering solution — deterministic tokenization at the edge and a dual-key architecture.
- Day-zero discipline is math, not philosophy — to have 6 months of logs on enforcement day, you need to start on February 1, 2026.
Next Monday, the architect draws the map of 12 fields on the whiteboard, identifies which ones are missing, and coordinates with Platform, Security, and Compliance on a roadmap to February 1, 2026. The audit trail stops being “some unresolved gap in observability” and becomes the second architectural column alongside observability.
Thank you for reading this far — it’s not an obvious choice for an article about hash chains and Article 12. If this piece changed how you think about LLM observability, pass it along to whoever is designing your organization’s audit trail over the next six months. Follow if you want the next episodes of the AI Governance & Enterprise Architecture series. I value most the comments where you describe your own cases — which of the twelve fields do you have in production today, which ones you don’t have, and why.
