Your Audit Log Never Saw the Break-In Attempt — Guards Run Before Interceptors
An audit log can be cryptographically intact, internally consistent, and completely wrong about what happened to your application.
Imagine an attacker trying to access an administrative endpoint without valid credentials. NestJS rejects the request with a 401 Unauthorized. Your global audit interceptor records every request that passes through it. The logs look healthy. The hash chain verifies.
But there is no record of the denied request.
The attacker tried to get in. Your application refused. Your audit log recorded neither the attempt nor the refusal.
This is not a cryptographic failure. It is a coverage failure.
The problem becomes obvious when you understand NestJS’s request lifecycle: guards execute before interceptors. When a guard rejects a request, execution never reaches the interceptor. An interceptor that claims to log every request can therefore silently miss the very authorization failures an auditor may care about most.
That discovery shaped Sigil, a tamper-evident audit logging service for NestJS.
Building it led to a broader realization: protecting audit records against modification is only half the problem. The other half is establishing that the records you have are a sufficiently complete account of what actually happened.
A log can prove that nobody changed its contents. It cannot, by itself, prove that every important event made it into the log.
1. The First Mistake: Assuming an Interceptor Sees Everything
NestJS gives developers a structured request lifecycle. Guards, interceptors, pipes, and other components have defined responsibilities, but those responsibilities also establish boundaries around what each component can observe.
A global interceptor is a natural place to begin an audit-logging implementation. It can capture request metadata, observe handler execution, and record outcomes for requests that reach it.
The problem is the word every.
Consider a protected administrative endpoint:
Incoming HTTP request
|
v
Guard
|
+---- Authentication or authorization fails
| |
| v
| Request rejected
| |
| X
|
v
Interceptor
|
v
Route handlerThe diagram contains an important boundary. If the guard rejects the request, the normal downstream interceptor execution never happens.
That means a global interceptor alone cannot provide comprehensive visibility into guard-level denials.
Suppose your audit trail contains these events:
10:14:02 User A updated account settings
10:14:11 User B accessed an administrative report
10:15:09 User A changed a billing configuration
10:17:43 User C downloaded an exportEverything looks ordinary.
But between 10:14 and 10:17, somebody tried to access an administrator-only endpoint without the required permissions. The guard rejected the request. The interceptor never observed it.
The resulting log is not necessarily corrupted. Every recorded event may be authentic, and every hash may verify. The missing event simply never entered the collection path.
This is why audit instrumentation must account for the lifecycle stages where failures can occur, rather than assuming one downstream component can observe them all.
An audit trail must record refusals, not just successful execution
Authorization failures are often among the most valuable security events in a system.
A successful administrative operation tells you what an authorized user did. A denied request can tell you that somebody attempted an operation they were not permitted to perform.
Neither event automatically proves malicious intent. Failed authorization can be caused by expired credentials, configuration mistakes, or legitimate users navigating to restricted endpoints. But repeated denials, unusual target selection, and changes in request patterns may provide the evidence needed for an investigation.
If your audit trail systematically misses guard-level denials, you lose that evidence before it reaches the storage layer.
The correct lesson is not that interceptors are useless. They are useful for the lifecycle stages they can observe.
The lesson is that audit coverage must follow the execution model of the framework, not the convenience of the logging hook.
For Sigil, this became the first architectural principle: identify where events originate and where they can be rejected or lost before choosing where to capture them.
2. Tamper-Evidence Is Not Completeness
Once an event enters an audit log, cryptography can help establish whether it has been modified.
A common technique is a hash chain. Each record incorporates the hash of the preceding record, linking the sequence together.
Conceptually:
Event 1
|
v
Hash 1 = H(Event 1)
|
v
Event 2 + Hash 1
|
v
Hash 2 = H(Event 2 + Hash 1)
|
v
Event 3 + Hash 2
|
v
Hash 3 = H(Event 3 + Hash 2)Changing an earlier event changes its hash and breaks the expected relationship with subsequent records. Provided the chain is anchored and verified appropriately, tampering with the recorded sequence becomes detectable.
This is valuable. It makes silent alteration more difficult and provides a way to verify the integrity of collected evidence.
But there is a fundamental limitation.
A hash chain proves something about the events that were recorded. It does not prove that every relevant event was recorded.
Imagine a system that captures 999 out of 1,000 security events. All 999 records are intact. Their hashes are correct. Their ordering is consistent. The chain verifies perfectly.
The one missing event does not invalidate the hashes of the remaining records.
From the chain alone, you cannot distinguish a complete record of 999 events from an incomplete record that should have contained 1,000.
Now consider a more adversarial scenario. A host crashes before forwarding a buffered event. A capture point never records a particular denial. A service stops reporting but the collector continues displaying the last known records as though nothing has changed.
The stored records may all be authentic.
The picture they present is still incomplete.
This is the difference between integrity and completeness:
- Integrity asks whether recorded evidence has been altered.
- Completeness asks whether the required evidence was captured and delivered.
- Availability asks whether that evidence can be retrieved when it is needed.
These are related properties, but none implies the others.
Sigil’s design treats completeness as an explicit problem rather than assuming cryptographic integrity solves it.
3. Make Missing Events Observable
The hard part of audit logging is often not protecting a record after it has been stored. It is detecting that an expected record never arrived.
A missing event does not carry its own explanation. The collector sees nothing. Without additional information, silence could mean the system had no events, the application was down, the event was dropped, or the reporting path stopped working.
Sigil approaches this problem through several complementary mechanisms: per-source sequence numbers, heartbeats, signed receipts, and reconciliation against an independent source.
Each addresses a different uncertainty.
Per-source sequence numbers: turn gaps into evidence
Suppose an application source emits audit events with monotonically increasing sequence numbers:
Source A:
101, 102, 103, 104, 105, 106The collector receives:
Source A:
101, 102, 103, 105, 106Sequence number 104 is absent.
Without sequence information, the collector has little basis for knowing that anything is missing. With it, the gap becomes observable.
The collector can identify that an event was expected in the sequence but is not present in the received records.
This does not automatically explain what happened. The event might have been lost in transit, delayed, or omitted by an upstream component. Sequence gaps establish evidence of an inconsistency, not necessarily its cause.
That distinction matters during incident response. A missing event should trigger investigation, not an unsupported claim that an attacker deleted it.
Sequence tracking also needs a clearly defined scope. A global counter shared across unrelated sources can make gaps difficult to attribute, while per-source counters let the collector reason about each producer’s stream independently.
Sigil uses per-source sequence numbers to make missing records detectable within those streams.
But sequence numbers alone are insufficient. If a source stops producing events entirely, no new number arrives to reveal a gap.
That requires another signal.
Heartbeats: make silence visible
A heartbeat lets a source periodically report that it is still alive and communicating with the collector, even when it has no audit events to send.
The distinction is between an inactive application and an absent reporting stream.
Without heartbeats, these situations may look identical:
- The application is healthy and has generated no audit events.
- The application is healthy, but its audit delivery path is broken.
- The application has crashed.
- The source is no longer reporting for an unknown reason.
A heartbeat narrows that ambiguity.
Application source Audit collector
| |
|---------- Heartbeat ------------>|
| |
|---------- Audit event ---------->|
| |
|---------- Heartbeat ------------>|
| |
X Connection or reporting fails |
|
No heartbeat
|
v
Missing-source signalThe absence of a heartbeat is not proof of a security incident. Networks fail, hosts restart, and deployments can temporarily interrupt reporting.
It is proof that the collector can no longer assume the source is communicating normally.
That is enough to make silence operationally meaningful.
A monitoring system can now alert on an expected source that has stopped reporting, rather than waiting for an investigator to notice an inexplicably quiet audit trail.
The heartbeat interval and timeout policy determine how quickly the system detects silence. Those values must account for normal delays and expected outages; overly aggressive thresholds create false alarms, while overly permissive ones delay detection.
The essential point is that silence should be a detectable state, not an invisible absence of data.
Signed receipts: make the collector’s promises checkable
The application is not the only component that can fail to preserve evidence. The collector is another part of the trust boundary.
Suppose an application sends an event and the collector acknowledges receipt. What does that acknowledgement mean?
Was the event merely accepted over the network? Was it validated? Was it durably stored? Does the application have any evidence of what the collector acknowledged?
Those are different guarantees.
A receipt makes the collector’s acknowledgement explicit. Signing the receipt allows the acknowledgement to be verified against the relevant signing key.
Conceptually:
Application source
|
|------ Audit event ------> Collector
| |
| Process event
| |
|<----- Signed receipt -------|
|
v
Verify receipt and retain evidenceThe purpose is not to make event delivery magically infallible. A signature cannot recover an event lost before the collector receives it, and it cannot establish a durability guarantee that the collector never actually provided.
Instead, a signed receipt makes an acknowledgement independently checkable. It provides evidence of what the collector claimed to have received or accepted, according to the receipt’s defined semantics.
That qualification is important. A production receipt protocol must define precisely what is being acknowledged, which event or sequence it refers to, and what storage or processing milestone the acknowledgement represents.
Otherwise, the signature can authenticate an ambiguous promise.
The broader principle is that the collection system should not ask every participant to trust an unverified statement that an event was received. It should provide evidence that can be checked later.
Reconciliation: compare independent views of reality
Even sequence numbers, heartbeats, and signed receipts leave a larger question:
How can you tell whether the application’s audit trail accounts for the requests that actually reached the system?
Get 0xKali’s stories in your inbox
Join Medium for free to get updates from this writer.
Sigil addresses this with a reconciler that compares the application’s log against the API gateway’s log.
The value of this comparison comes from observing the system at different points.
The application knows about events generated within its own execution path. The API gateway can provide a separate record of requests it observed before they entered the application.
Those views should not be identical in every respect. A gateway request and an application audit event may represent different lifecycle stages or use different identifiers. A useful reconciler must therefore compare events according to a defined relationship rather than assuming every gateway record maps one-to-one to an application event.
But when the mapping is established, disagreements become evidence worth investigating.
For example:
API gateway Application audit log
------------ ---------------------
Request A ------------------------> Request A recorded
Request B ------------------------> Request B recorded
Request C ------------------------> No matching event
Request D ------------------------> Request D recordedThe missing match for Request C is a discrepancy, not an automatic verdict. Perhaps the gateway rejected the request before it reached the application. Perhaps the request failed during processing. Perhaps an audit event was lost.
Those possibilities need different interpretations.
The reconciler’s job is to reveal the mismatch so the system can determine which explanation applies.
This is why independent capture points matter. If both the primary audit log and its only completeness check depend on the same application code, a failure in that code can make both views wrong in the same way.
A gateway log introduces a separate vantage point. It does not eliminate every common failure mode, but it makes certain forms of omission harder to hide.
Tamper-evidence protects the history you have. Independent reconciliation helps challenge whether that history is complete.
4. Why Database Sequence Numbers Can Break Your Audit Poller
Another lesson emerged from event ordering.
A database-generated sequence number looks like a convenient way to identify and process new audit events. A poller can remember the last sequence number it saw, query for larger numbers, and continue from there.
The problem is that sequence allocation order is not necessarily transaction commit order.
This distinction can create a subtle but permanent data-loss bug.
Consider two transactions.
Transaction A obtains sequence number 100 but remains open. Transaction B obtains 101 and commits first.
The state develops as follows:
Time
|
| Transaction A obtains sequence 100
|
| Transaction B obtains sequence 101
|
| Transaction B commits
|
| Poller reads event 101
| Poller advances its checkpoint to 101
|
| Transaction A commits event 100
|
vThe poller has already advanced its checkpoint to 101.
If its next query is effectively:
SELECT *
FROM audit_events
WHERE sequence_id > 101
ORDER BY sequence_id;event 100 no longer qualifies.
It may remain in the database forever, perfectly intact, while the poller never processes it.
No hash has been broken. No stored row has been modified. The data was not necessarily deleted.
The problem is that the poller’s checkpoint treated sequence allocation order as though it were commit order.
It was not.
Identity is not chronology
A database sequence can provide unique identifiers and a useful ordering of allocation. But that ordering does not automatically tell you which transactions have committed and become visible to a consumer.
This distinction is easy to miss because the resulting numbers appear orderly:
100, 101, 102, 103, 104A neat numeric order looks like a trustworthy event timeline. It is not sufficient evidence that a consumer has safely processed every committed record below its current checkpoint.
This is particularly dangerous in an audit system because the failure is quiet. The poller can keep running. New records continue arriving. Dashboards stay green. The database still contains the skipped record.
The ingestion pipeline appears healthy while its view of history is incomplete.
Checkpoints need ordering semantics that match the guarantee
The important design requirement is not merely to give events sequence numbers. It is to ensure that the mechanism used to advance the collector’s checkpoint cannot permanently skip a transaction that commits late.
Any solution must account for the relationship between transaction visibility and the consumer’s progress. A sequence value alone is insufficient if the consumer assumes that every lower value has already committed and been processed.
There are different approaches to reliable database change consumption, but the appropriate one depends on the actual database, transaction model, and delivery protocol. The architectural lesson here is independent of which approach is selected:
Never use identifier order as a substitute for commit order unless your system explicitly guarantees that relationship.
For an audit logging service, this is not a minor polling optimization. It determines whether a valid event can fall permanently outside the collector’s processing window.
5. Archiving Can Break Your Proof
Audit logs grow. Retaining every record in an active database indefinitely can become operationally expensive, so partitioning and archival are natural parts of an audit system’s lifecycle.
But archiving introduces another integrity boundary.
Suppose you build an integrity proof over records in an active partition. You then export those records to archival storage and drop the original partition.
If the archive is corrupted, truncated, or otherwise differs from the data used to construct the proof, what exactly have you established?
You may have a valid proof for records that no longer exist in the active database. But that does not automatically prove that the archived copy preserved the same records.
The act of moving data has created a new object that must be verified.
Sigil’s archival rule is straightforward:
Recompute the Merkle root from the archived copy before dropping the original partition.
A Merkle tree combines hashes of individual records into a hierarchy that produces a root hash. That root can serve as a compact commitment to the contents used to construct the tree.
At a high level, archival verification becomes:
Active audit partition
|
v
Archive records
|
v
Archived copy exists
|
v
Recompute Merkle root
from archived records
|
v
Compare with expected root
|
+---- Match ------> Archive integrity check passes
|
+---- Mismatch --> Do not drop original partitionThe critical ordering is that the archival copy is verified before the source partition is discarded.
If verification fails, the original data remains available for investigation or another archival attempt.
If the roots match, the archive’s content is consistent with the records used to produce the expected root, subject to the security of the hash function, the canonicalization rules, and the integrity of the expected root itself.
That last qualification matters. A Merkle root is not magic. If the expected root can be replaced along with the archived content, an attacker may be able to construct a new internally consistent archive and a matching new root.
A meaningful verification process must preserve the expected commitment in a trust boundary appropriate to the threat model.
The root comparison also verifies only the properties covered by the tree construction. It does not prove that the original archive operation included every event that should have been present, nor does it establish the truthfulness of the source records.
That distinction returns us to the central theme of Sigil: integrity is essential, but it is not the same as completeness.
Archival verification preserves the integrity of the evidence being moved. Sequence tracking, heartbeats, receipts, and independent reconciliation address different questions about whether the evidence was captured and delivered in the first place.
6. The Architecture Is a Chain of Evidence, Not Just a Chain of Hashes
These lessons fit together when audit logging is treated as an evidence pipeline.
A request produces events at different stages. Some requests are rejected before they reach downstream application components. Some records can be delayed or lost. A collector may acknowledge events, persist them, and expose them to consumers. Events may later be reconciled, verified, and archived.
Each boundary raises a separate question:
Boundary Question Request capture
Did we observe the event at the stage where it occurred?
Application lifecycle
Did an early rejection bypass the audit hook?
Delivery
Did the collector receive the event?
Collection acknowledgement
Can the receipt be verified, and what does it guarantee?
Ordering
Can the consumer checkpoint skip a late-committing event?
Completeness monitoring
Can the system detect missing records or a silent source?
Integrity verification
Can unauthorized modification be detected?ArchivalDoes the retained copy preserve the verified records?
These controls reinforce one another, but they are not interchangeable.
A hash chain cannot recover a denial that was never observed. A heartbeat cannot establish that every individual event arrived. A signed receipt cannot guarantee that an application generated the event in the first place. A Merkle root cannot establish that the original dataset was complete. And a database sequence cannot guarantee that a poller has processed every transaction whose sequence value is lower than its checkpoint.
A robust audit service must state which guarantee each mechanism provides and where that guarantee begins and ends.
For Sigil, the design direction is to make those gaps visible rather than conceal them behind a single claim of tamper-proof logging.
7. What Tamper-Evident Audit Logging Cannot Promise
It is tempting to describe an audit log as immutable, complete, and trustworthy after adding a hash chain, signing receipts, and preserving archived records.
Those labels can imply stronger guarantees than the implementation actually provides.
A hash chain can detect modifications to the covered chain when its verification assumptions hold. It cannot independently prove that a compromised application generated every required event.
Per-source sequence numbers can expose gaps in a stream. They cannot always tell you why an event is missing.
Heartbeats can reveal that a source has stopped communicating. They cannot prove that the source was reporting honestly before it went silent.
Signed receipts can authenticate a collector’s acknowledgement. They cannot make an ambiguous acknowledgement mean more than its protocol defines.
A gateway reconciler can expose discrepancies between two views of request activity. It cannot automatically determine which view is correct without additional evidence.
A verified Merkle root can establish a relationship between an archived dataset and an expected commitment. It cannot prove that the original dataset contained every event that should have been captured.
There are also important trust-boundary questions beyond these mechanisms. For example, a signing key that is compromised may undermine the authenticity of signatures made with it, and an application compromised before event generation may emit false or incomplete records. The provided design does not establish that Sigil solves every such threat.
These limitations do not make tamper-evident logging pointless. They make precise claims more important.
The objective is not to promise that an audit trail cannot fail. It is to detect more failure modes, preserve evidence about what happened, and avoid confusing a successful integrity check with proof that nothing is missing.
Conclusion: The Log Must Tell You When It Is Incomplete
Audit logging is usually presented as an integrity problem: make the records tamper-evident, chain their hashes, sign the important artifacts, and preserve them.
The cryptography matters. But much of the practical failure happens upstream.
The endpoint nobody instrumented.
The authorization denial that never reached the interceptor.
The event left behind when a host crashed.
The database transaction that committed after the poller had already advanced its checkpoint.
The service that quietly stopped reporting.
The archived copy that was never verified before the original partition was deleted.
Each failure can leave behind a log that verifies perfectly and still misrepresents the history it is supposed to preserve.
That is the problem Sigil is designed to confront.
Per-source sequence numbers turn gaps into evidence. Heartbeats turn silence into a signal. Signed receipts make collection acknowledgements checkable. Independent reconciliation exposes disagreements between the application and the gateway. Commit-aware consumption protects the integrity of the ingestion process, while recomputing Merkle roots before dropping partitions helps preserve the integrity of archived records.
None of these mechanisms alone makes the system infallible. Together, they shift audit logging away from the comforting assumption that intact records imply a complete history.
The real question is not only, “Has anyone altered my audit log?” It is also, “How would I know if something important never made it into the log?”
That is the difference between a collection of cryptographically protected records and an evidence system designed to expose its own blind spots.
