Auditing LLM Decisions: Logging Prompts and Responses for Compliance
Three weeks after we pushed an LLM-powered underwriting assistant to production, our compliance team asked a question I wasn’t ready for: “If a regulator asks why the model rejected this applicant, what do we show them?”
I didn’t have a good answer. We had logs, sure — but they were the kind of logs engineers write for themselves at 2am while debugging a timeout. Half the prompts were missing. The ones we did have still had the customer’s full ID number sitting in plaintext. Nobody could tell you which model version produced a given response six months later because we weren’t recording that either.
That conversation turned into a two-week project to build an actual audit logging layer for our LLM calls, and most of what follows comes from decisions we got wrong the first time.
Why “just log everything” doesn’t work
The instinct is to wrap your LLM client in middleware and dump the full request/response pair into a log file or a Postgres table. That gets you 80% of the way there and creates two new problems.
First, prompts and responses are full of PII — names, account numbers, medical details, whatever the user typed. If your audit log is a dumping ground for raw text, you’ve just built a second, less-protected copy of your production database, except now it’s sitting in a logging pipeline that twenty engineers have read access to.
Second, “log everything” isn’t actually an audit trail. An audit trail answers specific questions: who triggered this decision, what model and prompt template produced it, what the model actually said, and can we reproduce this six months from now when a regulator or a customer disputes it. A pile of unstructured log lines answers none of those reliably.
What we decided to log
We settled on a structured record per LLM call, not a blob of text. Here’s the shape of it:
type LLMAuditRecord struct {
ID string `json:"id"`
Timestamp time.Time `json:"timestamp"`
UserID string `json:"user_id"` // the human whose request triggered this
RequestID string `json:"request_id"` // ties back to the originating HTTP request
ModelName string `json:"model_name"`
ModelVersion string `json:"model_version"`
PromptTemplate string `json:"prompt_template_id"` // reference, not the literal filled prompt
PromptHash string `json:"prompt_hash"` // sha256 of the actual rendered prompt
RedactedPrompt string `json:"redacted_prompt"`
RedactedResp string `json:"redacted_response"`
Decision string `json:"decision,omitempty"` // e.g. "approved", "flagged"
TokensIn int `json:"tokens_in"`
TokensOut int `json:"tokens_out"`
LatencyMs int64 `json:"latency_ms"`
RawPromptRef string `json:"raw_prompt_ref,omitempty"` // pointer to encrypted storage, not inline
RawResponseRef string `json:"raw_response_ref,omitempty"`
}Two things to notice. The raw, unredacted prompt and response don’t live in this record — they live behind a reference (RawPromptRef) pointing to encrypted storage with much tighter access control. And we hash the actual rendered prompt so we can prove, later, that a specific prompt produced a specific output, without needing to store the full text in the easily queried table.
The decision field matters more than people expect. If the LLM call is feeding into something that affects a user — a rejection, a flag, a recommendation — you want that outcome captured as structured data you can aggregate on, not buried inside a response string you'd have to re-parse later.
What we redact before it touches a log
We built a redaction pass that runs before anything gets written to the queryable audit table. It’s deliberately conservative — false positives (over-redacting) are cheap, false negatives are the thing that gets you fined.
import re
from dataclasses import dataclass
PATTERNS = {
"ssn": re.compile(r"\b\d{3}-\d{2}-\d{4}\b"),
"credit_card": re.compile(r"\b(?:\d[ -]*?){13,16}\b"),
"email": re.compile(r"\b[\w.+-]+@[\w-]+\.[\w.-]+\b"),
"phone": re.compile(r"\b\+?\d{1,2}[\s-]?\(?\d{3}\)?[\s-]?\d{3}[\s-]?\d{4}\b"),
}
@dataclass
class RedactionResult:
text: str
redaction_count: int
def redact(text: str) -> RedactionResult:
count = 0
for label, pattern in PATTERNS.items():
text, n = pattern.subn(f"[REDACTED:{label.upper()}]", text)
count += n
return RedactionResult(text=text, redaction_count=count)This catches the obvious stuff — SSNs, card numbers, emails, phone numbers. It will not catch “my son’s name is Marcus and he goes to Lincoln Elementary,” because that’s context-dependent PII that a regex can’t see. For that we lean on a named-entity recognition pass as a second layer, and for the highest-sensitivity systems, we flag any record where NER confidence drops below a threshold for a human to review before it’s retained long-term.
Get Erwin Hermanto’s stories in your inbox
Join Medium for free to get updates from this writer.
The honest caveat here: no redaction system catches everything. We treat the redacted log as what gets queried day-to-day, and we keep the raw version encrypted and access-gated specifically for the cases where redaction missed something and compliance needs the ground truth.
Where it actually gets stored
We split storage into two tiers, and this split did more for our audit posture than any amount of regex tuning.
Hot tier — the redacted, structured records above, in a normal relational table, queryable by the compliance and support teams without a special access request. Retention here matches whatever the applicable regulation requires (for us, financial services rules meant seven years for anything tied to a credit decision).
Cold tier — the raw, unredacted prompt/response pairs, encrypted at rest, in object storage with access logged separately and gated behind a break-glass process. Nobody queries this casually. It exists so that if a dispute or regulatory inquiry needs the exact, unredacted exchange, it’s retrievable and its retrieval is itself auditable.
func (s *AuditStore) WriteRecord(ctx context.Context, rec LLMAuditRecord, rawPrompt, rawResponse string) error {
promptRef, err := s.coldStorage.PutEncrypted(ctx, rawPrompt)
if err != nil {
return fmt.Errorf("writing raw prompt to cold storage: %w", err)
}
respRef, err := s.coldStorage.PutEncrypted(ctx, rawResponse)
if err != nil {
return fmt.Errorf("writing raw response to cold storage: %w", err)
}
rec.RawPromptRef = promptRef
rec.RawResponseRef = respRef
rec.PromptHash = hashPrompt(rawPrompt)
return s.hotStore.Insert(ctx, rec)
}The cold storage write happens first and is required to succeed before the hot record is written — if we can’t guarantee the raw version is recoverable, we don’t want a hot record implying we have the full picture when we don’t.
How the volume actually breaks down
For a rough sense of scale: in our first month running this across three LLM-backed features, here’s roughly how record volume split by what triggered retention review.
Audit records by category (Oct, ~48K total LLM calls)
─────────────────────────────────────────────────────
Routine (auto-expire per retention policy) ████████████████████████ 38,100
Decision-affecting (7yr retention, hot+cold) ██████ 6,900
Flagged by NER for manual redaction review ██ 2,200
Accessed from cold storage (disputes/audit) ▏ 180The last row is the one that validated the whole approach for us — 180 cold-storage retrievals out of 48,000 calls. That’s the entire point: you’re not building this so you can read every prompt every day, you’re building it so that the rare moment someone needs the ground truth, it exists and is retrievable without a scramble.
A mistake worth mentioning
Our first version hashed the prompt template instead of the fully rendered prompt. That seemed fine until someone asked us to prove what the model actually saw for a specific user on a specific date — and the template hash told us nothing about the variables that got interpolated into it. We had to re-derive it from application logs that had a much shorter retention window than the audit table did. Hash the thing that was actually sent, not the thing that generated it.
What I’d tell someone starting this today
Don’t start with the redaction regex. Start by writing down the actual questions a regulator, an auditor, or an angry customer might ask you in twelve months, and work backward to what record would answer them. In our case that was: who, when, what model, what decision, can we reproduce it, and can we show the exact exchange without exposing it to everyone with database access. Everything in the design above falls out of those five questions.
The interesting part of audit logging isn’t the storage engine or the redaction library — it’s deciding, before you need to, what “defensible” looks like for your specific use case.
