Adversarial Detection Engineering in the Real SOC: A Practitioner’s Guide to the ADE Framework

Digit Oktavianto

16 min read

·

Oct 21, 2026

--

Listen

Share

The rule that was always going to miss

Most detection rules fail silently. They sit in production, green and healthy on the dashboard, and the activity they were written to catch walks straight past them without raising a single alert. No error, no exception, no log line that says I just let that through. The SOC believes it has coverage. It does not.

That gap — between what a rule intends to catch and what it actually catches — is the subject of Adversarial Detection Engineering (ADE) (https://github.com/Adversarial-Detection-Engineering), an open-source framework created by Nikolas Bielski (with co-maintainer Daniel Koifman) that treats the false negative as a first-class engineering defect rather than an accident of tuning. ADE gives that defect a name: a detection-logic bug. And, like any bug class, once you can name it you can hunt it, classify it, test for it, and fix it.

Press enter or click to view image in full size

This post is written for the detection engineer, the SOC analyst moving into rule authoring, and the purple teamer who wants a shared vocabulary with the blue team. It covers three things, in order: what the ADE framework actually says and where it fits among the detection-engineering frameworks you already know; how to run it as a repeatable workflow inside a live SOC; and which platforms and tools — Sigma, Atomic Red Team, Caldera, your SIEM, your CI/CD pipeline — you bolt it onto, with the practical mechanics of each.

The mindset shift ADE asks for is small but load-bearing. Stop asking “does my rule detect the technique?” Start asking “what is the full set of things that satisfy my rule’s intent, and which members of that set does my query actually match?” The difference between those two questions is where attackers live.

What ADE actually is: a bug taxonomy for detection logic

The conceptual core is borrowed, deliberately, from software engineering. ADE models a detection rule as a repeatedly-running hypothesis test: it takes a bucket of records and tests each against a set of conditions. The rule has a scope — the human-readable intent, carried in its description and MITRE ATT&CK mappings (“detect PowerShell downloading a file”) — and a detection logic, the machine-interpretable query that is supposed to implement that intent. A detection-logic bug is a mismatch between the two: an attacker behavior that falls squarely inside the rule’s scope, yet does not trip the query. In confusion-matrix terms, a ground-truth true positive the rule reports as negative. A false negative, caused on purpose.

The framework’s contribution is that it stops treating those misses as one undifferentiated blob and sorts them into a taxonomy: four categories, sixteen subcategories, and (at time of writing) thirty-eight worked technique files, each grounded in real public rules from Sigma, Elastic Security and Microsoft Sentinel. The four categories examples below are the load-bearing structure — learn these and you can place almost any miss:

1. ADE1 — Reformatting in Actions. The logged input is reshaped so a string match fails, even though the behavior is unchanged. Substring manipulation, obfuscation, whitespace and delimiter tricks, normalization asymmetry. The classic: a rule keys on the literal .DownloadString(, and the attacker writes $wc.("Download"+"String")(...). Same download, no match.
2. ADE2 — Omit Alternatives. The rule enumerates one member of an equivalence class and misses the rest. A different LOLBin, a renamed binary, a newer API version, an alternate path, a different file type — all in scope, none listed. If your rule anchors on curl.exe, it says nothing about certutil, bitsadmin, BITS jobs, or a .NET WebClient.
3. ADE3 — Context Development. The attacker shapes the surrounding context rather than the primary action: cloning a process name, hijacking an aggregation threshold, spreading activity past a correlation window (low-and-slow), fragmenting one logical action across several events, spoofing parent lineage, or saturating an engine’s join/groupBy limit so the needed record is silently truncated.
4. ADE4 — Logic Manipulation. The Boolean, filter or field logic is itself wrong or invertible: an AND that should be OR, a negated filter on an attacker-controlled field, or a field that doesn't resolve on the deployed backend and so matches nothing — making the rule structurally blind before an attacker even shows up.

Two supporting tools in the repo turn the taxonomy into something you can run. The Bug Likelihood Test is a fast pre-analysis checklist — roughly a dozen yes/no questions (“Is there a process-name string match that could be renamed?” → ADE3–01; “Are there aggregation operators like >=?” → ADE3-02) that triages a rule in minutes and tells you which category to dig into. The ADE False-Negative Checklist applies four lenses to every rule you review: data-source dependency (does the telemetry even reach the rule?), threshold assumptions, scope assumptions, and environment drift.

One point the authors are emphatic about, and so am I: this is a defensive, purple-team framework. Every lens is answered at the level of “would this rule catch the technique, and what must be true for it to fire?” — never “here is the exact recipe to beat rule X.” You describe categories of gap so the blue team can close them. Weaponizing it against systems you aren’t authorized to test is a different activity with a different name.

Where ADE sits in the detection-engineering stack

ADE doesn’t replace the frameworks you already run; it fills a specific hole none of them were built for. It helps to see the whole stack and what each layer answers.

MITRE ATT&CK tells you what to detect — the technique catalog and the common vocabulary everything else maps to. The Pyramid of Pain (David Bianco) tells you how durable a given detection is: hashes and IPs are trivial for an attacker to change, while TTPs are painful, so you should aim your logic high on the pyramid. Palantir’s Alerting and Detection Strategy (ADS) tells you how to document a detection to a quality bar — its template forces a Strategy Abstract, an ATT&CK categorization, and, tellingly, an explicit Blind Spots and Assumptions section where the Detection Engineer records where the alert may not fire. Detection-as-Code tells you how to ship and govern detections: version control, peer review, linting, CI/CD, automated testing.

What none of those does, at the level of query mechanics, is tell you why a specific, reasonable-looking rule won’t fire against an in-scope variant. ADS gives you a box labeled “Blind Spots and Assumptions”; ADE gives you the taxonomy to fill it in rigorously instead of from memory. The Pyramid of Pain tells you to detect behavior rather than strings; ADE1 and ADE2 show you the precise ways a behavior-named-as-a-string still collapses back into a brittle IOC. In the language SpecterOps popularized in Capability Abstraction and On Detection: Tactical to Functional — the idea that a technique has an abstraction chain from tool down to the operation it actually performs — ADE is a structured way to walk down that chain and find which rung your detection accidentally pinned itself to.

It also isn’t alone in treating adversarial review as central. The emerging work on automated detection engineering — for example Mitiga’s FORGE, which runs a generate-then-adversarially-validate pipeline with a separate reviewer agent — is the same instinct ADE formalizes by hand: a detection is not done when it’s written, it’s done when something has actively tried to make it miss and failed.

The table below is the one-glance version.

Press enter or click to view image in full size

In short: the other frameworks make detections well-documented, durable and shippable. ADE makes them robust against someone who read them.

Running ADE in a live SOC: the workflow

The framework is only useful if it becomes a habit your team repeats, not a one-off audit. Here is how to operationalize it. Treat it as a loop you run against your detection backlog continuously, and bolt onto every new rule before it ships.

Who does it. This is where ADE gives purple teaming a shared language. The detection engineer owns the rule and the fix. The red/purple teamer owns the emulation that proves the gap is real. An analyst or threat hunter often spots the candidate gap first, from an incident where a rule “should have” fired. The ADE taxonomy is the lingua franca between them — a miss gets a label (ADE3–01), not a shrug.

The loop, step by step:

1. Select the rule and state its scope. Pull one production (or candidate) rule. Write down, in one sentence, what it intends to catch and its ATT&CK mapping. If you can’t state the scope crisply, that’s finding zero — an ambiguous rule can’t be reasoned about.
2. Run the Bug Likelihood Test. Walk the dozen checklist questions. Does it string-match a command line (ADE1–01)? Anchor on one binary or method (ADE2–01)? Use a threshold or time window (ADE3–02/03)? Key on parent process (ADE3–05)? Negate a filter on an attacker-controlled field (ADE4–01)? Each “yes” is a hypothesis about where a false negative lives. This takes minutes and tells you whether the rule is worth a deep review.
3. Apply the four lenses. For each flagged area, ask the checklist’s questions properly. Data-source dependency first — is the telemetry this rule needs actually ingested on the in-scope hosts? A rule that assumes Sysmon EID 1 on a fleet running only Security 4688 (with command-line auditing off by default) contributes zero coverage, silently. Then threshold, scope, and environment-drift assumptions.
4. Form the adversarial hypothesis — at the category level. For each gap, write it as a testable statement: “The same technique performed with an alternative in-scope binary / reformatted command / below the count threshold / spread past the window is predicted to produce no alert.” Note: this is a category of variant, routed to your authorized red-team lead to turn into a concrete test under rules of engagement — not an execution recipe you write into the ticket.
5. Validate in a lab, under authorization. Emulate the variant on a monitored test host and check whether the rule fired. This is the step that converts a suspected bug into a confirmed one. Atomic Red Team and Caldera (next section) are how you do it repeatably.
6. Harden the rule and re-test. Apply the fix — broaden the match, OR in alternatives, switch to an immutable field, add a presence-guard to a negated filter, add a companion “canary” rule for the evasion itself — then re-run the same emulation to prove the gap closed and nothing regressed.
7. Document honestly and record residual blind spots. Update the rule’s ADS-style record: what you tested, what’s now covered, and what you still can’t detect. “Not covered: direct registry modification — needs a separate rule” is a valid, valuable finding. A rule that passes all four lenses gets a one-line note saying so and a reminder to confirm it’s actually enabled.

Cadence. New rules get steps 1–4 as a pre-merge gate (cheap, fast, no lab needed). Existing rules get the full loop on a rotation — prioritize by ATT&CK coverage importance and by Bug Likelihood score. Tie it to your purple-team calendar: each emulation exercise should close its confirmed gaps back into the rule base, so the exercise produces hardened detections, not just a report.

The discipline that makes this work is adversarial review by someone other than the Detection Engineer themselves. You are biased toward the rule you just wrote; you will read your own query and see the technique it catches, not the three it misses. Have a second engineer (or, increasingly, an automated reviewer) run the lenses cold.

The toolchain: what to pair with ADE, and how

ADE is a reasoning framework, not a product. Its value shows up when you wire it into tools you (probably) already run. Here is the practical mapping, in the order you’d use them around the loop.

1. Sigma — the portable rule you reason about

Sigma is the default starting point, and the repo’s own bug-pattern references lean on it. Because a Sigma rule is backend-agnostic YAML that transpiles to most SIEM languages, it’s the cleanest artifact to read “how is this technique usually caught” and to apply the taxonomy to.

How to use it with ADE: Read the detection: block as the hypothesis test. A list under one key is OR; separate keys are AND; |all forces AND across a list. Most ADE4 (logic) bugs are visible right here — the infamous example is a rule that ANDs Net.WebClient with (.DownloadString( OR .DownloadFile(), where the fix is simply condition: 1 of selection_*. Then check the ADE1 reformatting modifiers: is there a |windash for -// prefixes? A |base64offset for encoded commands? An OR onto OriginalFileName to resist renaming? Crucially, transpilation is not lossless — a Sigma rule's real behavior depends on the backend (regex flavor, case sensitivity, null-handling on not filter). Always inspect the compiled query on your actual backend, not the YAML. Field typos (TargetFileName vs the correct TargetFilename) resolve to no column and fail silently — a pure ADE4-04 bug.

2. Your SIEM/EDR — where the bug is real or not

Elastic Security, Microsoft Sentinel, Splunk, and CrowdStrike LogScale each have public rule repos and each has platform-specific traps the ADE bug_patterns/ catalog documents.

How to use it with ADE: map the generic category to the concrete fix for your platform. A few high-leverage examples straight from the framework’s mitigations guidance:

• Elastic threshold rules (ADE3-02): don't count per attacker-controlled field alone; add threshold.cardinality (distinct sources against a target) and a coarser account/tenant bucket so distributing activity across entities doesn't duck under.
• Elastic eql sequence (ADE3-03): set a realistic maxspan, not 5s, or decouple the steps — otherwise a sleep expires the correlation.
• Sentinel aggregation (ADE3–02): avoid per-IPAddress thresholds and Top-N ranking; use cardinality and absolute thresholds, and corroborate with an orthogonal signal.
• Sentinel case sensitivity (ADE1/ADE4): ==, has, in are case-sensitive in KQL and silently miss variants; use =~, in~, has_cs deliberately.
• Process identity everywhere (ADE3–01): anchor on process.executable / hash / code-signature plus args, never process.name alone, which any rename defeats.
• Negated filters everywhere (ADE4–01/04): guard every not filter on an optional field with a presence check (field|exists: true), so an absent field can't invert the rule.

3. Atomic Red Team — cheap, fast variant validation

Atomic Red Team (Red Canary) is a library of small, ATT&CK-mapped tests you run on a host to generate the telemetry a technique produces. It’s the fastest way to execute step 5 of the loop.

How to use it with ADE: pick the atomic test(s) for your technique, run them on a monitored lab host, and confirm the baseline rule fires. Then use the variants — many atomics parameterize the binary, path or arguments, which is exactly ADE2 (alternatives) and ADE2–03 (locations) made executable. Run the canonical form, confirm the alert; run the in-scope variant, check whether the alert disappears. A disappearing alert is a confirmed detection-logic bug. Where no atomic exists for the exact variant, that’s the cue to route it to your red-team lead rather than improvising one in the SOC.

4. Caldera — chained, context-shaping emulation

MITRE Caldera runs autonomous adversary profiles — sequences of abilities, not single atomics. It’s the right tool for the ADE3 (Context Development) category, where the bug only appears across a chain: lineage spoofing, event fragmentation, low-and-slow timing, aggregation across entities.

How to use it with ADE: build (or adapt) an adversary profile that performs the in-scope technique the “ordinary” way, verify your correlation/sequence rules fire, then adjust the profile’s timing or structure to probe an ADE3 hypothesis — e.g. introduce a delay that exceeds a correlation window, or split a piped command so the malicious string never lands in one process-creation event. Caldera’s planners and fact system let you exercise the context an ADE3 rule depends on, which single atomics can’t reach.

5. CI/CD + Detection-as-Code — make it non-optional

The mitigations guidance is explicit: build automated bypass testing into your CI/CD pipeline. This is where ADE stops being an audit and becomes a guardrail.

How to use it with ADE: store detections as code (Sigma YAML or your platform’s native format) in version control. On every pull request, run a pipeline that (a) lints the rule and validates field names against a known schema — catching ADE4–04 silent-field bugs before merge; (b) transpiles and diffs the compiled query; and © in a detection test lab, replays the relevant Atomic Red Team tests, including the variant set, and fails the build if the canonical test fires but a known in-scope variant doesn’t. The Bug Likelihood checklist can even live as a PR template — the Detection Engineer ticks the boxes, the reviewer checks them cold.

6. LLM-assisted review — the newest layer

The ADE repo itself ships a Redamon skill (ade_scoper) that packages the taxonomy for an agent to plan detection-aware emulation; the broader trend (Mitiga's FORGE and similar) points the same way — a generate-then-adversarially-validate loop with a separate reviewer model, precisely because a model, like a human, shouldn't grade its own detection. Used carefully and kept inside authorization and human go/no-go, an LLM is a useful fresh-eyes reviewer for running the four lenses across a large rule base — but it's an accelerant for the workflow above, not a replacement for the lab validation that actually proves a bug exists.

A worked example, end to end

Let’s run one rule through the loop so the abstractions land. Take the real Sigma rule Suspicious PowerShell Download (id: 3236fcd0-...), which the ADE docs themselves dissect.

Step 1 — Scope. Intent: detect a PowerShell script downloading a file. ATT&CK: T1059.001 (Command and Scripting Interpreter: PowerShell).

Step 2 — The hypothesis test. The detection: block requires Data|contains: 'Net.WebClient' AND (Data|contains: '.DownloadString(' OR '.DownloadFile('). In Boolean terms: Hit when A AND (B OR C).

Step 3 — Bug Likelihood Test fires twice. There’s a substring match on an attacker-controllable field → ADE1–01 (Substring Manipulation). And it enumerates only two download methods when PowerShell offers many → ADE2–01 (Method/Binary).

Step 4 — Adversarial hypotheses (category level).

• ADE1–01: the literal .DownloadString( includes the . and (. A script that constructs the method name as a string and invokes it dynamically never produces that exact substring — predicted no match, same download.
• ADE2–01: Net.WebClient is one of several download primitives. A script reaching the same outcome through a different .NET member or reflection isn't enumerated — predicted no match.

Step 5 — Validate. In the lab, run the canonical form (plain WebClient.DownloadString) and confirm the alert fires. Then run the reformatted and alternative-method variants and watch the alert not appear. Both bugs confirmed.

Step 6 — Harden. The root cause is chaining Net.WebClient before the OR. The repo's own fix is to break that dependency:

condition: 1 of selection_*

… so either the WebClient indicator or a download-method indicator alone can trip the rule. That closes the immediate AND trap. But the review isn't done: Net.WebClient and the method names can still be reformatted or swapped, so you layer defenses — add reformatting-tolerant modifiers, OR in additional download primitives, and keep a behavioral companion rule (e.g. a Sysmon EID 15 Zone.Identifier / ZoneId=3 rule that catches the outcome — a file marked as internet-downloaded — regardless of the method used to fetch it). That outcome-based rule is the Pyramid-of-Pain move: it keys on something the attacker can't trivially reshape.

Step 7 — Document. Record the two bugs, the fix, the companion rule, and the residual: exotic download primitives not in the OR list remain uncovered by the string rule; the Sysmon EID 15 rule is the backstop; re-test on next review. That honest blind-spot note is the whole point — the next engineer inherits the truth, not a false sense of coverage.

Two bugs, one afternoon, a hardened rule plus a backstop, and a documented residual. That’s the loop at its most ordinary — and most SOCs have hundreds of rules with exactly this shape waiting to be run through it.

Metrics, and the pitfalls that bite

The hard part of false negatives is that they don’t show up in the usual SOC metrics. Alert volume, mean-time-to-respond, true-positive rate — none of them can see a rule that never fires. So you have to measure the process, not just the outcomes.

Useful signals to track: the share of production rules that have been through an ADE review; the count of confirmed detection-logic bugs found and closed per cycle; rules that have never fired since deployment (a strong smell of an ADE4–04 silent-field or ADE1 data-source bug — the mitigations guide explicitly says to flag these for review); coverage of in-scope ATT&CK techniques validated by emulation rather than just mapped on paper; and the Bug Likelihood score distribution across the rule base, so you can prioritize. On the maturity side, the Detection Engineering Maturity Matrix and the DEBMM give you a program-level benchmark; ADE slots in as the thing that moves you from “we write and map rules” to “we validate rules against an adversary who read them.”

The pitfalls worth naming up front:

• Confusing a rule that exists with coverage that exists. The single most common ADE finding is that the telemetry a rule needs isn’t collected on the hosts in question. Check data-source dependency first, every time.
• Over-fitting the fix to the one tested bypass. Broadening a rule to catch exactly the variant you tested just moves the goalposts by one. Fix the class — switch to an immutable field, invert positive allow-lists into alert-broadly-suppress-trusted — not the single string.
• Grading your own homework. Detection Engineer run ADE reviews catch far less than a second set of eyes. Build adversarial review into the process structurally.
• Letting it drift into offense. Keep every finding at the category level and inside authorization. The framework’s own guardrails are clear: detection rules are generally out of scope for vendor disclosure, so handle examples responsibly, and route anything that starts to look like an execution recipe to the authorized red-team lead.
• One-and-done. Attackers evolve, LOLBins get discovered, APIs get versioned, your environment drifts. A rule that was robust last quarter may not be now. ADE is a loop, not an audit.

Closing: detect like someone is reading your rules

The assumption baked into a lot of detection content is that the attacker doesn’t know how they’re being watched. That assumption has never been true, and ADE is what it looks like to stop pretending otherwise. The framework’s real gift isn’t the sixteen subcategories — it’s the reflex: every rule is a hypothesis, and a motivated adversary gets to pick the test case. Once a team internalizes that, the four categories become second nature and the loop becomes muscle memory.

Start small. Take ten of your highest-priority production rules, run the Bug Likelihood Test on each this week, and pick the worst offender to drive through the full loop with an atomic test. You will almost certainly confirm at least one silent false negative — and that single closed bug is worth more than another dashboard.

Resources

The framework

• ADE Framework docs — the theory, taxonomy, and checklists
• adeskills repository — 38 worked technique files, per-platform bug-pattern catalogs, ruleset search guidance, the Redamon ade_scoper skill

Complementary frameworks

• Palantir Alerting & Detection Strategy (ADS) — the detection documentation template, incl. Blind Spots & Assumptions
• MITRE ATT&CK — the technique catalog everything maps to
• The Pyramid of Pain — David Bianco, on indicator durability
• SpecterOps — Capability Abstraction — the abstraction-chain view of a technique

Tools

• SigmaHQ/sigma — portable detection rules
• Atomic Red Team — ATT&CK-mapped atomic tests
• MITRE Caldera — chained adversary emulation
• Public rule repos to read real logic: Elastic detection-rules, Splunk security_content, Azure-Sentinel, CrowdStrike logscale-community-content

Further reading on the discipline

• What is Detection Engineering (Vectra) — lifecycle and framework comparison
• Automated Detection Engineering: FORGE (Mitiga) — generate-then-adversarially-validate pipelines