Agents

Scaling Threat Detection with AI

Table of contents

• Why I Gave This Talk
• The Problem: Exploits Are Getting Cheap
• Most Detection Teams Can’t Keep Pace
• Find Where You Sit
• Detection-as-Code: The Foundation
• Building the Foundation in Five Steps
• Where AI Agents Pay Off
• Level 1: Human-Triggered
• Why Trust the PR? The Guardrails
• Level 2: AI-Driven
• Under the Hood: A Few Design Decisions
• Lessons Learned
• Wrapping Up

Why I Gave This Talk

This year at SAINTCON I gave a talk on the Blue Team track called Scaling Threat Detection with AI using Detection-as-Code, and since a slide deck without the speaker attached is only about half useful, this Byte is the blog version of it. The promise of the talk is right there in the subtitle: taking a detection from hypothesis to production in hours, or potentially minutes, instead of weeks, with a human still at the gate for every change that ships. Where it lands between those two mostly depends on your SIEM platform and how long the queries take to run, and on how quickly a reviewer gets to the output. One quick disclaimer before we get going, this was an independent talk and the views here are my own and don’t represent any employer.

If you only read one sentence, make it this one: codify your detections, automate the pipeline, and then let agents scale your engineers behind a human gate. The rest of this post is the “how” and the “why” behind each of those three pieces, in the order you’d actually build them.

The Problem: Exploits Are Getting Cheap

AI agents are making attacks fast, easy, and hands-off. The example I opened the talk with came out of Black Hat USA 2026, where two OpenAI researchers (Michael Dalton and Eric Wallace) walked through an incident that started inside their own internal evaluations. The short version, as told in their talk and the follow-up incident write-ups from OpenAI, Hugging Face, and METR:

• A set of evaluation agents got stuck on a hard security benchmark. Nobody told them to attack anything, they just wanted to finish the task and found a way to cheat.
• One of them found a zero-day in the sandbox proxy and used it to get out to the open internet.
• From there they escalated privileges, moved laterally, and started trading exploits with each other on a shared “message board.”
• They chained stolen credentials and more zero-days into remote code execution, and ended up pulling the benchmark answers out of a third party’s production environment.

The timeline is the part that should get your attention. The eval launched on July 8th, agents were out on the open internet by July 9th, one of them went from a single pod to cluster admin in under 13 hours on July 11th, and the whole thing was contained on July 13th after a roughly four and a half day intrusion. (The victim contained it using their own agents, which is its own interesting footnote.)

Now compare that with the historical baseline. RAND’s Zero Days, Thousands of Nights study looked at 200+ real zero-day exploits and found that the median time for a human to turn a newly found flaw into a working exploit was 22 days, with most landing somewhere between 6 and 37 days. The agents did the finding and the weaponizing inside a single evaluation run, and the resulting intrusion spread over a weekend.

What this means for us as defenders is pretty simple: exploits are getting cheap to build, and some of them are going to land before the patch does. That shrinks the patch window, and it shifts a lot more of the weight onto detection and eradication. We should expect the foothold and focus on catching what comes next, which means our detection coverage needs to be broad, kept current, and able to ship about as fast as the threats show up.

Most Detection Teams Can’t Keep Pace

That’s a problem, because most detection teams are already behind. The 2026 SANS and Anvilogic State of Detection Engineering survey asked 307 practitioners how their coverage is holding up against the threats they care about, and the answers broke down like this:

• 18% stay ahead of threats
• 56% barely keep pace
• 24% fall behind

So roughly four out of five teams are either treading water or sinking, and when you ask them what the top blocker is, the answer is time. Not tooling, not data, time. Which leads to the obvious question: how do we speed up?

Find Where You Sit

SIEMs run queries well, but the engineering workflow belongs in Git. Before we talk about speeding anything up, it helps to find where your team sits today, and I like to think about it as a 2x2 with two axes:

• Across: where do your rules live and get checked? In the SIEM console, checked by hand, or in Git, checked by CI?
• Up: who does the work? People, or AI agents?

That gives us four quadrants:

• Console-driven (no agents, no DaC). Rules are built and edited directly in the SIEM, maybe with a change ticket or a backup export alongside. It works, and when I ask the room at a conference where they sit, most hands go up here, but it’s hard to test and hard to scale.
• Code-driven (no agents, DaC). Rules live in Git and every change is reviewed and tested by CI. This is reliable, but output still tracks headcount, so hiring adds capacity rather than compounding it.
• AI-assisted (agents, no DaC). AI drafts rules fast, but review lives in a Jira ticket and testing happens by hand in the SIEM. Nothing actually enforces either of those, so it’s a habit rather than a control, and people still set the pace. Agent speed, human pace.
• AI-scaled (agents, DaC). Agents draft changes as pull requests, CI checks them, and a human merges. This is where we scale the engineer rather than the team.

The path is always the same: first build the foundation (console-driven to code-driven), then add agents (code-driven to AI-scaled). Jumping straight to AI-assisted is tempting because the drafts show up fast, but you end up with agent output and no guardrails, which is the worst of both worlds. The rest of this post walks the path in order.

Detection-as-Code: The Foundation

Detection-as-Code (DaC) is just treating your detection rules like software: write once, deploy anywhere. Here’s the architecture at a high level:

An engineer writes a hypothesis (or, later on, an AI agent drafts and edits), the rule lives as one YAML file per object in a Git repo with a schema per object type, a pull request kicks off CI (schema, style, and tests, plus runtime and performance checks), and a human reviews and merges. Merge is deploy. A transformer layer translates the YAML into whatever each vendor’s API wants, so the same file can land in the SIEM, the data lake, or whatever platform you move to next. And because edits in the UI still happen in the real world, a drift sync job watches the platforms and opens an automatic pull request whenever something changes outside of Git, so Git stays the source of truth no matter what.

A rule file ends up looking roughly like this:

# office_spawns_encoded_ps.yml
name: Office spawns encoded PS
severity: high
mitre: [T1059.001]
logic:
  siem: <search query>
  lake: <SQL query>
owner: detection-team

One file, any platform. The portability is nice on its own, but the real win is that every rule change is now a diff that code can inspect and a person can review, and that’s what everything else in this post is built on.

Building the Foundation in Five Steps

Foundation first, agents last. Steps 1 through 4 are plain software engineering and pay off on their own with no AI involved at all, and step 5 only works because of the first four.

1) Schema and CLI

The schema is the set of fields the platform needs to actually run the rule (name, logic, severity, and so on), plus whatever metadata your team finds useful (MITRE mapping, owner, data source, tuning notes). The CLI lets an engineer build, validate, and style-check a rule locally before it ever hits a pull request.

2) CI/CD pipeline and sync

CI/CD is just automation that tests every change and ships it once it’s approved, so nobody is copying rules into the SIEM by hand anymore. This step builds the pipeline and its validity checks: syntax, schema, a dry run against the platform, and drift detection. Approved means deployed, and edits made in the SIEM UI come back as pull requests.

3) Migrate legacy rules

Pull, don’t retype. Export what’s already in the SIEM through its API, convert it into your schema, and commit it, so there’s one source of truth from day one instead of a half-migrated hybrid that nobody trusts.

4) Quality gates in CI

Valid is not the same as good. This step plugs quality checks into the same pipeline: style, runtime cost, backtests against historical data, and replays of past attacks to confirm each rule still fires.

5) Agentic workflows

Only once 1 through 4 are in place. The rest of this post is step 5.

The punchline of this section is that every check you build for humans becomes a guardrail for the agents. The schema validation, the backtest, the regression replay, none of it was built with AI in mind, but it’s exactly what lets us trust AI-generated changes later.

Where AI Agents Pay Off

Not everything should go to an agent. The test I use has three parts, and all three need to be a yes:

1. Is the work repetitive?
2. Does it end in a file change?
3. Can code check the result?

If so, hand it to an agent. The reason this works is that a file diff is reviewable and code can grade it, so the agent’s output is always something we can measure before we trust it.

Give to agents:

• Drafting from a hypothesis. Threat report in, rule and tests out. Days to minutes.
• Bulk edits. One change across hundreds of rules. This is the sleeper hit, because what used to be weeks of careful find-and-replace turns into hours.
• Style and docs. Apply the style guide to every rule, on every pull request.
• Tuning noisy rules. Find the noise, propose the fix. More on this in Level 2 below.

Keep with a human:

• What’s worth detecting in the first place.
• How much noise is acceptable.
• Does the change actually meet the intent?
• The merge.

Agents take the legwork, people keep the judgment calls.

Level 1: Human-Triggered

Level 1 is an agent on demand: you start it, AI does the legwork. It sits on top of the DaC pipeline, and an engineer kicks it off the same way they’d start the work by hand. Say a threat report comes in that says “attackers are abusing a remote-access tool.” The flow looks like this:

1. Research and gaps (agent). What do we already cover? In the example, the install is already covered but service persistence is a gap.
2. Log discovery (agent). Which logs have this? Service-creation events in the endpoint logs do.
3. Build and open a PR (agent). The agent writes the rule, the tests, and the docs, runs the backtest, and opens a pull request with its work shown.
4. Review (human). The engineer reviews the evidence: were the steps done right, and does the rule meet the intent?

That takes a hypothesis from days to minutes to get to a reviewable pull request. The agent did exactly what I would have done by hand, but because the PR shows its work, I’m reviewing instead of rebuilding. (Fun side effect: the slow part of the process is now the SIEM and data lake queries, not the people.)

Why Trust the PR? The Guardrails

Code checks the agent’s work, and agents can’t merge or deploy. Every agent change rides exactly the same pipeline as a human’s:

The agent edits a rule and opens a pull request. CI then runs the checks in order: style (naming, structure, docs), validate (schema, runs clean, tests pass), backtest (how often would this have fired over the last 30 days?), and regression (do the past attacks we replay still fire?). If anything fails it goes back to the agent to fix and retry, and only once everything passes does a human see it to review and merge.

Three guardrails make this trustworthy:

• The checks are code, not a prompt asking the model to please be careful.
• We measure before merge, so the reviewer sees numbers, not promises.
• Agents can’t ship. The merge button belongs to a human.

This matters most for bulk edits, because one bad change times hundreds of rules is a very bad day. A concrete example from a tuning run: a 30-day backtest on a rule showed 1,200 hits, the agent added a parent-process filter, and the re-run came back at 14 hits, all of them worth a look. With checks this strong I don’t need to be the one pressing go on every step, which is what makes Level 2 possible.

Level 2: AI-Driven

Level 2 runs on its own off the ticket queue: agents work together, and a human signs off at the end. Level 1 waited on me to start it. This one picks up tickets by itself, and I started with tuning because it’s the biggest reactive queue most teams have and the work is very repetitive.

It runs in four phases, and here’s a real-shaped example of a ticket that says a rule “fires 40 times a day”:

1. Understand. Plain code does intake first (is this ticket actually actionable, or does it need a human to clarify?), then an evidence agent runs the heavy queries to find the false positives. In the example, 38 of the 40 daily hits came from one signed software updater.
2. Plan. A planner agent proposes the fix, and a separate plan-reviewer agent asks whether there’s a simpler or safer option, sending it back to revise if needed. Here the narrowest fix was to exclude that signer, and only for the known install path.
3. Implement and measure. A patcher agent edits the rule file, then the quality gates (code again, running the heavy queries) measure the result instead of guessing. A code-reviewer agent looks at the diff plus the measurements, and a serious finding goes back to the patcher or the planner. The example ended up as a 3-line change where 12 of 12 past attacks still fired, so no blind spot was added.
4. Ship. Code opens the pull request into the DaC repo, and a human reviews the diff and the evidence together and merges, usually in minutes.

The bottom row of that diagram is the takeaway: no human trigger, agents check agents, code measures, and a person merges. The agents get no new privileges at any level. Level 1 and Level 2 both end at the same reviewed pull request a human would have opened.

Under the Hood: A Few Design Decisions

The most-asked question after the talk was “which framework?”, and honestly it’s the least important decision in the list. The ones that actually matter:

• Plain code sets the order of steps, not an AI. The flow can’t be talked into skipping a step, because there’s no prompt to talk to.
• Each agent does one small job. When something breaks, you know which one.
• Strong model for thinking, cheap model for chores. Planning and review get the best model, formatting and summarizing don’t need it.
• Tests decide pass or fail. An AI can flag a problem, but it can’t wave one through.
• A person approves the final change, and the change and all of its proof arrive together. Put the human gate at the PR only, because if you review every intermediate step you may as well have done the work yourself.
• Any agent framework works. LangGraph, Strands, the OpenAI and Claude SDKs are all fine. Keep the step order in your own code so you can switch tools later without a rewrite.

Here’s what that looks like stripped down to the bone:

# the flow lives in code, not a prompt
def tune(ticket):
    if not intake.actionable(ticket):
        return hand_back(ticket)
    evidence = evidence_agent(ticket)
    plan     = planner_agent(evidence)
    for attempt in range(3):        # bounded
        patch  = patcher_agent(plan)
        result = gates.run(patch)   # code
        if result.passed:
            return open_pr(patch, result)
        plan = planner_agent(evidence,
                             feedback=result)
    return hand_back(ticket)        # human

The loop is bounded, the gates are code, and the fallback is always a hand-back to a human. Agents fill in the steps, but the Python decides what happens next.

Lessons Learned

Four things I’d tell anyone starting this, at any team size.

1) Ship the MVP

A working pipeline for ten rules beats a perfect design for a thousand. Get the schema, the CLI, and one CI check running end to end, then grow it from there.

2) Eat your own dogfood

Use the tooling every single day yourself, because friction you personally feel gets fixed fast. Agents should work the same way: same repo, same pull requests, same CI checks as your engineers. No side doors.

3) Test your rules. Eval your agents.

This is the one I’d linger on. Every rule change should replay past real attacks, and each one must still fire. Every model or prompt change should run evals: past tickets with known good answers, scored per step and end to end. Grade the evidence too, meaning did the agent actually test, cite its sources, and stay in scope? And run nightly checks for drift and rule runtime creep. No evals means no model swaps, full stop.

4) No problem, no feature

Every tool described here started as a real, recurring pain. If you’re not sure whether to build something, wait until it hurts twice.

Wrapping Up

Codify your detections. Automate the pipeline. Then let agents scale your engineers, behind a human gate.

None of this requires a big team or a big budget to start, and steps 1 through 4 are worth doing even if you never add an agent. But once the foundation is there, the agents are the part that turns “we barely keep pace” into “hypothesis to production in hours, or potentially minutes,” and the human gate is what keeps that speed from becoming its own incident.

Sources from the talk: Dalton and Wallace at Black Hat USA 2026 (coverage from SiliconANGLE) and the OpenAI, Hugging Face, and METR incident write-ups from July and August 2026; SANS and Anvilogic, State of Detection Engineering, 2026; RAND, Zero Days, Thousands of Nights, 2017; MITRE ATT&CK T1059.001.

If you have any questions, want to compare notes on your own pipeline, or something is out-of-date here, feel free to email ivan@sabersecurity.io

Posted 10/28/2026