Building an Agentic SOC on a Stream

Modern Security Operations Centers (SOCs) are transitioning towards agentic architectures. Traditionally, SOCs relied on rigid, rule-based automation and high human effort. We built an Agentic SOC powered by AI agents that don't just follow rigid, hardcoded rules, but also autonomously investigate, reason, and act like human analysts.

In practice, detection was never our bottleneck, but high-volume, low-fidelity alert streams demand immense manual effort, meaning they naturally receive the least attention. This creates the exact blind spot where patient attackers love to hide. To prevent this, we built an automated investigator rather than a simple classification tool. This architecture unites a central triage coordinator, specialized evidence agents, an adversarial evaluator, and a self-learning knowledge base. 

Over a 30-day period, our detection stack produced roughly 4,700 alerts. Priority-based routing sent about 3% of them to on-call. In contrast, the automated pipeline analyzed every alert, escalating roughly 5% and surfacing more than 250 true positives for analyst review.

The Denominator Problem

By continuously pulling in security signals and cloud audit anomalies, our detection stack did exactly what it was built to do. The core bottleneck remains human ability because investigation capacity stays fixed while alert volume scales relentlessly.. Every security team copes with this reality the same way: they sort by severity, work down the queue, and abandon the lowest-priority items.

Data Loss Prevention (DLP) alerts often end up in an unexamined backlog. Because these alerts are high in volume and mostly benign, reviewing them manually is rarely practical, which gives attackers a convenient place to hide. As corporate integrations or organizational growth drive alert numbers higher, leaving these lower-priority signals unscrutinized becomes an unsustainable security risk. 

Moving Beyond Simple Classification

Every security leader has weighed the standard options: hire more analysts, raise alert thresholds, or automate. We chose to automate, but not by simply asking an AI model to guess if an alert is malicious. An alert never contains its own verdict. For example, an isolated signal like a suspicious process execution requires deep context to resolve. You need to know what else ran on that host, normal user behavior, and if an analyst previously documented the pattern as benign.

This context lives scattered across various security tools and the team's collective memory. Therefore, the goal cannot just be classification, it must be a true investigation that forms a hypothesis and queries systems for evidence. This depth takes a few minutes per alert, rather than seconds, but it provides the critical reasoning required behind every decision.

A Smarter Agent Architecture

The intuitive approach is to build a specific AI agent for every detection source. While fast to set up, this creates a maintenance nightmare. Every new source demands a new agent, becoming a permanent tax on your engineering team. Instead, we specialized our agents by evidence domain and treated source-specific quirks as data rather than code.

Core evidence-gathering skills are stable and worth hand-crafting into dedicated agents. Conversely, day-to-day context changes constantly and belongs in a dynamic knowledge base that updates itself. Now, onboarding a new detection source requires a simple configuration file. Within a few alerts, the knowledge base learns the source’s unique quirks automatically, naturally growing the system's accuracy over time.

Inside the Investigation Process

Our investigation process is structured to be efficient and thorough. Before any AI models run, we filter out duplicate alerts to save resources. The remaining alerts are then enriched with the team’s existing institutional knowledge.

A central triage coordinator reviews this information to decide if the alert can be resolved immediately. If it requires further investigation, the coordinator dispatches specialist agents. These specialists focus on specific evidence domains, like endpoint forensics or cloud audit trails, allowing them to excel at their distinct tasks. They run deep queries and return compact, clear findings to the coordinator. Next, a separate evaluator agent reviews the findings to actively look for untested hypotheses or unsupported conclusions. If gaps exist, the case is sent back to the coordinator for further review.

Finally, the system records what it learned into a knowledge base. Findings are embedded and retrieved with hybrid vector-plus-keyword search, so past investigations surface by meaning rather than exact wording. And because an incorrect memory is worse than no memory, entries go through semi-automated curation before they can steer future verdicts.

The Power of Adversarial Review

The largest jump in our automated verdict quality did not come from a better model or more tools. It came from introducing a secondary agent whose sole job is to find holes in the first agent's work. AI investigation agents often jump to conclusions that are only partially supported by evidence. While these conclusions read well, they fall apart when challenged to prove untested hypotheses. By requiring our agents to survive this automated peer review, we transformed basic opinions into rigorous, reliable investigations.

Humans Remain in Control

Automation that silently closes critical alerts is a massive security risk. Because of this, we designed an automated system that has strict boundaries. It is only permitted to automatically close low and medium-severity alerts that it assesses as completely benign. It cannot silence anything high-severity, even if the AI is 100% confident a critical alert is a false positive, the case stays open and requires human sign-off. When a human does step in, they are handed a fully formed ticket with the entire investigation already completed, saving significant time while maintaining human oversight.

Treating Alerts as a Continuous Stream

Initially, we treated our AI agent as a standard application reading from a queue. However, security alerts are actually an unpredictable, continuous stream of events. Problems like low-priority alerts delaying critical ones are fundamentally data-streaming issues.

By moving our pipeline to a streaming architecture, we effectively organized the chaos. We established priority lanes so that critical alerts bypass the queue, ensuring low-priority volume never monopolizes system capacity. High-volume, deterministic routing is built on top of Confluent Cloud for Apache Flink®, while the complex, expensive judgment work is reserved for the AI agents.

Closing the Loop

The pipeline runs in both directions. Agent telemetry (per-alert and per-model latency, token spend and cost, verdicts, tool usage, evaluator loop counts) streams back into Confluent Cloud as its own Apache Kafka® topics. The Real-Time Context Engine, a fully managed feature of Confluent Intelligence, serves that live context directly to AI agents, so we can ask about the SOC's current state and get answers grounded in what happened seconds ago, not in last night's dashboard build. Latency spikes, cost regressions, and quality drift surface while they're still easy to fix. 

The Results

The success of this system comes down to coverage. Every alert now receives a full investigation before any escalation decision gets made. Over a specific 30-day period, the pipeline surfaced more than 250 true positives at roughly a 5% escalation rate, more than the entire on-call ticket count for the period. This completely transformed how we handle historically unmanageable feeds.

Our headcount didn’t change. Instead, we changed where our team’s time goes: analysts now spend their hours on critical investigations that truly require human judgment, while the massive tail of low-priority alerts no longer goes unread. AI didn't replace our analysts, it supercharged them by turning an overwhelming queue into a clear set of actionable priorities.