It is 2 a.m. and an alert just fired. Somewhere between pulling the logs, checking the user's recent activity, reading the event context, and writing up a summary, an analyst has to answer one question: is this real? That question takes time. Anywhere from 30 minutes to over an hour depending on the alert type and volume. Time the attacker already has. Now picture that same process playing out hundreds of times a month across every part of the stack, and you start to see the problem.
This is the story of how our Security Operations team at Fullscript built an automated L1 triage system that performs that entire first pass in under five minutes for every alert, across every severity, giving analysts a structured assessment before they even open the ticket.
The Problem We Were Solving
Our SOC monitors security events across a wide surface area: cloud infrastructure, identity and access management, endpoint protection, web application firewalls, network firewalls and VPN, email security, SaaS platforms, and cloud security posture management. Each of these systems generate alerts when something looks suspicious, and all of those alerts need to be investigated.
The challenge was never detection. Our alert coverage is strong and getting stronger. The challenge was the time between an alert firing and an analyst understanding what actually happened. That gap is where risk lives. Every alert is a question: is this real, or is this noise? The faster you answer that question, the smaller the blast radius when the answer is "real."
We have hundreds (if not thousands) of enabled alert rules spanning P1 through P4 across all platforms. Some fire daily. Some fire weekly. Some fire once a quarter. But every single one needs the same initial investigation: pull the logs, read the context, assess severity, and decide what to do next.
Manual triage does not scale to that volume. This is not a team size problem, it is a throughput problem. There are only so many hours in a day, and alerts do not wait for business hours.
What We Built
We built an automated pipeline that sits between our alerting infrastructure and our analysts. When a security alert fires, the system performs the same investigative steps a human analyst would, and delivers the results as a structured triage report attached to the tracking ticket. By the time an analyst opens the issue, the initial assessment is already there.
At a high level, the pipeline does six things:
1. Receives and normalizes the alert. Security alerts arrive from different platforms in different formats with different severity scales. The first step standardizes everything into a common structure: what fired, when, how severe the source system thinks it is, and what entities (users, IPs, hostnames) are involved.
2. Keeps everything in sync. A single alert can span multiple tools. The pipeline keeps them all in sync bidirectionally, so when an analyst updates a status in one place, it reflects everywhere. No one has to manually close the same alert in three different tools or chase down context scattered across systems. The coordination happens automagically.
3. Fetches the actual evidence. This was one of the harder problems to solve. Alert payloads contain metadata about what happened, but they rarely contain the actual log data (in some cases, this is by design). The system takes the alert's metadata (time window, source, filters) and queries our SIEM's log API to retrieve the raw events that triggered the alert. For high volume alerts with thousands of events, it runs statistical aggregation queries first, then pulls a representative sample, so the AI gets both the big picture and the details.
4. Cross references with surrounding activity. Alert logs alone do not tell the full story. The system also pulls in the broader context around the event: what was the identified user doing across other systems during that time window? Was there a recent infrastructure change, a deployment, or a configuration update that would explain the alert? This cross referencing is what turns raw log data into an informed assessment. An alert that looks suspicious in isolation often has an obvious explanation once you see the merge request that was deployed ten minutes earlier or the user's login activity across other platforms.
5. Routes to a domain specific AI agent. This is where we moved from "good" to "great." Our first version used a single, generic AI agent for every alert type. It worked, but it produced generic assessments. A brute force attempt against an identity provider and a web application firewall block are fundamentally different events that require different investigative logic. We built 13 specialized AI agents, each trained on its own domain with tailored playbooks and context. A cloud infrastructure agent analyzes through the lens of privilege chains and service automation patterns. An endpoint agent reasons about process trees, code signing, and persistence mechanisms. A network firewall agent evaluates traffic patterns and VPN session context. Each agent thinks like a specialist, not a generalist.
6. Delivers a structured triage report. The AI output follows a consistent format: what happened, what indicators were found, what systems are affected, a severity assessment with reasoning, recommended immediate actions, and an escalation recommendation. This gets posted directly to the tracking ticket so analysts have a complete first pass before they begin their own review.
The Details That Made It Work
Building the pipeline was straightforward in concept. Making it reliable across a thousand different alert types and a dozen different log formats was where the real work happened.
Log retrieval turned out to be the hardest problem. Our SIEM stores logs across multiple tiers with different query interfaces and different data retention windows. Alert metadata arrives in one format, but log queries need a different syntax entirely. We built a translation layer that converts alert filter expressions into the query language our SIEM's API expects. That translator has been through more than a dozen iterations, each one handling a new edge case we discovered from production alerts. Case sensitivity mismatches between metadata and stored labels. Time window calculations that pointed at the wrong slice of data. Cardinality alerts that report unique value counts instead of total event counts. Each bug taught us something about the gap between how alert systems describe events and how log systems store them.
High volume alerts needed a different approach. When an alert represents 40,000 events in a 30 minute window, pulling all of them is impractical. We built a two path system: alerts under a threshold get full log retrieval, while alerts above it get a statistical breakdown (grouped by the most meaningful fields for that alert type) plus a small representative sample. AI receives both the forest and a few trees.
False positive detection improved dramatically with infrastructure awareness. Early on, AI had no sense of what "normal" looked like in our environment. It would flag a service account running a scheduled job as suspicious, or treat traffic from a known office location the same as traffic from an unfamiliar source. An experienced analyst carries institutional knowledge about what is normal in their environment. They know which service accounts run nightly jobs, which network ranges belong to the company, and what routine operational activity looks like. We gave our agents that same institutional knowledge. Each one understands the environment it is analyzing, so it can reason about whether an event is expected behavior or something that genuinely stands out. That shifted triage quality significantly, because the difference between a true positive and a false positive is often just context that a generic model would never have.
Multi provider AI keeps triage running around the clock. A triage pipeline that goes down when an AI provider has an outage is not much of an autopilot. We built the system with provider failover: if the primary model is unavailable or returns an error, a backup model from a different provider picks up the work automatically. The analyst sees the same structured triage report regardless of which model produced it. This was a deliberate design choice. Reliability matters more than squeezing the last percentage point out of any single model, and in practice, both providers produce strong triage output because the real value lives in the investigative framework we built around them: the specialized playbooks, the enriched context, and the institutional knowledge each agent carries into every analysis.
The MSSP integration closed the communication loop. Many of our alerts route through a managed security partner. Their analysts post findings and questions in a dedicated Slack channel, threaded per alert. We built an automation that monitors those threads, matches each one to the corresponding internal tracking ticket, and syncs the conversation as comments. When an internal analyst opens a ticket, they see the AI triage report, the MSSP's findings, and the full conversation history in one place, without switching between tools.
What Changed for the Team
In a world where attackers increasingly use AI to find and exploit vulnerabilities at machine speed, the speed of your response is not just operational efficiency, it is your actual defense. The most visible change here is exactly that: speed. Every alert, from P1 to P4, gets a full triage assessment within minutes of firing. By the time an analyst opens a ticket, the investigation context is already laid out.
But the less obvious change is cognitive. Triage is repetitive, high volume work that consumes the mental energy analysts need for the hard problems. Automating the first pass frees them to spend their time on the alerts that actually need human judgment: the ambiguous ones, the ones that cross multiple systems, the ones where the answer is not in the logs but in a conversation with the affected user.
We also gained consistency. A human analyst triaging their 50th alert of the day will naturally cut corners. The automated system applies the same rigor to alert number 500 as it does to alert number 1. Every alert gets its logs pulled, its entities enriched, its context analyzed, and its severity assessed, regardless of when it fires or how many came before it.
Lessons Along the Way
Start generic, then specialize. Our first version used a single generalist AI agent across every alert type. Even that delivered real value: alerts that previously had no automated analysis were getting structured triage reports within minutes. But the quality gap between a generalist assessment and a domain aware one became obvious fast. We started building specialized agents incrementally, beginning with the highest volume alert sources, and each one raised the bar for that category. If we had tried to build all 13 agents before shipping anything, we would have delayed value by months for the sake of perfection.
The translation layer is the bottleneck, not the AI. Most of our bugs were in log retrieval: wrong time windows, missing filters, case mismatches, query syntax edge cases. The AI itself was remarkably good at producing useful analysis once it received the right data. Getting the right data to the AI is the real engineering problem.
Automated systems need the same observability you give production services. We built health monitoring alerts for the automation itself: if log retrieval consistently returns zero results, if the AI produces truncated output, if the workflow execution time spikes. Treating the triage pipeline as a production system, not a side project, was essential to keeping it reliable.
Keep humans in the loop where judgment matters. The system does not auto close alerts. It does not take containment actions. It does not message users on behalf of the SOC. It produces an assessment and hands it to an analyst, who decides what to do next. The value is in accelerating understanding, not in removing human decision making.
What Comes Next
With L1 running on autopilot, we are now stepping into L2 automation where it matters most, while keeping quality front and center. One area is using triage data to continuously fine tune our alert rules: identifying patterns that consistently produce false positives and tightening the detection logic to reduce noise without losing the signals that matter. Better rules upstream means higher quality alerts downstream, which makes every part of the pipeline more effective.
We have also built automated user outreach into parts of the triage flow. For certain alert types, like impossible travel, unfamiliar country logins, and new location detections, the system already identifies the affected user and reaches out to ask whether the activity was intentional. Their response feeds directly back into the ticket, giving analysts one more verified signal before they make a call. We are now expanding that pattern across more alert categories, closing the loop between detection and validation without requiring an analyst to chase someone down manually.
Another area we are investing in is making the translation layer between alert metadata and log retrieval smarter. Today, when a new alert format or edge case surfaces, the system requires manual tuning. We want to move toward a self adapting layer that can recognize new formats and adjust its query construction automatically, so the system gets better on its own with every new alert it processes.
Security operations at scale is fundamentally an information problem. The signals are there. The challenge is turning a firehose of events into a narrative an analyst can act on, fast enough to matter. Automation is not here to replace the analyst. It is here to make sure that when they sit down to investigate, the story is already written.
