Part II: Inside Alert AI Analysis: How We Built an Agent Harness for Incident Investigation

TL;DR:

This is the engineering companion to our announcement post, Upgraded Alert AI Analysis: Automated Incident Investigation, read that one for what the new generation does for your team; read on for how it works under the hood.

Interested in hearing more? Book a demo to see the Alert AI Analysis Agent live.

Root cause analysis is one of the harshest tests you can give an AI. When production breaks, a report that is merely plausible isn’t good enough: engineers need to know what the findings are based on, and they need the analysis to hold up under the same scrutiny a manual investigation would. Fluency alone is not competence, and getting past that gap takes more than a good prompt.

The first generation of Alert AI Analysis, a single prompted agent, established AI-assisted incident investigation and ran thousands of real-world analyses. Building its successor, we made one decision that shaped everything else: the new generation is not a bigger prompt or a better model. It’s an agent harness.

1) The harness

What an agent harness actually is

The name comes from the test bench. A harness is the rigid structure you bolt an engine into so you can run it hard: it holds the engine in place, feeds it controlled inputs, measures everything it does, and keeps it from tearing itself loose. The engine provides the power; the harness provides the guarantees.

An agent harness is the same idea applied to LLM agents. The models provide the reasoning, reading log patterns, forming hypotheses, writing the analysis. The harness is everything engineered around them that doesn’t depend on a model behaving well: what runs, in what order, with which tools, under which output contracts, and with which independent checks. Concretely, ours has five load-bearing parts:

• An orchestration layer. A coordinating agent that plans the investigation and delegates to specialists, and structurally cannot skip the process, because it holds no data tools of its own.
• Scoped specialist agents. Each investigator gets exactly the tools for its signal domain, its own context window, and an independent brief.
• A typed output contract. The verdict is a schema enforced in code, with validators that actively correct inconsistent output, not a formatting suggestion in a prompt.
• An evidence ledger. Every piece of telemetry any agent retrieves is recorded outside the model, so claims can be verified against what was actually observed.
• Server-side verification. Deterministic checks, code, not another LLM, that classify every citation and every claimed number in the final verdict against that ledger.

The through-line: in a harness, the important properties are guaranteed by construction. A prompt can ask a model to only state what the data supports. A harness can check.

2) Architecture

Specialists, coordinated: one incident, every signal

Modern incidents rarely live in one signal. An error spike in the logs, a performance shift in the metrics, a deployment event, a related ticket, the cause is usually in the correlation between them. So the harness’s defining architectural choice is easy to state: the orchestrator that writes your analysis has no data tools at all. It cannot search a log or fetch a metric. The only things it can do are plan, delegate to specialist agents, and synthesize what they return.

Each specialist is scoped to one domain, a logs agent, a metrics agent, and, where integrations are connected, an integration agent for tickets, deployments, and other operational systems. The telemetry layer underneath is backend-agnostic: the same investigation runs against different observability backends through adapters that expose what each backend is actually good at. Access to external integrations goes through a hardened HTTP layer, domain allowlists, DNS-rebinding defense, and credentials bound to their own service so a token for one system can’t be replayed against another.

If this shape looks familiar, it should. In our webinar From Raw Telemetry to Actionable RCA, we laid out an RCA blueprint of four moves, every incident, in order: Orient (one timeline, one clock), Isolate (what changed?), Hypothesize (rank candidate causes by evidence), Verify & Act (confirm, take the smallest safe action, record), under a motto we take seriously when building: framework first, AI second. Automate chaos and you just get faster chaos; a bad process plus AI is confident, wrong answers, delivered faster. The harness exists to run a good process:

• Orient. The orchestrator starts every investigation by surveying the connected signal domains and planning, before any data is touched.
• Isolate. Independent specialists work each domain, logs, metrics, and change context from connected systems, asking what changed, without contaminating each other’s narrative.
• Hypothesize. Synthesis promotes only causes an investigator backed with a specific datum, hypotheses with receipts, never hunches.
• Verify & Act. Grounding checks every claim and every number against the evidence ledger, and the verdict closes with concrete, fact-tied recommendations.

We’ll say it plainly: the harness doesn’t run the whole blueprint end to end yet. Orient doesn’t yet pull similar past incidents onto the timeline, and Act stops at recommendations rather than approval-gated execution. But this generation is the major step in that direction, the moves it does run, it runs in order, by construction.

One detail we care about a lot: the specialists are dispatched in parallel, in the same turn, and their briefs are deliberately kept independent, the logs agent’s brief is not allowed to reference what the metrics agent found, and vice versa. That’s a bias defense. Sequential investigation anchors: logs-first investigations produce log-shaped causes, metrics-first investigations produce metric-shaped causes. Independent parallel investigation, then synthesis, is the same reason you want two engineers looking at an incident from different angles before comparing notes.

Synthesis is where correlation actually happens, and it’s rule-bound. The orchestrator may only promote a finding into the verdict if an investigator backed it with a specific datum, and it must carry that datum’s evidence reference through unchanged. And we don’t rely on the prompt alone: when only a single signal domain is connected, server-side code deterministically strips cross-domain correlation claims from the verdict, because a correlation the system could never have observed has no business in the report.

3) Grounding

Every claim carries a receipt

The harness’s verification machinery attacks the plausible-but-unsupported claim head-on. Every tool result an investigator retrieves, every log histogram, every metric series, every sample set, is content-hashed into a short evidence ID, something like [e:3fa4c1], and recorded in an evidence ledger scoped to that investigation.

The output contract then requires the agent to attach one of those IDs to every finding, every causal step, every piece of contradicting evidence, copied verbatim from the tool result that produced it. The model isn’t allowed to paraphrase evidence into existence: no ID, no claim.

Because the ledger lives outside the model, we can verify the report without trusting the reporter. After the verdict is produced, server-side checks, deterministic code, not another LLM, classify every citation: does this ID exist in the ledger? Was it minted by a tool call that actually returned data, or by an empty result? Is the same ID being reused across claims it can’t support?

Then we go a level deeper, because there’s a subtler failure class in the wild: an analysis can cite real data points correctly and then invent smooth intermediate numbers between them to make the narrative read better. Citation checking alone can’t catch that, the citation is real, in-window, non-empty. So numeric grounding extracts every count, rate, and percentage claimed in the verdict and checks the number itself against a typed projection of the cited tool result, classifying each one as grounded, misattributed, fabricated, or uncited.

One thing you won’t do as a reader is bookkeep any of this. The citations are working machinery, not homework: before the report is streamed to you, they’re stripped out at the wire, so what an engineer reads is a clean narrative. The full cited verdict never leaves the backend, it’s retained there for the verification layers and the audit trail. The receipts exist so the system can check the report, not so you have to.

Two honest caveats, because grounding invites overclaiming. First, what this machinery guarantees is traceability: every claim maps to something the agent actually retrieved, and claims that don’t are detected and can be acted on, it is a verification system, not a proof that no model anywhere can be wrong. Second, the guarantee is grounded in retrieved telemetry: if the investigation queried the wrong window entirely, its claims can be faithfully grounded in the wrong data. That’s why enforcement rolls out gate by gate, each layer measured in detection mode before it’s allowed to rewrite a verdict, and why the report carries an explicit list of evidence gaps instead of papering over what wasn’t checked.

4) Standardization

One verdict schema, enforced in code

Every investigation returns the same typed verdict, the same fixed schema, every time. Status. Confidence tier. Key findings. The causal chain. Contradicting evidence. Evidence gaps. Recommendations. An executive summary. Rendered for humans as a report card; retained in full, citations and all, for audit.

Two design choices make this a standard rather than a template. First, the schema’s field order is the reasoning order. The model must emit its evidence fields, data sources checked, key findings, contradicting evidence, evidence gaps, before it is allowed to emit a status, a root cause, or a confidence tier, and the executive summary comes last. Structured output follows field order, so the schema itself forces the model to show its work before concluding. “Conclusion first, justification after” is exactly the order in which unsupported analyses get written.

Second, the schema fights back. Validators actively rewrite inconsistent verdicts server-side: a verdict claiming root_cause_found with zero supporting findings is downgraded to insufficient_data before anyone sees it; a confident tier that its own evidence doesn’t support gets the same treatment. There is no mid-stream retry with an LLM, once the model has committed to an overconfident answer, the guardrail has to exist in code.

And crucially, insufficient_data is a respectable answer. The output contract includes worked examples teaching the model to refuse tempting conclusions, including a strong log/metric correlation whose causal mechanism lives in a layer the system has no visibility into. Strong correlation is not enough, a lesson learned in production, encoded in the contract rather than wished for in a prompt.

The payoff of standardization compounds: reports are comparable across incidents, reviewable against a fixed rubric, and machine-consumable, dashboards, escalation policies, and postmortem tooling can all be built against one verdict shape instead of parsing prose.

5) Triggers

From alert to investigation

An analysis is most valuable in the minutes after something breaks, which is usually when nobody has time to prompt an AI. So investigations no longer wait for a human to ask: Alert AI Analysis can be activated directly on Open 360 AI alerts, and when the alert fires, the investigation starts. The same harness serves an engineer asking a question in chat, a scheduled health check, and an alert-triggered run, one investigation architecture, multiple doors in. By the time you open the incident channel, the correlated, cited first pass is already waiting.

6) Closing

The delta, in one sentence

The first generation proved the idea: a language model can drive a real incident investigation. The harness completes it: it plans before it digs, sends independent specialists at every signal you have, refuses conclusions the data doesn’t support, writes the same rigorous report every time, and staples the receipts to every claim.

For what this means for your team day to day, automatic investigation on Open 360 AI alerts, evidence-backed findings, a consistent investigation format, see the announcement post: Upgraded Alert AI Analysis: Automated Incident Investigation.

FAQs