Operational intelligence · AI incident investigation

AI Incident Investigation & SRE Automation

An evidence-driven investigation system that can inspect telemetry, revise a hypothesis when facts disagree, propose a bounded action, wait for human approval and verify the result.

Automation may investigate. A human authorizes change.Tools collect evidence; an isolated executor and explicit allowlist bound remediation.
MaturityImplemented local reference system
SurfaceReact control room + FastAPI agent service
AuthorityExplicit approval before remediation
EvidenceRetained synthetic live-run trace
The operating problem

Incident response is a sequence of evidence updates, not one model answer.

Logs, metrics and configuration can contradict an initial hypothesis. A credible agent must retain the investigation trail, distinguish observation from inference, and stop before operational change unless the action is allowed and approved.

Incident signal symptom and initial context
↓
Tool-guided investigation logs · metrics · configuration
↓
Evidence-backed hypothesis revised when facts contradict it
↓
Bounded remediation proposal allowlisted action only
↓
Human approval explicit change authority
↓
Recovery verification same session, retained evidence
Implemented system

A control room around the investigation agent.

Agent session

Investigation state, hypotheses, tool calls and approval remain tied to one inspectable session.

Evidence tools

Logs, metrics and configuration are separate tool boundaries rather than prompt-only context.

Isolated executor

Remediation runs inside a constrained Docker execution path.

Allowlisted change

The agent cannot invent arbitrary operational commands.

Human approval

Proposed remediation pauses until the operator explicitly authorizes it.

Verification

The session checks post-action evidence instead of assuming the change worked.

Retained run evidence

The hypothesis changed when the telemetry changed.

Payment timeout→Initial hypothesis→Contradictory evidence→Revised diagnosis→Bounded proposal→Human approval→Recovery check

A retained synthetic run shows the agent revising its hypothesis, proposing a bounded remediation, receiving explicit approval and verifying recovery from the same investigation session.

What this evidence does not prove

It does not establish production SRE performance, sustained recovery, lower MTTR, broad service coverage, production security approval or Microsoft production-agent behavior.

Current scope

An observable reference slice, not autonomous production operations.

The current system demonstrates investigation workflow, tool boundaries, evidence retention, approval and verification in a local/synthetic environment. Production connectors, policy ownership, tenancy, secrets management, workload isolation, rollback strategy and service-level validation remain deployment-specific work.

Related system

Explore governed customer operations across AWS and Azure.

Explore Customer Operations ↗