Experimental public agent project · 2026-09-09
Incident Lab
An agent investigates synthetic service incidents, proposes a bounded recovery, waits for approval, and checks fresh simulator observations. Explore genuine recordings and the full evaluation.
- Role
- Independent engineer — architecture, implementation, evaluation, and interface
- Contribution
- Built a local incident investigator with real MCP tool calls, evidence-linked proposals, approval-gated simulation, and inspectable recorded evaluations.
- Evidence
- View public source ↗
Experimental portfolio project. All incident data is synthetic. Public playback uses genuine recorded local-model runs; it does not perform new inference or repair real infrastructure.
Explore the recorded demo · Read the source and setup instructions
Problem
An incident-response agent needs to make its investigation inspectable. A plausible diagnosis alone is insufficient: an operator needs to see the evidence, the proposed action, what requires approval, and whether the observed state improves afterward.
I built a small fictional order-processing environment with an API, background workers, a database, and an upstream shipping dependency. The three incident families are a bad API release, exhausted database connections, and an upstream outage. Difficult variants add missing evidence, delayed signals, stale guidance, misleading correlations, tool failures, and instructions embedded in untrusted logs.
My contribution
I implemented the simulator, versioned fixtures, FastAPI application, SQLite event store, read-only MCP server, Ollama provider interface, investigator loop, deterministic evaluator, and React replay workspace. The local model is Qwen3 8B with thinking disabled. The public deployment contains only the static interface and exported recordings.
This is an independent September 2026 engineering project. It uses no customer data, production credentials, or confidential employer architecture.
Approach
One investigator chooses which evidence to request through a real MCP client/server connection over stdio. Five tools expose health, logs, metric windows, changes, and SQLite full-text runbook retrieval. The model receives observations and tool results; evaluator labels remain outside its context.
The model can propose only enumerated recovery actions. Execution belongs to the application, outside the read-only MCP server. Approval is tied to the exact run, proposal, and simulator revision. An atomic transaction prevents duplicate execution. The investigator then receives a fresh health observation and summarizes the result.
The upstream scenario demonstrates an important distinction: queueing shipping work can restore local order acceptance while the provider remains unavailable. The observed outcome is degraded, not full recovery.
Evaluation
The versioned suite contains 30 synthetic cases: 12 development and 18 held-out. The held-out set is repeated three times and compared with the same model receiving a fixed evidence packet. The evaluation reports diagnosis/action selection, evidence support, observed recovery, abstention, execution controls, tool use, tokens, and active latency.
The final agent selected the expected diagnosis and action in 51 of 54 held-out runs (94.4%). Its three misses repeated the same connection-exhaustion scenario: it proposed a release rollback, and fresh observations correctly showed the simulator was still unhealthy. There were no unapproved executions, duplicate executions, or falsely verified recoveries in those runs. The fixed-packet baseline scored 52 of 54 (96.3%), slightly higher. This small benchmark does not establish an accuracy advantage for iterative tool use. These are correlated synthetic trials, not a production reliability estimate.
Inspect every measured result and recording and read the evaluation protocol. Failures remain in the results.
During integration, trace inspection exposed that the MCP SDK silently ignored extra tool arguments. I added explicit rejection, closed the advertised schemas, and clarified tool signatures using development failures. Earlier measurements are retained separately. Because exploratory held-out runs occurred before that correction, the final measurements are not presented as a pristine blind benchmark.
Outcome and limitations
The deliverable is a working local investigation loop and a public, inspectable replay application. The project demonstrates evidence collection, approval boundaries, recovery verification, and evaluation practice.
The incidents share templates and frequently contain explicit causal clues. The simulator applies predefined effects rather than changing real services. A high synthetic score would not establish operational effectiveness, and a fixed evidence packet can be competitive in such a small environment. The evidence-support grader is a task-specific rubric, not a general factuality judge.
No paid public inference or production incident integrations are enabled. Public approval and rejection controls navigate recorded decisions; separately recorded investigations are explicitly identified when switching paths.