Building AI SRE in Public #6: Investigation Graph

This is episode 6 of Building AI SRE in Public. Episodes 2–4 gave us a compiler, a typed PlanResult, and a safety wall. Episode 5 stretched that wall across kubeconfig contexts. Investigation Graph answers a quieter design question: when the work is wide — many signals, many namespaces — what shape should the AI take? A chat line that queues “and then” forever, or a gated graph that fans out, verifies independently, and merges into one refuse-able artifact?

The claim

Chat is a UI. Ops is a graph with gates. Natural language (or continuous Observe) is the source. Investigation / InvestigationReport / PlanResult are the IR. Approval is the link-edit. Apply is the binary. Verify closes on evidence — not on a second model call that inherited the analyzer’s session and politely agreed.

Lines vs graphs

Most multi-step agents become a line: step one, step two, step three — each waiting whether or not it needed the previous output. Half those waits are fake edges. The habit that starts everything: for every “and then,” ask whether the next step actually reads the previous step’s output. If no data crosses between two boxes, they are independent — candidates for fan-out, not forced queues.

If you cannot find two boxes with no arrow between them, there is no graph to build. Stay a loop. Forcing a fleet onto a true chain only burns money and invents failure modes.

The gated Investigation Graph

Signal nodes (Events / logs / Endpoints / …)
    → Reason nodes (RCA / findings)
    → Verify edge (independent EvidenceRef / probe / hard deny)
    → Merge artifact (Investigation / Report)
    → optional PlanResult + safety
    → approve → apply → post-apply verify

CLI path (on-demand): investigate multi-hop → Investigation JSON → optional suggested PlanResult → approve → apply → verify. Runtime path (always-on): Observe → Detect/RCA → InvestigationReport → Coordinator handoff → probe/verify → merge → notify → optional Autopilot propose (never silent apply). Soft product illustration — the contract matters more than the binary name.

Independent verify — the part that actually breaks

When an agent checks its own work in the same conversation, it goes easy on itself. A graph of agents sharing one context is a single loop in a costume — it fails later, more expensively, with more green lights on the way down. Verify edges must rest on anchors the optimizer cannot invent: fresh EvidenceRef / probe reads, schema and risk stamps, hard denies in policy code, post-apply readiness checks. A second LLM in the same session is agreement in a different font.

• Shipped: Coordinator merge caps soft-agree without probe EvidenceRef (confidence ≤ 0.4)
• Shipped: CLI pre-trust (`internal/pretrust`) clamps high confidence without EvidenceRef / contradicting re-read
• Shipped: suggested-fix approve UX withheld when pre-trust fails; T-070 after apply
• Non-goal: “looks good” as a substitute for EvidenceRef

Worker isolation

Parallelism without isolation races. Namespace Agents stay Role-scoped. Coordinator mutate stays off by default. Foreign-ns facts travel only via CoordinatorHandoff / reply — not inventing peer root cause locally. Two writers sharing one mutable workspace is an anti-pattern; forbid it structurally.

What we parallelize (and what we do not)

On investigate today: Explain (Deployment → RS → Pods) runs beside Service/Endpoints discovery when both only need the target identity. Endpoints Gets fan out per matching Service. After pods are known, Events and Logs fan out — no fake “and then.” The Deployment → ReplicaSet → Pods chain stays sequential because each hop reads the last.

kprompt "investigate api" -n payments
kprompt "investigate api" -n payments -o json | jq '{summary, confidence, degraded, findings: [.findings[].code]}'
kprompt "why is api crashlooping" -n payments

What this is not

• Not Claude-style “1000 agents in one window” / free-form dynamic workflow fleets
• Not competing with Kagent as a general multi-agent framework
• Not silent or default LLM-said-so apply
• Not treating chat transcript as the IR
• Not uploading raw cluster dumps to a control plane by default

Why this belongs in an AI SRE series

Episode 3 said every production AI agent needs an approval boundary. Episode 4 said the boundary is empty without a typed plan to guard. Investigation Graph says the investigation that feeds that plan must itself be honest under width: independent verify, frozen anchors, isolated workers. Otherwise you scale confidence theater — more agents, more green lights, same Goodhart failure.

What ships vs building

• Shipped: investigate / why / timeline → Investigation; PlanResult → approve → apply → verify
• Shipped: Observe → InvestigationReport → Coordinator handoff + probe + independent merge
• Shipped: named Investigation Graph + reality-anchors docs; pre-trust; hop parallelization
• Building: richer Prom/OTel hops still listed in degraded[] until wired
• Non-goal: agent-fleet chat product; unsupervised Autopilot

Try the shape

# kind + a broken workload (see kprompt-examples)
kprompt "investigate api" -n payments
kprompt "investigate api" -n payments -o json | jq '.confidence, .degraded, .suggestedPlanHint'
# Review PlanResult if a suggested fix appears — never --approve blind

If your “multi-agent ops” story cannot show the last Investigation a human refused as data — and a wipe-class prompt that cannot apply — you still have a chatbot with kubeconfig. Draw the graph. Keep the anchors frozen.

Next

Episode 7 is AI Timeline — ordered EvidenceRef chronology as a first-class artifact, not a chat scroll of “what we tried.” The hub tracks the rest of the arc.