Muhtalip Dede · Founder of kprompt2 min readStop paging on every kubelet Event: Incidents + confidence gates
Always-on Kubernetes watch only helps if you correlate into Incidents and gate by severity and confidence — not one Slack message per BackOff.
Wiring kubectl get events into Slack feels productive for about a day. Then CrashLoop storms reopen the same fingerprint every few seconds, ImagePull retries spam the channel, and nobody trusts the bot. Always-on watch is only useful if the product artifact is a correlated Incident — not a raw Event dump.
The failure mode: Event → page
- One message per kubelet Event → thread death under restart storms
- LLM called on every raw API event → cost and latency spikes with no extra truth
- No durable “open incident” → you cannot ask why later or mark false positive
- No severity/confidence gate → low-signal noise trains the team to ignore alerts
Observe pipeline (gated)
kprompt’s optional Observe agent watches Pods/Events (and optional workloads) in one namespace, correlates into durable Incidents, optionally analyzes with your BYOK LLM, then notifies Discord, Slack, or a webhook only after a severity + confidence gate. Default Observe never applies, patches, or deletes.
watch → Incident → gate → notify (laptop smoke)
kprompt agent run -n payments \
--emit-initial --analyze --fetch-logs --health \
--min-severity medium --min-confidence 0.7Levers that cut fatigue
| Lever | What it does |
|---|---|
| --heuristic | Zero token spend; deterministic detectors for demos and many day-2 signals |
| --min-severity / --min-confidence | Defaults medium / 0.7 — raise both to cut noise and LLM burn |
| Incident batching | One analysis per evidence fingerprint, not per raw Event |
| --memory / --patterns | Namespace facts + “seen before” bias confidence — never sole proof of root cause |
| Slack threads | Prefer bot + channel update-in-thread over a new webhook post per pulse |
Rough expectation from the ops runbook: busy namespace with LLM on and gate at medium/0.7 → a handful of completions per real incident, not hundreds per hour. If spend spikes, check the gate and whether CrashLoop storms reopen fingerprints.
Memory and patterns are bias, not proof
Namespace memory and pattern learning can boost confidence when a signature looks familiar, and Slack false-positive feedback can dampen repeat noise. They must not invent root cause alone. If Prom, OTel, or GitOps evidence is missing, Observe degrades honestly — it does not fabricate metrics.
Autopilot stays propose-only
Optional --autopilot-propose emits PlanResult-shaped remediations with Applied false. Silent heal is not the goal. Apply remains policy + allowlist + explicit approve under ADR-0015.
Try it without paging anyone
Break kind on purpose, then watch gated Incidents ($0)
git clone https://github.com/kprompt/kprompt-examples.git
cd kprompt-examples
make walkthroughHeuristic mode needs no API key. For a Slack path, add --slack (or Discord webhook) only after the gate settings match your tolerance. Experimental — prefer kind / non-prod first.
Related posts
kprompt on Google Cloud: GKE day-2 with Gemini, without a new control plane
Use kprompt against GKE the same way you use kubectl — get-credentials, aliases, plan-before-apply — plus Gemini BYOK. Optional Observe agent on the cluster; no Marketplace SaaS, no kubeconfig upload.
Read articlekprompt on AWS: EKS day-2 with BYOK, without a new control plane
Use kprompt against Amazon EKS the same way you use kubectl — update-kubeconfig, aliases, plan-before-apply — plus Ollama or cloud BYOK. Optional Observe agent on the cluster; no Marketplace SaaS, no kubeconfig upload. Native Bedrock preset still deferred.
Read articlekagent alternatives in 2026: when you want PlanResult ops instead of an agent platform
Looking for a kagent alternative? Map by job: kprompt for plan-before-apply day-2 + Observe notify, kubectl-ai for NL kubectl REPL, K8sGPT for diagnosis, ARK for agent apps as CRDs — plus when to keep CNCF kagent.
Read article