Friday night CrashLoopBackOff: diagnosing a checkout service with kprompt instead of kubectl roulette

It is Friday night. Slack lights up: checkout is failing in staging, then a quieter ping that production payment confirmations look flaky. kubectl get pods shows the familiar red: CrashLoopBackOff on the checkout Deployment. Restart count climbing. Nobody wants to play kubectl roulette — get, describe, logs, wrong logs, logs --previous, events, guess, patch, hope.

This is the first post in kprompt in the wild: real operator scenarios, not product manifesto. The reference ladder for CrashLoopBackOff still lives in our CrashLoopBackOff guide. Here I walk what we actually did with kprompt — read first, plan second, approve last.

What we saw

Namespace payments. Deployment checkout. Three replicas, zero Ready. Last State exit code 1 — application error, not OOM (137) and not ImagePullBackOff. That already rules out “just bump memory” and “registry is down.” The useful logs are on the previous container, not the one currently starting.

kubectl get pods -n payments -l app=checkout
# NAME                         READY   STATUS             RESTARTS
# checkout-7d8f9c4b6d-xk2n4    0/1     CrashLoopBackOff   12

Ask for the ladder, do not invent it

Instead of hand-rolling five kubectl commands in the wrong order, we asked kprompt to explain. Reads run immediately — no approval when nothing mutates. The explain path walks Deployment → ReplicaSet → Pods → Events → previous logs and reports what it found.

kprompt "explain why checkout is crashing" -n payments
kprompt "logs checkout" -n payments
kprompt "why is checkout not ready" -n payments

What came back matched the classic dependency shape: checkout started, tried the orders Service, got connection refused, exited 1. Events showed Back-off restarting failed container. The model can still be wrong — that is why we kept the previous logs and the Service/endpoints check in view. In our case the orders pods were not Ready either; checkout was the noisy symptom, not the only patient.

Investigate when one hop is not enough

When the first explain points at a dependency, investigate (or why) helps stitch the hops without opening twelve terminals. Still read-only. Still no mutate.

kprompt "investigate checkout" -n payments
kprompt "why is checkout CrashLoopBackOff" -n payments

The finding we trusted: orders ConfigMap had a bad DB host after a partial roll. Orders could not become Ready; checkout kept dying on connect. Fixing checkout alone would have been theater.

The fix still became a plan

We did not --approve a blind “restart checkout.” We asked for a bounded change: restore the last-good ConfigMap value for orders (or roll back the bad revision), then re-check checkout. Mutations produce a plan with risk and a diff. On a TTY you answer y/N. That boundary is the point when you are tired and Slack is loud.

$ kprompt "rollback orders config to last good" -n payments

Plan
  1. rollout undo Deployment/orders   # or patch ConfigMap — review the diff

Risk: medium
Approve? [y/N]

After orders came Ready, checkout stopped looping without a heroic checkout-side patch. We still verified with a short read: explain / get pods — Ready 3/3. If the plan had proposed delete namespace or an unscoped wipe, hard-deny would have stopped it before apply. We did not need that scare this night; we needed the habit.

What I would tell next on-call

• CrashLoopBackOff is a symptom — read exit code and previous logs before any patch
• Use explain / investigate for the ladder; keep kubectl when you need a ticket paste
• Never restart-loop a Deployment hoping the dependency heals itself
• Mutations stay plan → approve → apply — even on Friday night
• Practice on kind first: kprompt-examples scenario 01-crashloop

git clone https://github.com/kprompt/kprompt-examples.git
cd kprompt-examples
make up && make break SCENARIO=01-crashloop && make verify
kprompt "explain why api is crashing" -n payments

What is next in this series

Next up from this lane: ImagePullBackOff after a registry rotate — plan the fix before production. Install path if you want to try the same loop locally: Install and Quickstart. Deeper CrashLoop mechanics: the CrashLoopBackOff guide.