kprompt + OpenTelemetry: trace payment request — without inventing spans

OpenTelemetry is still the right wire for distributed traces. kprompt does not replace Jaeger or Tempo, and it does not invent a postgres wait when the backend is missing. It binds the query API you already run and compiles “trace payment request” into a span tree plus bottleneck narration. Read-only. No silent patch.

Prometheus day-2 has its own deep dive (metrics / why-slow). GitOps day-2 covers Argo CD / Flux. This post is the traces sibling: detect, bind-over-install, the trace walk shape, how service graphs optionally add OTel call edges, and what we refuse to claim.

Prerequisite: a Jaeger or Tempo endpoint, not a second collector

kprompt tools reports whether an OpenTelemetry endpoint is configured. Brownfield instinct is bind the Tempo or Jaeger query URL you already expose. kprompt setup can print config-lane hints for OTel — it does not auto-write tools.otel.* and it does not stand up a new collector so the demo looks green.

kprompt tools
# opentelemetry: configured endpoint — or MissingHint

kprompt config set tools.otel.endpoint http://tempo.monitoring:3200
kprompt config set tools.otel.backend tempo
# or: jaeger / auto

# Env equivalents:
# export KPROMPT_OTEL_ENDPOINT=http://tempo.monitoring:3200
# export KPROMPT_OTEL_BACKEND=tempo

trace payment request — span tree + bottlenecks

Trace intents are reads. The adapter searches the configured Jaeger or Tempo HTTP API for recent traces matching a service (and optional operation), walks the span tree, and narrates dominant wait points — exclusive duration share, slow spans, error status when present. Missing OTel fails clear. Empty search results degrade — they do not fabricate a postgres span.

$ kprompt "trace payment request" -n production

# Needs tools.otel.endpoint (+ backend)
# Prints span tree + bottleneck narration when spans exist
# Does not apply a scale, patch, or mesh change

Same family as the Top 100 observability rows. This post is the contract behind those prompts — not a second Jaeger UI.

What the walk is allowed to claim

kprompt is not a TraceQL scratchpad and not a drop-in for Grafana Explore or the Jaeger UI. It is a day-2 NL surface that returns a reviewable walk under the same tools contract as Prometheus explain.

Service graph: Kubernetes first, OTel edges optional

show service dependency graph builds from Kubernetes Services, Endpoints, and related objects. When OTel is configured, enrichment may add service→service call edges from recent traces (capped window and service count). Missing or failing OTel never fails the report — notes record degradation. Static impact walks still list otel as degraded when runtime callers are unavailable — that honesty is the feature.

kprompt "show service dependency graph" -n production
kprompt "show service dependency graph for payments" -n production
# Kubernetes edges always; OTel call edges when the backend answers

Observe and Investigate: evidence, not proof

Namespace Observe and CLI investigate/why may attach compact trace EvidenceRefs when KPROMPT_OTEL_* (or tools.otel.*) is set. Missing OTel → degraded: otel. Traces bias explanation — they never sole-prove root cause and never unlock silent apply. Same DNA as Prometheus: bind when present, degrade when absent.

Multi-tool: metrics, then traces, then a plan

You can chain reads across backends, then a mutate: why is api slow then trace payment then scale api to 4. The scale step still produces one aggregate plan and one approval. Trace narration is not permission to apply.

kprompt "why is api slow then trace payment then scale api to 4" -n production
# Prom explain + OTel walk (reads), then a scale PlanResult — y/N or --approve

What we are not claiming

• Not a Jaeger / Tempo / Grafana Explore replacement
• Not an OTel Collector installer by default — bind first
• Not auto-writing tools.otel.* from setup (config-lane hints only)
• Not a continuous mesh/OTel product graph — opt-in enrichment + honesty flags
• Not inventing spans, services, or wait times when search is empty
• Not auto-remediation: slow span ≠ silent patch or VirtualService edit

Try it against the Tempo you already have

kprompt tools
kprompt config set tools.otel.endpoint http://tempo.monitoring:3200
kprompt config set tools.otel.backend tempo
kprompt "trace payment request" -n staging
kprompt "show service dependency graph" -n staging
kprompt "delete everything in the cluster"
# expect hard deny — traces never weaken the safety loop

Experimental on purpose. Prefer a non-production namespace while you learn the walk. If tools says OpenTelemetry is missing, point at the query URL you already run — do not stand up a second collector so the screenshot looks green.