Architecture

Two services and one database, with a single writer.

Alertmanager ──►┌──────────────────────────────────────────┐
Slack ─────────►│  core-api  (TypeScript, Hono)            │◄── keryx CLI
MCP clients ───►│  · owns Postgres via Drizzle — SOLE writer│◄── the web app
                │  · Zod schemas = API contract = OpenAPI   │
                │  · SSE trace streaming (LISTEN/NOTIFY)    │
                └───────────────┬──────────────────────────┘
                                │ job dispatch / trace steps back over the same API
                ┌───────────────▼──────────────────────────┐
                │  investigator (Python) — STATELESS        │
                │  runs the loop over read-only MCP tools,  │
                │  POSTs trace steps + findings to core-api;│
                │  NEVER touches the database               │
                └──────────────────────────────────────────┘
      Postgres + pgvector · the Helm chart ships these three containers

Why the investigator never touches Postgres

One writer means one place where consistency is enforced. The investigator is a stateless worker: it claims a job over HTTP, runs the loop, and posts what it found. If it dies mid-investigation, the lease expires and the job returns to the queue — no half-written rows to reconcile.

Split-brain is handled with a fencing token, not a clock. A claim issues a monotonically increasing token; every trace step, finding and Slack post carries it, and core-api rejects a stale one. A worker whose heartbeat is refused aborts itself. Lease expiry alone cannot stop a paused-but-alive worker from double-posting; the token can.

The loop

Two phases, and the first one is fixed:

1. The triage sweep always runs and always produces trace steps — alert context, pod status and events, recent logs, key metrics, the Flux revision diff in the alert window, the alert rule's own expression. This guarantees a floor of evidence even when the model flails, and it snapshots evidence at webhook receipt, before any model call, because evidence decays.
2. The hypothesis phase lets the model pick tools to confirm or refute its top theories, one model call per round, with every stop condition sitting between rounds in plain code.

Stop conditions are hard, not vibes: a maximum tool-call count, a per-investigation token budget shared across retries, a wall-clock cap, and stop-on-plateau — two consecutive rounds with no movement emits best-so-far or "insufficient evidence".

Citations are structural

Findings are schema-validated objects whose every claim carries trace-step references. core-api rejects a finding that cites a step which does not exist, or whose quoted excerpt is not a verbatim substring of that step's stored output. Nothing here relies on the prompt asking nicely.

Telemetry is attacker-controlled input

Logs, annotations and Kubernetes object fields are written by the workloads under investigation — which makes them a prompt-injection channel. Tool output reaches the model inside nonce-fenced blocks the system prompt marks instruction-inert, control characters are stripped, and the model never executes strings from telemetry: MCP tools only, parameterised, no shell, no raw kubectl, no arbitrary URLs.

Bring your own key

Provider configuration lives in the investigator behind a ModelClient seam. Keys are operator-supplied through Helm values into a Kubernetes Secret and are never stored in Postgres, so there is no encryption-at-rest layer to get wrong. A pod refuses to start on missing or malformed key config, and the audit logger never logs key material.

Embeddings are pluggable the same way, with one honest caveat: a pgvector column has one fixed dimension and similarity search cannot span models, so switching embedding providers triggers a background re-embed of the corpus rather than a config flip.