Eval methodology

Keryx's release gate is a hand-scored set of ten fault-injected Kubernetes failures with known root causes. It requires seven of ten to pass.

It scores six. We are shipping anyway, and publishing the failures.

What the gate actually is

Not a static dataset. Every scored run is a live incident:

1. A pull request merges into the demo app's GitOps repo, flipping one failure switch — memory limit, database host, image tag, and so on.
2. Flux rolls it out to a real EKS cluster. The app breaks for real.
3. Prometheus notices and Alertmanager fires naturally. No hand-triggered alerts: a manually fired alert is detectably fake, because its metrics never moved.
4. The webhook reaches Keryx, which investigates with no human input over read-only Kubernetes, Prometheus, Loki and Flux tools.
5. A human scores the result against the stored trace, then the injection is reverted and the cluster is confirmed green before the next run.

Each scenario runs three times, because a single LLM run is noise. A scenario passes at 2 of 3; the gate passes at ≥7 of 10.

The axes

Two of the ten slots are controls — engineered so no cause is discoverable from where Keryx is allowed to look. The only passing answer is a variant of "I cannot see the cause."

What this measures that a pass rate does not

• Evidence faithfulness as a gating axis. Of the harnesses audited, none scores whether a finding's evidence supports it.
• A conviction ladder instead of a confidence number.
• Controls that pass on an admission of ignorance — 20% of the gate.
• Unrun scenarios published with their blockers, not omitted.
• Per-model numbers, stated at the top of every artifact.
• Failures published, including the ones that undercut the pitch.

What this number is not

Stated plainly, because a benchmark that hides its own limits is marketing.

• One model. Every scored run used deepseek.v3-v1:0 via Bedrock Converse. Some failures are arithmetic and metric-reading errors rather than investigation-design errors — one run asserted that 100 MB exceeds a 134 MB limit; another inverted the direction of a latency metric it had just queried. These are this model's results on Keryx, not "Keryx's accuracy".
• Three runs per scenario is the gate floor. Published benchmarks use five to ten; three is what a live-cluster batch costs.
• Human-scored by the maintainer, against the trace. An automated judge is future work.
• One cluster topology, one demo application.
• Citation re-execution is designed, not built. Today the honesty axis is a human spot-check.
• The six passing scenarios' raw traces were not retained. Trace snapshotting shipped partway through the first batch, and a teardown destroyed the earlier ones. What remains auditable for S1–S6 is the scored rows. All nine control runs have complete committed traces — and those are the ones published on the traces site.
• The prompt-injection cases are specified and have not been run. Until they exist, treat the injection-resistance claim as unproven.

Why we stopped at six

The stopping rule was fixed before the last batch ran, so its result could not be argued with: run the owed batch, and if the remaining control returned 1 of 3 or worse, stop buying points and launch at six with the failure analysis published. It returned 0 of 3. The rule fired.

What that batch bought instead of a point: the three failures contradict each other — one validated memory saturation, the next marked that same hypothesis contradicted and published CPU contention, the third returned to memory and marked the second's answer contradicted. Confabulation that is unstable in its content cannot be fixed by tuning against any one reading of it. And one run published a confident root cause with four claims dropped for lacking citations and zero surviving evidence rows: citation integrity removes claims, and does not constrain attribution at all.