index / writing
How I think about making agents trustworthy and affordable in production — the same methods behind the audits and teardowns.
A check the agent cannot skip removed every silent failure in 40 injected runs at the cost of no check. As an optional tool it doubled the bill and let 6 through.
Per-call accounting for an LLM self-check: 64% more cost, zero wrong answers caught because none reached it, and false alarms that ate 37% of the budget.
I pinned one model id. A month later the API served another for the same request, and my same-model gate passed. What it actually checked, and the fix.
The headline metric was 'wrong things marked correct.' Nine times the harness measuring it turned out to be doing the same. A field report on attacking your own eval before you trust it.
One model rejected all 50 planted bugs when checking a single answer. Over 12-step tasks, all 45 wrong runs claimed success. What that means for agents.
After fixing six agent failure modes, the system graduated from outcome checking to path checking. A field report on trajectory evals and generalization.
58 runs, ~93% autonomous pass, and why that number is honest evidence — not a reliability proof. A field report on agent evals at solo scale.
How fail-closed design, headless execution seams, and converting incidents to fixtures stop AI agents lying about success. A field report from a solo dev-loop.
A strict contract enforced on only one side produces false rejections as reliably as a loose one produces false acceptances. A field report from my fleet.
A PR merged over a red CI job while dev-loop QA said green. Here's the root cause, the fix, and why partial greens are the most dangerous lies in agent systems.
How a silent None from an AI agent registered as a pass — and the fail-closed contract that fixed it. A field report from a production agent fleet.
Six ways my autonomous dev-loop reported success when nothing had landed — and the fail-closed gates that caught each one before they corrupted the ledger.