Part of the series: Six False-Greens: Field Notes from a Self-Auditing Agent Pipeline
Every green on your eval dashboard is a lie if your parser is permissive enough. I learned this the hard way when my walker was recording passes on agent runs where the agent's actual verdict had never been read.
The Failure Mode Nobody Warns You About
When you build an agent pipeline, you spend most of your design budget on what the agent does — the tools it calls, the memory it reads, the context it receives. The output contract gets treated as a detail. You define a JSON schema, you write a parser, the parser handles the happy path, and you move on. What you implicitly decide in that moment — without realizing you're deciding anything — is what happens when the output is malformed, truncated, or missing a key field entirely. Most implementations make the same choice: be permissive, default to None, log a warning, keep moving. That choice is the bug.
The failure I'm describing isn't loud. There's no exception, no stack trace, no alert firing at 2 AM. The pipeline runs cleanly. Every downstream metric looks fine. The agent ran, the walker logged a result, the eval suite stays green. The only thing that didn't happen is that the agent's actual judgment was never incorporated into any decision. The system produced confident output from a null signal.
This is what I mean by a false-green factory: infrastructure that converts absence-of-signal into positive-signal at scale, silently, on every run.
What the Walker Actually Does
In my fleet, the walker is the component that dispatches agents, collects their envelopes, and decides what to do with the result. It sits between policy gating and downstream execution. When an agent finishes, it returns a structured envelope — a block of JSON that contains the verdict, the reasoning trace, confidence metadata, and any structured outputs the agent produced. The walker reads that envelope and either propagates the result, escalates, or stops.
flowchart TD
A[Agent Run Completes] --> B[Agent Envelope Returned]
B --> C[Walker: Extract Last Fenced JSON Block]
C --> D{Valid Result Block?}
D -- Yes --> E[Read Verdict Field]
E --> F{Verdict}
F -- Pass --> G[Propagate Result Downstream]
F -- Fail --> H[Escalate]
F -- Refused --> I[Stop]
D -- No / None --> J[STOP and Raise — fail-closed]
The critical section is what I call the §5.3 result block: the authoritative verdict field that tells the walker whether the agent passed, failed, or refused to decide. Everything downstream depends on reading that field correctly.
The contract between agent and walker (the "§5.3 result block") was the root cause. Specifically: the walker was not extracting the result block from the last fenced JSON block in the agent's envelope. It was scanning for a result key in the full output string, finding the first match — which in some cases was a JSON fragment in the agent's reasoning trace, not the final structured verdict — and using that. When no match was found, it returned None. And None, by default, was not a STOP condition. It was treated as a pass.
The mechanism is almost elegant in how it fails. The agent reasons through a problem, and in the middle of that reasoning it might emit partial JSON — an illustrative example, a tool-call payload, a scratchpad structure. The walker finds that fragment, extracts a field that happens to share a name with the verdict key, and records it as the agent's conclusion. Or the output is clean but the last fenced block is simply absent, and the walker silently defaults. Either way, the downstream system receives a confident signal from noise.
Why "Be Permissive" Feels Right and Isn't
The instinct toward permissive parsing comes from a real place. LLM outputs are stochastic. Agents sometimes truncate. Edge cases multiply faster than you can enumerate them. If you make your parser strict and it breaks on a formatting variant, you've introduced fragility for no user-visible benefit. Being permissive feels like defensive engineering.
The problem is that this reasoning applies the wrong risk model. In a data pipeline where you're parsing logs or user input, a missing field usually means you skip that record and continue. The cost of a miss is a gap in your data. But in an agent eval pipeline, a missing verdict field doesn't mean "no data for this run" — it means "the agent's judgment about whether something is safe, correct, or complete was not recorded, and we are about to proceed as if it said yes." The cost of a miss is a false pass propagated into every downstream system that trusts the eval result.
Permissive parsing in a data context is often the right call. Permissive parsing in a trust-gated agent context is a systematic bias toward false positives. These are not the same problem.
The Fix: Fail-Closed at the Contract Boundary
The solution has two parts, and both matter equally.
Parse the Last Fenced Block, Not the First Match
The agent's envelope is a sequence of reasoning, tool calls, and structured outputs, with the final verdict at the end. The authoritative result is always the last fenced JSON block in the envelope — not the first occurrence of a result-shaped key. Updating the walker to seek backward from the end of the envelope and extract the final fenced block removed the class of failures where a mid-reasoning JSON fragment was mistaken for a verdict.
The implementation is straightforward: find all fenced JSON blocks in the envelope, take the last one, attempt to deserialize it against the result block schema, and fail explicitly if deserialization fails. No fallback to the first match. No scanning the raw string for key names. The last block is the contract; everything else is scratchpad.
Every None Path Is a STOP
The second part is the policy change. Before the fix, a None result from the parser was treated as a pass — the walker would log a warning and continue. After the fix, None is a refusal. If the parser cannot extract a valid result block, the walker does not proceed. It does not default to pass. It does not default to fail. It stops and raises, because the correct response to "I don't know what the agent decided" is not to guess.
This is the fail-closed principle applied at the contract boundary. The system's default state — in the absence of a clear signal — is stopped, not running. This means some runs that previously silently passed will now explicitly surface as errors. That's the right tradeoff. A noisy error you can investigate is strictly better than a silent false pass you can't see.
The change to the walker's None handling is a one-line policy decision with significant operational consequences. Runs that previously passed silently will now require investigation. Agents that occasionally emit malformed envelopes will now surface as reliability issues rather than disappearing into the noise. The eval suite will look noisier in the short term. It will be more accurate.
What This Changes About Eval Trustworthiness
Evals are only as trustworthy as the pipeline that runs them. If your eval harness has a permissive parser, your eval pass rate is not measuring agent quality — it's measuring a combination of agent quality and parser tolerance. Separating those two signals requires making the parser strict enough that a pass means the agent actually produced a valid, readable verdict, not just that the output didn't crash the extractor.
After applying the fail-closed fix, the eval suite became a more accurate instrument. Runs that surfaced as new errors weren't regressions — they were previously invisible failures that the permissive parser had been absorbing. The green count dropped. The green rate became meaningful.
This is the core tension in eval infrastructure: strict parsers produce noisier-looking results, which creates pressure to loosen them, which produces cleaner-looking results that mean less. The pressure to loosen is real and constant. Resisting it requires treating parse failures as signal, not noise.
Operational Consequences and Tradeoffs
Making the walker fail-closed introduces a new class of operational event: runs that stop at the contract boundary because the agent's envelope was unreadable. These need to be triaged differently from runs that stop because the agent explicitly failed. A contract boundary stop means the agent may have done the right work but emitted a malformed envelope — that's a formatting reliability issue. An explicit agent failure means the agent ran and decided the answer was no. Conflating them produces bad debugging.
The triage pattern I use: contract boundary stops go to an envelope inspection queue; explicit agent failures go to the standard failure review flow. The two queues have different owners and different remediation paths. Keeping them separate is worth the overhead.
There's also a latency consideration. Extracting and deserializing the last fenced block is slightly more expensive than scanning for a key name in raw text. In a high-throughput pipeline this is measurable. In a trust-gated agent fleet where the unit of work is an agent run, it's noise. Know which regime you're in.
FAQ
What is a false-green in an agent eval pipeline? A false-green is a run that the eval harness records as a pass even though no valid agent verdict was extracted. It happens when the result parser defaults to a passing state on malformed or absent output rather than stopping.
What does fail-closed mean at the contract boundary? Fail-closed means the system's default state, in the absence of a valid signal, is stopped rather than continued. Applied to the result block contract: if the walker cannot extract a valid verdict from the agent's envelope, it stops and raises instead of defaulting to pass.
Why parse the last fenced JSON block instead of the first? Agent envelopes typically contain reasoning traces, tool-call payloads, and scratchpad structures before the final verdict. Scanning for the first occurrence of a result-shaped key can match a mid-reasoning fragment rather than the authoritative output. The last fenced block is the contract terminus.
Doesn't a strict parser make evals noisier? Yes, in the short term. Runs that previously passed silently will surface as contract errors. That noise is accurate signal — it represents failures that were previously invisible. A lower pass rate from a strict parser is more informative than a higher pass rate from a permissive one.
How do you triage contract boundary stops versus explicit agent failures? Route them to separate queues. A contract boundary stop means the agent may have done valid work but emitted an unreadable envelope — that's a formatting reliability issue. An explicit agent failure means the agent ran and returned a negative verdict. The remediation paths are different and should not be merged.
The Transferable Lesson
The absence of a signal must never be interpreted as a positive signal. This principle applies wherever you have a structured contract between two components and one of them can fail to produce output — which is every component boundary in every pipeline you will ever build. Permissive parsing feels like resilience. It is actually a systematic mechanism for converting failures into false passes. Make your contract boundaries fail-closed, treat None as a refusal, and accept the short-term noise of surfaced errors in exchange for the long-term accuracy of a trustworthy eval suite.