When the generator outgrows the verifier

"The only way to make sense out of change is to plunge into it, move with it, and join the dance." -Alan Watts

Where we left off

In June's post, I walked through a failure mode in causal reasoning over an energy-industry corpus: a strong-but-not-frontier LLM (Llama-3.3-70B) flattening causal structure, treating negative wholesale prices and two-sided CfDs as independent risks rather than symptoms of a common constraint. The fixes were two-sided: GraphRAG (Neo4j entity graph + vector retrieval) and Chain-of-Verification (CoV), which decomposes the draft into atomic claims, re-grounds each one with targeted retrieval, and rewrites, keeping only what survives.

That left an obvious question hanging: how much of the problem was the generator? If you swap the 70B model for a frontier model, does the causal chain still collapse and does verification still earn its keep?

I ran that experiment with Claude Fable 5 (Anthropic's current frontier model), holding everything else fixed: same corpus (five IEA reports), same retrieval, same gpt-4.1 verification stack, same causal-correctness metric (an independent judge scores each causal claim in an answer as supported/inferred/unsupported against retrieved evidence; score = (supported + 0.5·inferred)/total).

The short version: the frontier model fixed the reasoning failures, exposed a failure mode in the verifier instead, and forced a redesign of CoV. Each in turn.

Part 1 — The frontier model fixes the causal collapse (but not the grounding problem)

First, the no-retrieval floor. Same minimal prompt as June's baseline ("You are a helpful assistant"), no context, all five benchmark questions:

No-retrieval floor: Llama vs Fable 5 (causal correctness, tuned-RAG evidence pool)

Two things stand out.

The qualitative collapses are gone. On the CfD question, Fable 5 gets the causal direction correct without retrieval. Solar fleets depress prices exactly when they produce; two-sided CfDs are correctly identified as a policy response to that dynamic, not an independent driver. On the coal question, Llama had inverted the 2025 US policy direction entirely. Fable 5 correctly identifies the causal sequence: the emergency declaration, the Section 202(c) orders keeping scheduled-retirement plants online, the MATS exemptions, and the correct synthesis that this converts coal's retirement path into a plateau without restoring its competitiveness.

But look at Q5. The frontier model scores worse than the 70B model on the most niche question. Why? It knows more. It wrote a 3,400-token deep-dive making 27 causal claims about tax-credit mechanics and feedstock economics, most of them plausibly true in the real world, but 16 of 27 absent from the evidence pool the judge scores against. The metric measures corpus groundedness, not truth, and that's the right thing to measure for an auditable briefing.

That's the cleanest demonstration I've seen of why retrieval still matters for frontier models: more knowledge produces more claims, and without retrieval, more claims means more unverifiable assertions. The floor rises from 0.415 to 0.500, nowhere near what grounding provides.

Part 2 — Frontier generator + GraphRAG: the generation problem is basically solved

With the same GraphRAG retrieval from June (vector seeds + entity-graph expansion, eight chunks per query):

The frontier drafts also do something subtler: they mine the retrieved chunks harder. Where the 70B drafts stuck to the headline facts of each chunk, Fable 5 pulls specific figures out of chunk interiors: regional generation upticks, capacity trajectories, utility-level procurement details, and cites them. The drafts are denser with verifiable specifics.

Which is exactly what broke the verifier.

Part 3 — The failure mode: verification retrieval can't find what the draft used

Running the original CoV on top of the Fable 5 drafts, verification flagged 17 of 99 claims as unsupported, and the rewrite diligently deleted them.

I audited every flagged claim against the raw corpus. Essentially all of them were in the reports, several verbatim:

• Draft: "PJM coal output rose almost 18% y-o-y." Corpus: "In the PJM transmission region, coal output rose by almost 18% y-o-y in September."
• Draft: "forecasts revised down ~50% on policy changes." Corpus: "The US forecast is revised down by almost 50% across all technologies except geothermal." (The verifier compared against the global revision figure — 5% — and called it unsupported. A scope error.)
• Draft: "AEP and LG&E signed multi-year coal procurement." Both in the corpus — "LG&E" is the model correctly abbreviating "Louisville Gas and Electric Company."

The mechanism is structural, not a bad judge. CoV verifies each claim with independent targeted retrieval; a fresh query built from the claim and its entities. That retrieval often lands on different pages of the same reports than the chunks the draft actually used. The claim's true source chunk never reaches the verifier, so a strict entailment judge correctly reports "not in the evidence I was shown," and the pipeline records a hallucination that never happened.

The 70B-era runs never exposed this, because the 70B drafts made fewer, vaguer claims hugging the top-retrieved text; claims independent retrieval could easily re-find. The frontier model's deep-mined specifics have a much higher re-retrieval miss rate. The verifier's precision, not the generator's accuracy, became the binding constraint.

And the delete-only rewrite makes false positives expensive: every one removes a true, cited claim from the final briefing. On this run, CoV made the frontier drafts strictly worse.

One flag, though, was a genuine catch, and it points at the second design flaw. On the Q5 draft, the model honestly wrote that the provided excerpts contained no information on West Coast refinery rationalization (true for its eight chunks). Verification's targeted retrieval found that content elsewhere in the corpus. The original CoV can only delete claims, so the final briefing simply dropped the honest caveat. This is the worst possible response to discovering a coverage gap.

Part 4 — CoV v2: verify-and-repair

Two findings, two updates:

1. Union evidence. The new CoV verifies each claim against the draft's own retrieved contexts with fresh targeted retrieval. The verifier can no longer miss the chunk the draft actually used, while keeping the targeted function that catches real gaps.

2. Repair, not delete. The verdict set gains a fourth category: contradicted (the evidence actively disagrees, including "the sources do cover the topic you said they didn't"), distinct from unsupported (genuinely absent). And every chunk targeted retrieval surfaces beyond the draft's context goes into a supplementary evidence pool handed to the rewriter, which now:

• keeps supported claims with their citations,
• marks inferred claims transparently,
• corrects contradicted claims against the evidence,
• incorporates supplementary evidence that fills gaps the draft flagged,
• deletes only what is genuinely nowhere in the corpus.

Results, on the identical drafts and claims (only the verification machinery changed):

• All 17 false flags flipped to supported 100% agreement with the manual corpus audit. Final verdict distribution: 97 supported, 2 contradicted, 0 unsupported.
• The 2 contradicted verdicts were exactly the "no refinery info" claims, and the repaired briefing now contains the actual West Coast refinery findings, cited, with the causal bridge the corpus doesn't explicitly state marked as an inference ("the report does not state this directly, but it implies…").

The scoring told an instructive story:

By ratio, v2 doesn't look like much of a win. But by absolute verified content, it's the clear winner: the repaired briefings carry the most supported causal claims of any arm, in addition to coverage the draft's retrieval missed and a per-claim audit trail. The ratio dips because repair adds explicitly-hedged bridging inferences, which score half credit. The metric treats transparency as a penalty; a reader should treat it as the product.

Overview

Putting the whole arc in one place:

1. Frontier models fix causal reasoning, not grounding. The collapse cases from June disappear with no retrieval at all, but ungrounded frontier answers are harder to audit, not easier, because they assert more.
2. With retrieval, frontier generation is near-ceiling (0.970 vs 0.918), with zero detected hallucinations across 99 audited claims.
3. Verification built for weak generators fails against strong ones in the false-positive direction. Independent re-retrieval misses the draft's own evidence, and every miss deletes true content. The assumption baked into CoV, i.e., that flagged claims are probably fabricated, inverts.
4. Fixing this is easy: we can verify claims against the union of the draft's evidence and targeted retrieval, split the "contradicted" ones from those that are "absent," and repair instead of delete. Verification then evolves from being a filter and becomes what it should have been all along: a certification-and-augmentation layer.

The June post ended with "semantic similarity retrieves facts; relational structure retrieves reasoning." This round adds a corollary:

As the generator gets stronger, verification stops guarding against what the model invents and starts having to keep up with what it finds.

Caveats, as always: n=5 questions, single trial per arm, one corpus, and the causal-correctness deltas between the three Fable 5 arms (0.951–0.970) sit within the trial-to-trial spread I measured on the baselines. The verdict-precision results (17/17 false flags corrected, 2/2 genuine contradictions caught) are the robust finding.