ARCHITECT RAG VERIFICATION

Retrieval-augmented generation fails on two surfaces, and most evaluations blur them. The first is faithfulness: the model asserts claims the retrieved passages do not support. The second is completeness: the retriever misses the passage that mattered, and the model answers faithfully from what it was given, so the answer is faithful and wrong. This repository keeps the two apart from the first line of its pre-registration to the last row of its results, because a single score over both hides exactly the failure that matters.

The build followed the same discipline as its sibling one card over, the World Cup forecasting engine. I directed; Claude Code executed under a governance file read at the start of every session, with a session log recording the reasoning behind each unit of work. The evaluation was pre-registered before any answer existed: the queries, the ground truth, the metrics and the pass rules committed first, so the commit history itself is the proof that the scoring rules predate the results. A CI workflow re-runs the documented fresh-clone path and asserts the published figures against the run.

Fifty queries, three model tiers, three conditions. Raw is standard RAG. Layer is the same model on the identical first pass, followed by deterministic checks: nothing inside the layer is a model. It resolves the references printed in the query and the retrieved passages, fetches by identifier anything named but absent, flags the claims the context does not support, issues one second call, and abstains if it still cannot ground. A third condition with no retrieved context at all measures how much the models already know these public frameworks, which turns the contamination caveat into a number.

The finding, stated the way the repository states it: the layer's measured effect on this corpus is mostly abstention and denominator change, and almost none of it is unsupported content disappearing. Zero of the 109 flagged claim units were rescued by the fetched context, on any tier. Under the layer, all three tiers abstain on all eight adversarial rows, where abstention is the only correct answer. And of twenty-six pre-registered predictions, fifteen were contradicted; every contradicted line stands as written, because a prediction that gets edited afterward is not a prediction.

The completeness check works because the corpus is bounded: 1,150 units in 1,294 chunks across the EU AI Act and three NIST AI RMF documents, frozen before any query existed, small enough that every printed reference can be resolved per query. The repository says plainly that this does not scale, names the rough point where the approach stops being practical, and claims nothing past it.

The repository is the record. The pre-registration committed before any answer was generated. The results ship every rate beside its unit counts, nine standing failures, five named exclusions, and the fixes that were available and were refused, each with its reason. Generation ran once, for 3.218898 dollars in total; every number then re-derives from committed files with no API key and no cost, and three result artifacts are pinned by digest and asserted by a test suite more than a thousand tests strong. Start with PREREGISTRATION.md, then RESULTS.md, then REPRODUCE.md.