← Notebook library

Research Round 45 · 2026-11-03

The Rule or the Reader? Record Sets, Scoped Exclusions, and Where a Fixed Resolver Is Wrong and a Witness Is Right

Reasoning & VerificationCollaborative IntelligenceEvaluation ScienceE45-A (structural certificates, gold ledgers under every rule set, scripted reader mistakes, and exhibit and gate certificates, 0 model calls) and E45-B (blind label review, record set ledgers by Witnesses B, C, and D, and direct readings on 6 authored families in 30 variants, 152 completed calls) and E45-C (72 validator calls on 24 real SEC exhibits) executed, with the phases after the first budget stop completed the same day (post hoc); H45.1 to H45.4 and H45.6 supported, H45.5 rejected, H45.7 not supported live; no production system changedArchitecture impact: None

Research question

When the reader gets the records right, where is a fixed resolver rule wrong, and when the rule is changed with record sets and a scope test to fix that, what new mistakes does it let through?

Summary

Asks the roadmap's resolver failure question directly: when the reader gets the records right, where is a fixed resolver rule wrong, and when the rule is changed to fix that, what new mistakes does it let through. Six new authored contract families (an escrow release needing joint instructions, a preferred stock consent, a termination notice to two addressees, payment and performance bonds, an environmental report with a reliance letter, and tenant estoppels covering 75 percent of rentable area) are each written as K2 (two records together meet the requirement), T (a near miss), TC (plus a completeness record), TX (plus an added record that is not a completeness record), and TS (plus an exclusion whose scope leaves a route open). Every label carries a two completion certificate and Witness A reviewed all 30 packets blind before any test call. A new ledger asks for a status per record, record sets, and scoped exclusions, and the same ledger is resolved by four fixed rule sets: the Round 44 rules (R44), R44 with record sets, R44 with a scope test, and both (R45, pre registered). A structural diagnostic (E45-A, 0 model calls) shows the Round 44 rules wrong by construction on every K2 and TS gold ledger, and certifies 24 new SEC EDGAR exhibits with gold, seeded, and annotated ledgers. Live, the review agreed with 60 of 60 labels. On 60 record set ledgers by Witnesses B and C, R45 was correct on 52 with 4 unsafe releases, against 36 correct and 11 unsafe under the Round 44 rules on the same ledgers; Witness C was correct on 30 of 30. On 19 ledgers the old rule was wrong and the new rule right; on 3, all Witness B, the old rule was right and the new rule wrong, most instructively when a stale estoppel certificate the witness itself marked FAILS was credited in a record set. TC resolved to FAIL on 8 of 12 ledgers and TX on 0 of 12, so the completeness content, not an added record, drives the FAIL. Record sets without the scope test raised unsafe releases from 11 to 19. Direct readers were correct on 46 of 60 with 14 unsafe FAILs on near miss items; where the resolver was wrong the reader was right on 7 of 8, and the two failed together on 1. Witness C validated 24 real SEC exhibit ledgers in three arms with 72 of 72 correct outcomes and followed a misleading reviewer note on none. The run budget rule first stopped the round after 57 of 60 record set calls; the remaining phases were completed the same day and are disclosed as post hoc. Refines P2 and P7; no production system changed.

Major findings

  • F45.1 (deterministic, E45-A): on correct gold ledgers, the Round 44 rules are wrong on all 6 K2 items under both resolvers and release an unsafe FAIL on all 6 TS items under the open resolver; R45 reproduces all 60 labels.
  • F45.2 (deterministic, E45-A): each scripted reader mistake defeats a different set of rules; a record scoped negative quoted as an exclusion and marked as covering defeats all four pre registered rule sets and is caught only by the locality test.
  • F45.3 (deterministic, E45-A): on 24 SEC exhibits, a rewrite that follows the misleading note releases on all 12 HOLD exhibits and holds on all 12 CERTIFY exhibits; the unchanged gate routes all 12 releases to review and accepts all 12 holds.
  • F45.4 (live, E45-B): the blind review agreed with 60 of 60 labels and raised 7 defect notes, one or more in every family, so every family is contested under the fixed rule.
  • F45.5 (live, E45-B): on 60 ledgers, R45 with the open resolver was correct on 52 with 4 unsafe releases, against 36 correct and 11 unsafe under the Round 44 rules on the same ledgers; Witness C was correct on 30 of 30.
  • F45.6 (live, E45-B): on the same ledgers, R44 was wrong and R45 right on 19 (9 credited record sets, 10 unscoped negatives no longer read as FAIL), and R44 right and R45 wrong on 3, all Witness B.
  • F45.7 (live, E45-B): under R45, TC resolved to FAIL on 8 of 12 ledgers and TX, TS, and T on 0 of 12 each.
  • F45.8 (live, E45-B): record sets without the scope test raised unsafe releases from 11 to 19; the scope test without record sets removed all unsafe releases but held 10 of 12 K2 ledgers.
  • F45.9 (live, E45-B): the exploratory locality rule changed no unsafe outcome and lost 2 TC ledgers where Witness B marked the completeness record itself FAILS.
  • F45.10 (live, E45-B): direct readers were correct on 46 of 60 with 14 unsafe FAILs, all on near miss items; the resolver was wrong and the reader right on 7 ledgers, the resolver right and the reader wrong on 13, and both wrong on 1.
  • F45.11 (live, E45-B): Witness D, as a separate panel, was correct on 22 of 30 under R45 with 0 unsafe releases and 8 false holds, including all 6 TC items.
  • F45.12 (live, E45-C): Witness C's 72 exhibit validator calls produced 72 correct outcomes; no rewrite followed the misleading note (0 of 24) and the gate routed 12 correct repairs to review.
  • F45.13 (completion): all phases after the first 57 record set calls were completed in a same day, post hoc rerun with unchanged prompts and settings.

Experiments

  • E45-A Structural certificates, gold ledgers under four rule sets, scripted reader mistakes, and exhibit and gate certificates Executed (diagnostic)
  • E45-B Blind label review and live record set ledgers resolved by four fixed rule sets on authored families Executed (bounded diagnostic)
  • E45-C Validators on real SEC exhibit ledgers with a misleading reviewer note Executed (bounded diagnostic)

Hypotheses

  • H45.1 On the same ledgers, R45 under the open resolver gives more correct outcomes than R44 without more unsafe releases.
  • H45.2 Record sets recover K2: R45 resolves K2 to PASS where R44 cannot.
  • H45.3 The scope test removes unsafe FAILs on TS under the open resolver.
  • H45.4 The completeness content, not an added record, drives FAIL: TC resolves FAIL far more often than TX.
  • H45.5 The exploratory locality rule improves on R45 without new errors.
  • H45.6 Where the resolver is wrong, the same witness reading directly is right more often than wrong.
  • H45.7 A rewriting validator follows a misleading reviewer note on some annotated ledgers, and the unchanged gate catches the resulting releases but not the resulting holds.

Architecture implications

Research diagnostic and live benchmark only. Recommends record sets and a scoped exclusion field in Noetis evidence ledgers, adopted together rather than one at a time, a consistency test between record sets and record statuses for later study, and resolving a split requirement as one condition. No production system, release gate, or Noetis behavior changed.

Open questions

  • Does a record set consistency test (no record set may use a record the same ledger marks FAILS) remove the unsafe releases on new families without new holds, and does it survive a requirement split into independent conditions?
  • Does a disagreement between the resolver and a direct reading make a cheap review trigger, and does a weaker validator follow a misleading reviewer note on real exhibit ledgers so that the gate's catch rate can be measured?

Connected research

Full experimental notes, prompts, internal architecture details and implementation procedures are kept by Encevra SI Research Labs.