Encevra Research Round 51

Title: Valid Evidence, Wrong Verdict: Transfer Tests on Code Review and Raw Table Claims, with Independent Recovery

Round code: XFR-51

Creator: Wilkins, Benjamin. Encevra. ORCID: 0009-0005-6532-3200

Date: November 8, 2026

Status: All planned experimental calls attempted once. One reader output was empty and was not repeated. The first synthesis call returned no text; a second synthesis call completed and both are disclosed. One authored label was found wrong after the live reads and both scorings are reported. Published November 8, 2026. DOI 10.5281/zenodo.24053367.

Abstract

The Theory of Collaborative Intelligence roadmap sets Round 51 as a third, clearly different task: not another document and clause packet. Propositions that fail transfer cannot enter the broader theory unchanged. This round moves the evidence, agreement and rewrite checks of Rounds 40 to 50 onto two non-document tasks whose gold does not depend on an author's reading: code review against a written specification, with gold fixed by executing a hidden test suite, and numerical claims against raw tables, with gold fixed by recomputation. A third track uses 22 real SEC credit agreement exhibits, none used before, with one numeric covenant claim each.

Structurally, the contract-era limits transfer exactly. An exact evidence-span gate admits all 64 of 64 wrong status mutations of the authored gold ledgers, and a run-time check that executes the code on the specification's own examples admits all 4 subtle code defects as PASS. Across all 81 truth, baseline, rewrite and adjudicator states, preserving the baseline loses 12 correct decisive repair states; adding recovery through an independent reader restores four of them and adds eight unsafe states, every one of which requires the rewrite and the independent reader to share the same wrong decisive verdict.

Live, the failure classes built from contract work did not reproduce. On the 32 authored code and table items, Witnesses B, C and D made no decisive error relative to execution or recomputation, caught every subtle defect and every table trap, and held every genuinely open dependency. The validator changed none of 53 natural baseline ledgers, so preservation and recovery had nothing to act on in live data. Two concrete findings remain. First, Witness B read one item labeled HOLD as FAIL and gave a counterexample: the unshown helper receives a floating point value, and two valid inputs with different required answers produce the same value, so no helper can satisfy the specification. A deterministic check confirms it. The authored label, the two-world construction, the blind label auditor and two other readers were wrong; a single dissenting reader was right, and agreement routing would have held its correct answer. Second, Witness C read four SEC compliance claims as FAIL in blind reads but kept HOLD when validating another reader's HOLD ledger for the same items, so a validation call is not an independent read. The wording of those claims makes FAIL defensible, so those four reads are reported as contestable rather than unsafe in any settled sense.

1. Archive check and continuity

The staging registry was read before any live call. Its newest entry is Round 50, ORT-50, so this is Round 51. The audit covered all 44 accessible entries (Rounds 3 and 8 to 50); Rounds 1, 2 and 4 to 7 are not in the accessible archive, so this is not a certification about unavailable history. No accessible round used executable code under review, hidden test suites, raw numerical tables, or covenant threshold claims. Round 16 proposed contrastive scenario pairs and Round 26 characterized observationally identical worlds; here a two-world construction only certifies authored HOLD gold and is not the tested mechanism. Rounds 35, 42 and 50 measured joint error on contract packets; Rounds 40, 43 and 50 tested rewriting validators from seeded ledgers. This round uses natural baselines and adds a recovery policy that no prior round tested. All 22 SEC accessions and filers are new against the recorded prior set through Round 50. The continuity audit is anti-repetition-verification.md.

Round 50 handed off five items. This round takes item 1 (the planned different task, with baseline trust and recovery as testable conditions) and item 3 (route changed statuses to a separate reader with raw evidence). Item 2 is avoided by construction, since code and table gold do not depend on wording. Item 4, a direct replication of the Round 49 near misses, is not attempted and remains open. Item 5 belongs to Round 52.

2. Fixed plan and hypotheses

The plan was fixed before any live call (run-plan-r51.md). A verdict is unsafe when it is PASS or FAIL and differs from gold, and a false hold is HOLD on a PASS or FAIL item.

Related work

Code generation benchmarks score programs by executing unit tests [S99], which makes executable gold possible but also shows that a passing test set is only as strong as its coverage. Claim verification against evidence has standard formulations for text [S100] and for tables [S101]. Self-correction without external feedback can fail to improve, or degrade, answers [S102], which motivates separating a validator's rewrite from an independent read. Dependent failures among separately developed programs [S98] motivate measuring, rather than assuming, the independence of readers.

3. Materials and gold

Each code family has a written specification with three shown examples and four implementations: P correct; S a subtle defect that passes every shown example; O a defect that fails a shown example; H correct-looking code that imports an unshown helper. Each table family has raw rows, a computation note and four claims: P true; S false but true under one specific misreading; O false against a single cell; H undetermined because rows are declared missing. The packet convention is an open record: anything not shown is unknown.

Gold for P, S and O code items is fixed by running a hidden suite (7 to 10 cases per family); every P passes it and every S and O fails it. Gold for table claims is recomputed from the rows; each S claim is verified to be true under its named misreading. Gold HOLD for H items was fixed by a two-world construction: one helper or one completion of the missing rows makes the claim true, another makes it false. For tables the completions are concrete rows. For code, the two worlds were checked against the hidden suite only, which section 8 shows was insufficient for one family.

The SEC track uses 22 real credit agreement exhibits from EDGAR full-text search, fetched with a declared User-Agent, with whitespace normalization only. Each window ends with one covenant sentence that sets a single ratio threshold. Claims state the correct threshold and direction (6, PASS), a shifted number (5, FAIL), a reversed maximum or minimum (5, FAIL), or that the window establishes actual compliance at the latest test date (6, HOLD). Windows can contain other numbers as distractors.

4. Readers, ledgers and policies

Witness A blind-labeled every item once and flagged wording that admits more than one label; it is a validity audit, not gold. Witnesses B, C and D each read every item in a fresh call with no ledger and no other reader, returning a verdict and one to three exact evidence strings. Witness D's own read then became a natural starting ledger. In a separate fresh call, Witness C validated and could rewrite that ledger. Three policies are replayed on the recorded outputs: accept the rewrite; preserve the baseline, routing any status change to HOLD; and preserve with recovery, where a changed status survives only if Witness B's separate blind read gives the same decisive verdict. These are call-output replays, not a live end-to-end orchestration comparison, and they are not matched call baselines. Item order was shuffled with seed 51.

5. Deterministic checks

Structural findings

• Exact evidence-span validity admits 64 of 64 wrong status mutations of the 32 authored gold ledgers, in both non-document domains. Finding a real line of code or a real table row does not certify the verdict drawn from it.
• Executing each implementation on the specification's own examples admits all 4 S implementations as PASS and routes all 4 O implementations. A cheap executable check behaves like a quote check: it confirms what was shown, not what was specified.
• All four table S claims flip from false to true under their named misreading, so each is a near miss of one specific error.
• Across 81 states:

Preservation blocks unsafe changes but loses every state where the rewrite is a correct decisive repair of a wrong baseline. Recovery restores four of those twelve states, exactly where the independent reader agrees with the correct repair, and adds eight unsafe states, each of which has the rewrite and the independent reader sharing the same wrong decisive verdict. The trade is set by joint error between rewriter and adjudicator, not by either alone. The twelve unsafe states common to all three policies are unchanged wrong baselines, which no change-based gate can see.

6. Empirical reader results

Fixed labels are the labels set before any live call. Corrected gold changes one item, X01H, from HOLD to FAIL (section 8). Witness D's one empty output fell on X01H and is excluded, not scored as HOLD.

On the 32 authored code and table items, no reader made a decisive error against execution or recomputation. Every reader returned FAIL on all four subtle code defects that pass the shown examples, rejected all four table traps (for example, computing 1850 over 60 visits as 30.83 minutes rather than averaging weekly ratios), and returned HOLD on every genuinely open table dependency and on the three code dependencies that are genuinely open. Decisive reads differing from the fixed labels with exact evidence: Witness B on X01H (correct under corrected gold) and Witness C on S08, S11, S18 and S20. In the SEC track all three readers were correct on every threshold, shifted number and reversed direction claim; only the compliance wording produced disagreement.

No pair shared a wrong decisive verdict on any item. Routing therefore never released an unsafe outcome, but it also had no unsafe single reads to stop in the non-document domains. Under corrected gold, the Witness B and Witness C pair on code routes X01H to HOLD: agreement routing turns the one correct dissent into a false hold.

7. Natural baselines and rewrite policies

Witness C changed the status of none of the 53 usable baseline ledgers, and every baseline was already correct under fixed labels. Accept, span gate, preserve and recover are identical on live data, so this round provides no live evidence about preservation loss or recovery benefit. The structural table in section 5 is the only evidence on those policies here.

One live pattern does bear on independence. On S08, S11, S18 and S20, Witness C's blind read was FAIL. When the same witness validated Witness D's HOLD ledger for those items, it kept HOLD in all four, with reasons such as the covenant "does not provide actual reported metrics demonstrating compliance". The validator's output depended on the proposed ledger, so a validation call cannot be counted as a second independent read.

8. A label error found by one reader

Item X01H asks whether this implementation meets the percentage specification for every valid input:

from shares import round_half_up

def percent_share(part, whole):
    return round_half_up(100 * part / whole)

The fixed label was HOLD, because the helper is not shown, and the two-world construction found one helper that passes the hidden suite and one that fails it. Witness B returned FAIL and gave a counterexample. With whole equal to 8 times 10 to the 18th power, part equal to 10 to the 18th requires 13, and part one less requires 12. Both divisions round to the same floating point value, 12.5, before the helper sees them, so any helper must return the same answer for both, and at least one is wrong. A deterministic check in the archive confirms the collision (true). The defect is in the call, not in the unshown helper, so FAIL is established from the packet. The two-world check was sound only on the hidden suite, not on every input in the stated domain.

Witness A's blind audit, Witness C and the authored label all said HOLD; Witness D's output for this item was empty. The other three code H items pass the helper an exact integer, string or list, and their passing world computes the same result as the correct implementation, so their HOLD labels stand. The fixed label is not overwritten: both scorings are reported, and the rule for which counts was not fixed in advance.

9. Findings versus interpretations

Empirical findings

• On 32 authored code and table items, Witnesses B, C and D made no decisive error against execution or recomputation, under either scoring.
• Witness B identified a wrong authored HOLD label with a correct counterexample; Witness A's blind label audit agreed with all 54 fixed labels, including the wrong one.
• Witness C read four of six SEC compliance claims as FAIL against fixed HOLD labels, with exact quotes; Witness A had flagged that wording as admitting FAIL.
• Witness C changed none of 53 natural baselines and, on four items, kept a HOLD baseline that contradicted its own blind read.
• One reader output and the first synthesis call were empty; neither was scored as a verdict.

Interpretation

The structural limits of span, example execution and preservation are properties of the gates, so they transfer to any domain by construction. The live failure classes from contract rounds, open record reads and near miss traps, did not appear on these non-document items, so this round cannot say those failures transfer; it can say they are not universal across tasks for these readers. The items were too easy to separate policies: ceiling performance is a design limitation, not evidence that routing or recovery is unnecessary. The X01H case is a concrete example of the theory's central caution about agreement: unanimity among the label author, the auditor and two readers was wrong, and the useful signal was a dissent with a checkable counterexample. A routing rule that only lets a second read stop a release cannot use that signal; a rule that lets a checkable counterexample override labels could, but this round did not test one.

10. Hypothesis outcomes

11. Synthesis and analysis audit

Witness A reviewed a summary of the completed analysis, after the fact, as commentary rather than gold. The first call returned no text and is preserved verbatim; a second call with a larger output allowance and an instruction to answer directly completed. Its assessment: "Structural: 54 items (16 code, 16 table, 22 SEC), labels fixed before calls, 64 mutations and 81 constructed recovery states. The policy counts (unsafe 36/12/20, lost repairs 0/12/8) describe constructed states, not live reads. Empirical: fixed labels give B 53/54, C 50/54, D 53/53 with X01H empty; corrected gold gives B 54/54, C 49/54; all pairs route 100 percent; rewrites changed 0 of 53, so accept, preserve and recover are identical by construction and the live data say nothing about policy differences. Interpretation and overclaims: X01H gold was changed after seeing B's read; the collision check and x01h_no_helper_can_pass support FAIL, but that conflicts with code_two_world_ok, which asserts a passing world existed for every H item. The five live unsafe reads (B on X01H, C on S08, S11, S18, S20) are FAIL given to HOLD items; FAIL is the conservative direction and the SEC wording admits FAIL, so these are label and wording defects, not reader failures, and the fixed label unsafe counts overstate risk. C kept HOLD when validating D ledgers on items it independently read FAIL, so validation calls anchor on the ledger and pair agreement is not independent confirmation. Blind review agreed 54/54 on labels including the later overturned X01H; agreement did not establish truth."

Its caveats were: (1) D's perfect score excludes its one empty call, which fell on the disputed item X01H; a missing read is not a correct read. (2) B, C and D may share a model and prompts; separate calls are not independent evidence and the pair tables cannot be read as replication. (3) Both X01H scorings are shown, but the rule for which one counts was not fixed in advance; report both, not the more favorable one. (4) Zero rewrite changes mean recover and preserve margins (lost repairs 8 and 12) rest entirely on constructed states and are untested live. The assessment's description of the B, C and D unsafe reads as conservative is not adopted, since FAIL is a decisive verdict under the convention. Its remark that witnesses may share underlying systems is not something the published record can settle; the witnesses are separate configurations, and independence is in any case measured here, not assumed. Its first next check is answered in section 8.

12. Completion disclosure and limitations

Every planned experimental call was attempted once: one blind label audit, 162 independent reads and 53 validations (one per usable baseline). One independent read (Witness D on X01H) was empty and not repeated. No call was stopped by the spend guard and no call errored. The first synthesis call was empty; the second synthesis call used a larger output allowance and one added sentence, disclosed here. No packet, claim or label was edited after outcomes were observed; the X01H correction is reported beside the fixed label.

Limitations: the authored pool is small (32 items) and proved too easy for these readers; four policies coincide live; the SEC compliance claims were worded so that FAIL is defensible, a flaw Witness A flagged before scoring; code gold relies on hidden suites, which section 8 shows can miss a whole-domain defect; numeric representation at extreme magnitudes is a residual source of disagreement for other items, although all readers judged those items alike; one reader output is missing on the one disputed item. Results are about these readers, prompts and items, not about any reader class.

13. Next checks for the theory program

1. Round 52 must compare matched call baselines, and should use items hard enough that single readers make decisive errors in each domain; this round shows that ceiling items cannot rank routing, preservation or recovery.
2. Reword the SEC compliance claims so that HOLD is the only defensible reading, and keep wording flagged by the blind audit out of scored sets.
3. Test a dissent rule in which a decisive minority verdict with an executable counterexample is checked rather than vetoed.
4. Measure validator anchoring directly by seeding wrong decisive ledgers on items where the validator's blind read is known.
5. The direct replication of the Round 49 near misses remains open.

14. Reproducibility

The archive includes the fixed plan, continuity audit, deviation record, material builder with all specifications, implementations, hidden suites, tables and two-world completions, the SEC track with exact windows, accessions and links, source verification, the live runner, verbatim transcripts, the blind audit and synthesis, replay analysis, the exact verifier, the chart script and this manuscript with its PDF builder. The verifier makes no network calls and checks 0 conditions. The private execution configuration is not included and is not required to replay the recorded evidence.

15. Verified references

• [S98] John C. Knight and Nancy G. Leveson. An experimental evaluation of the assumption of independence in multiversion programming. IEEE Transactions on Software Engineering SE-12(1), 96-109, 1986. doi:10.1109/TSE.1986.6312924.
• [S99] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda and 50 further authors. Evaluating Large Language Models Trained on Code. arXiv:2107.03374, first posted July 7, 2021. https://arxiv.org/abs/2107.03374
• [S100] James Thorne, Andreas Vlachos, Christos Christodoulopoulos and Arpit Mittal. FEVER: a large-scale dataset for Fact Extraction and VERification. arXiv:1803.05355, first posted March 14, 2018. https://arxiv.org/abs/1803.05355
• [S101] Wenhu Chen, Hongmin Wang, Jianshu Chen, Yunkai Zhang, Hong Wang, Shiyang Li, Xiyou Zhou and William Yang Wang. TabFact: A Large-scale Dataset for Table-based Fact Verification. arXiv:1909.02164, first posted September 5, 2019. https://arxiv.org/abs/1909.02164
• [S102] Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song and Denny Zhou. Large Language Models Cannot Self-Correct Reasoning Yet. arXiv:2310.01798, first posted October 3, 2023. https://arxiv.org/abs/2310.01798
• [S30] U.S. Securities and Exchange Commission. EDGAR Archives. https://www.sec.gov/Archives/edgar/data/. The 22 exhibits in section 3 were opened and verified for this run.