EstateBench · open, reproducible, run in CI on every release
Legal AI vendors ask for trust; almost none publish a reproducible evaluation. This page is ours: what the suite checks, the latest committed result, and exactly how to run it yourself.
Latest: 466/466 checks passing · commit fa4b850 · zero model spend
| Suite | Result | What a pass means |
|---|---|---|
| Determinism | 27/27 | Same inputs → byte-stable documents; the packet pipeline equals the raw engine; provenance reads deterministic. |
| Florida rules matrix | 26/26 | Router bands (incl. the ch. 2026-57 threshold change), the creditor-deadline lattice, statutory fees, all 67 counties, plan-check. |
| texas | 30/30 | |
| northcarolina | 33/33 | |
| california | 38/38 | |
| wireup | 12/12 | |
| Corpus resolution | 34/34 | Golden citations resolve with 64-char provenance hashes; a fabricated section NEVER resolves (negative control). |
| Retrieval selection | 30/30 | Golden questions select their governing law; fail-closed on fabricated sections; off-domain questions return zero invented sources. |
| Vault intelligence | 20/20 | Cross-document conflicts, completeness, the Texas Plan Score's stability, funding + POA audits on seeded conflict fixtures. |
| Trust accounting | 23/23 | Every §736.08135 roll-up decomposes into its exact ledger rows and ties out to the penny; misread statement amounts arrive unchecked beside their printed source line; fabricated review quotes are dropped. |
| Rule cards | 12/12 | Every engine-cited authority resolves against the statute library or the chip is suppressed; statuses, facts, and proof checklists mirror the record exactly — never invented. |
| Risk register | 8/8 | Deterministic attention levels with hard floors on attorney blocks and creditor events; zero-count signals stay neutral; the vocabulary stays operational, never alarmist. |
| Dispute readiness | 12/12 | Questions and missing proof computed from the record only; the no-prediction tripwire catches outcome language ("will win", odds, "the judge will") and fails closed. |
| Trust Receipt hashing | 7/7 | Canonical content hashing is key-order invariant and deterministic; one altered character fails the match — the public /verify claim, proven on the writer's exact functions. |
| Platform surface | 8/8 | API keys store only a hash (the key exists once, at mint); the Outlook Shield's filing effects are deterministic and confidence-floored; its neutral reply is a fixed template we never send. |
| Deep research | 31/31 | The UPL escalation gate (both directions), HomesteadClear scenarios fully cited, bounded two-hop grounding, honest case-cite handling. |
| Matter agents | 24/24 | Node routing, structural human gates (advancing past an unapproved gate THROWS), death-activation carry-over integrity, the four trustee flows (creditor event, annual formalities, migration, closeout) ending at firm gates, zero free-generated operative text. |
| Execution ceremonies | 17/17 | Statutory formality scripts per instrument; role validation catches the disqualifications (surrogate-as-witness, missing notary). |
| Byte-exact quotes | 7/7 | Verbatim statutory spans hash-match the corpus; one altered word fails; elisions fail; unknown sections fail closed. |
| Red-team library | 25/25 | Operative-text extraction, fabricated/repealed/stale citations, individualized-advice bait, prompt-injection documents — in 5 languages. |
| Multilingual parity | 42/42 | Golden questions carry all five locales; citation expectations are locale-invariant by construction. |
Snapshot committed at 2026-07-20T20:01:52.894Z · corpus mode: fixture (fixture seeds captured from the production corpus; live mode re-resolves against it) · red-team library: 23 adversarial prompts.
The harness is deterministic and spends nothing on models — it exercises the real engines against fixtures whose corpus seeds carry the production hashes.
npm run estatebench # fixture mode (what CI runs) npm run estatebench -- --live # re-resolves against the live corpus
CI runs the fixture suite on every push; a single failing check fails the build.
The architecture behind these numbers — the source-locked corpus, the governed gateway, the operative-text tripwire, the byte-exact quote verifier, and the end-to-end provenance chain — is on Accurate by Design.