Archive: the retired Study A accuracy head-to-head
Archived 2026-08-12 · preserved exactly as published · not current
The retired table, as published on the benchmark hub
Study A: accuracy and silent-wrong (n = 388 adjudicated rows). Bootstrap 95% CIs and the full protocol are on the methodology page.
| System (Aug 2026) | Accuracy (abstain-excluded) | Overall category accuracy (all 388 rows) | Silent-wrong rate |
|---|---|---|---|
| gpt-5.5 | 94.5% | 93.0% | 5.4% |
| CryptoTaxEdge engine (reference) | 92.3% | routes to review instead | |
| grok-4.5 | 89.6% | 9.5% | |
| claude-sonnet-5 | 79.9% | 14.2% |
The note that accompanied the table, unchanged: “The top of the accuracy table is statistically tied: 93.0% vs 92.3% across the full 388-transaction adjudicated key (overall category accuracy; review-routed rows count against the engine) is a 0.7-point gap, statistically indistinguishable, and we publish the tie rather than bury it. Silent-wrong rates span 5.4–14.2% of rows. In the repeat arm, per-model run-to-run flip rates spanned 5.0–11.5%: the same question, asked twice, gets a different answer on up to one row in nine. The engine's review-routing exists to push exactly the silent-wrong class of rows to needs-review instead of asserting them.”
The retired summary, as published on the methodology page
Accuracy: statistically tied at the top. gpt-5.5 scored 93.0% across the full 388-transaction adjudicated key (94.5% abstain-excluded) vs the CryptoTaxEdge engine at 92.3% on the same full key (overall category accuracy; review-routed rows count against the engine), a 0.7-point gap, statistically indistinguishable. grok-4.5 scored 89.6% overall; claude-sonnet-5 scored 79.9% overall. We publish the tie rather than bury it.
Provenance, unchanged
- Bake-off protocol pre-registered 2026-07-22, before execution; artifacts (enrichment packets, raw outputs, scorecard) frozen 2026-08-05.
- Scored against the 388-row adjudicated answer key; the key and its per-row labels are open under CC BY 4.0 on the benchmark hub, with its limitations stated there, including that the reference engine was later tuned against this key and can no longer be measured by it.
- Full protocol, disclosures, and limitations: the methodology page.
Why this page exists
Deleting a published number silently is the marketing equivalent of a silent wrong answer, which is the failure mode this product exists to prevent. The comparison is retired because a single run cannot rank systems inside the measured noise band; the record of what was published stays reachable here, numbers intact, so anyone who cited it can still verify what it said.
This page reports archived measurements, not tax advice and not current results. Classification of any specific transaction depends on facts and elections; verify with a qualified tax professional before filing.