Benchmarks 2026 H1: frontier-AI stability on crypto tax classification

All results as of August 2026 · model snapshots: claude-sonnet-5 · gpt-5.5 · grok-4.5

This is the benchmark hub for the two 2026 H1 studies documented on the methodology page: what was measured, the exact transaction sets (as public hash lists anyone can verify on-chain), how disagreements were adjudicated, and the aggregate results with confidence intervals. The methodology page carries the full protocol, disclosures, and limitations; this page carries the datasets and the numbers in one citable place.

1. The studies

Study A: the 388-row bake-off (silent-wrong and stability)

Three flagship models classified transactions from a frozen 777-row corpus against a 388-row adjudicated answer key; the CryptoTaxEdge engine was scored against the same key as a reference. Both studies use one frozen protocol: a single fixed prompt (verified by SHA-256 hash at every run), single-shot, no tools, no retrieval, a closed 17-category taxonomy, abstention explicitly permitted. Sampling rules, metrics, and the analysis plan were locked before any model was called. A companion raw-hash arm ran the same 388 transactions giving each model only the hash and chain, exactly what a real user enters in a chat window.

Study B: the 96-transaction complexity ladder (stability)

The same three models classified 96 fresh 2026 transactions spanning 12 protocol rungs, pooled into three pre-registered complexity bands. Every transaction ran at least twice per model on byte-identical input; hard rungs ran three times: 744 model calls, 100% strict-JSON parse rate, zero transport errors. Study B measures internal consistency (run-to-run flips, inter-model disagreement, abstention, divergence from the engine); it has no ground-truth labels by design and none of its metrics are error rates.

Bands were fixed a priori: simple = rungs {1, 8}, mid = {2, 3, 4, 10}, exotic = {5, 6, 7, 9, 11, 12}. Per-rung numbers (n≈8) are context, never headlines.

2. Datasets, labels, and hash lists

Transaction hashes are public on-chain identifiers: anyone can fetch the underlying transactions, receipts, and logs from public nodes and check every aggregate on this page. Every list is frozen; no transaction was added or removed after any model output was seen. Everything on this page is published under CC BY 4.0.

bakeoff-labels.json · 388 rows, labelled bakeoff-labels.csv · the same, flat bakeoff-hashes.json · 388 txs ladder-hashes.json · 96 txs

• Study A labelled corpus (388 rows, five fields each): hash, chain, stratum (the sampling stratum the row was drawn from, so you can recompute any alternative split rather than take ours), category (the adjudicated ground truth, from the closed 17-category set fixed before any model was called; 13 of the 17 occur here), and treatment (the default US tax treatment that category maps to, derived from the category and carrying no information beyond it). By treatment: 213 disposal, 150 non_taxable, 22 expense, 3 income. Section 5 explains what is open, what is not, and why.
• Study A hash list (388 rows: 262 Ethereum, 77 Polygon, 49 Arbitrum): the same transactions without labels, as first published. The eligible answer-key rows from the frozen 777-row corpus. The eligible key skews answerable by construction; 187 hard rows where the engine abstained have no truth key and are unscored for every system. The remaining exclusions: 12 duplicate hashes, 14 rows outside the closed taxonomy, and 176 disagreement rows not yet adjudicated at freeze time.
• Study B list (96 rows across Ethereum, Arbitrum, and Solana, with rung and band per row): a convenience sample of recent live activity, stated plainly: for each rung contract, the most recent qualifying 2026 transactions at curation time, successful only, verified target contract, diversity caps, global dedupe against previously used hashes. The full selection rule is frozen inside the source set file.
• Prompt integrity: both studies used one fixed prompt, SHA-256 430794e24a49107cfdbf5c559d17de3d3074c2a2f92c496221f1a605d55050d1, verified at every run.

3. Adjudication rubric summary

The answer key behind Study A was built under a protocol pre-registered on 2026-07-22, before execution. The rubric, summarized:

• Locked in advance: sampling frames, metrics, category-equivalence maps, and adjudication rules were fixed and committed before any model was called. The equivalence map was fixed in the scorer before adjudication, so vocabulary synonyms cannot move scores after the fact.
• Blind adjudication from raw evidence: disagreement rows were adjudicated from raw on-chain receipts and logs by two independent model lenses (a net-token-flow lens and a protocol-event lens), with system outputs relabeled before review so neither lens knew which system claimed what. All evidence was recorded and frozen.
• Abstention honored: rows where the engine abstained have no agreed key and are unscored for every system, including ours. A silent-wrong is a confident, unflagged, wrong answer; a flagged row can never be silent-wrong.
• Zero post-hoc exclusions: no row was added or removed after any model output was seen. The set re-curations that happened before execution are listed in the changelog.
• Disclosed limits: the key is model-adjudicated, not independent-CPA-labeled; no human tax professional has adjudicated it, and we say so rather than imply otherwise.

4. Aggregate results with confidence intervals

Every number below traces to the methodology page and its frozen artifacts. Confidence intervals are bootstrap 95% CIs (2,000 resamples over transactions, seeded).

Study A: silent-wrong rates (n = 388 adjudicated rows)

Retired 2026-08-12: the accuracy head-to-head. This section previously led with a single-run category-accuracy table ranking the three models and the engine against this key. It is retired, in both directions: our published run-to-run measurements show a model's verdict on byte-identical input flips between 2.1% and 17.4% of cases by band, the deltas at the top of that table sat inside that noise, and a single run cannot rank systems inside its own noise band. The table is preserved unchanged, numbers intact, in the archive. What this corpus supports, and what stays published: silent-wrong rates, repeat-arm flip rates, inter-model disagreement, and raw-hash refusal behavior, each with deltas that exceed the measured noise band. Comparative rankings appear on this site only from pre-registered, multi-run designs with deltas outside measured run-to-run noise.

A silent-wrong is a confident, unflagged, wrong answer against the adjudicated key; a flagged row can never be silent-wrong. Silent-wrong rates span 5.4–14.2% of rows, a spread that exceeds the measured run-to-run noise band. In the repeat arm, per-model run-to-run flip rates spanned 5.0–11.5%: the same question, asked twice, gets a different answer on up to one row in nine. The engine's review-routing exists to push exactly the silent-wrong class of rows to needs-review instead of asserting them.

Study B: run-to-run stability by band (same model, byte-identical input)

The simple and exotic CIs do not overlap. Per model, pooled across all 96 transactions: claude-sonnet-5 flipped 14.6% (14/96), gpt-5.5 10.4% (10/96), grok-4.5 10.4% (10/96); no model is flip-free.

Study B: agreement, uncertainty, and escalation by band

A “state” is one (transaction, run) with all three models parsed. Raw and synonym-normalized disagreement were identical on every rung. The headline asymmetry: flip rate rises roughly 8× from simple to exotic while mean stated confidence slips 9 points (91.8 → 83.4) and abstention stays under 6%; the engine's needs-review rate climbs 19% → 28% → 44% across the same bands.

Study B: high-confidence disagreement

Raw-hash arm: what a chat window actually gets

Given only the hash and chain (no tools, no retrieval, abstention permitted), all three models declined to classify every transaction: 1,164 calls, 1,164 schema-valid abstentions. With a full evidence packet, the same models classify the corpus; with only the hash, they answer nothing. The models behaved honestly here; the result is evidence that the enrichment layer is what makes classification possible, not that chatbots hallucinate crypto tax answers.

5. Gold labels: what is open, what is not, and why

The Study A per-row labels are open. All 388 of them, no email gate, no form, CC BY 4.0. Take them, score your own system, disagree with us in public.

They were gated until 2026-08-12, and the stated reason was that a public key lets any system be tuned to it. That reason was sound and it no longer applies to this corpus, because the tuning already happened and it was ours. Six classification rules added to our engine on 2026-08-13 were discovered by mining this key's misses. Our engine is now fitted to this corpus. A fitted score cannot honestly be published as a head-to-head, which means this key has stopped being an accuracy instrument for us, and holding it back would protect nothing except the appearance of rigor.

The other half of the reason to open it is about who gets cited. A gated dataset earns close to no citations and reads as lead generation dressed as open science. Researchers who cannot download a thing do not build on it, and asking a stranger to email for permission is a filter that selects for sales prospects rather than scrutiny. If the point of publishing measurements is to be checked, the labels have to be reachable without asking us first.

What a published ranking requires here: comparative accuracy rankings ship on this site only from pre-registered, multi-run designs, on a corpus none of the compared systems was tuned against, with deltas outside measured run-to-run noise. And a corpus serving as a held-out instrument stays unpublished for as long as it is in use, because publishing a live instrument destroys it and lets it leak into training data. Fully open on a corpus that is spent; fully closed on a live instrument. Half-open would be the worst of the three positions.

What the open labels are still good for: taxonomy work, error analysis, and evaluating any system that had no part in building them. What they are not good for is any current claim about our own accuracy, and we would rather say that here than let someone discover it later.

Read before you score against them. The key is model-adjudicated, not independent-CPA-labelled: no human tax professional adjudicated it. The eligible key skews answerable by construction, so a score computed here is a score on the answerable subset, not on crypto tax classification in general. Per-row label provenance is not distributed: roughly half the key was labelled by blind two-lens adjudication and the rest by agreement between two independent classification systems, and publishing which row came from which path would disclose a third party's per-row agreement by subtraction. Corpus-level provenance is in section 3 and in the file's own metadata. And a category with its default treatment is a starting point for professional review, not a tax position.

6. How to cite

DOI · 10.5281/zenodo.21911546 Hugging Face · dataset mirror

Canonical URL: https://cryptotaxedge.com/research/benchmarks/2026-h1/
DOI: 10.5281/zenodo.21911546 (the corpus and the Standard, archived on Zenodo; concept DOI for all versions: 10.5281/zenodo.21911545)
License: Creative Commons Attribution 4.0 International (CC BY 4.0). Redistribute, adapt, and build on this with attribution, including commercially.
Attribute to: CryptoTaxEdge
Version: 1.2 (2026-08-12)

Citation blurb:

CryptoTaxEdge (2026). Frontier-AI stability on crypto tax classification:
2026 H1 benchmarks (388-row labelled bake-off corpus and 96-transaction
complexity ladder). Version 1.2, CC BY 4.0.
https://cryptotaxedge.com/research/benchmarks/2026-h1/
DOI: https://doi.org/10.5281/zenodo.21911546
Methodology: https://cryptotaxedge.com/methodology/
Classification schema: https://cryptotaxedge.com/standard/

@misc{cryptotaxedge2026benchmarks,
  author       = {CryptoTaxEdge},
  title        = {Frontier-AI stability on crypto tax classification: 2026 H1 benchmarks},
  year         = {2026},
  month        = {August},
  version      = {1.2},
  license      = {CC BY 4.0},
  doi          = {10.5281/zenodo.21911546},
  howpublished = {Zenodo, https://doi.org/10.5281/zenodo.21911546},
  url          = {https://cryptotaxedge.com/research/benchmarks/2026-h1/},
  note         = {388-row labelled corpus and 96-transaction complexity ladder;
                  model snapshots claude-sonnet-5, gpt-5.5, grok-4.5;
                  results as of August 2026}
}

If you cite the labelled corpus specifically, cite the file: https://cryptotaxedge.com/research/benchmarks/2026-h1/bakeoff-labels.json. It carries its own version, license, and limitations in its _meta block, so a copy that travels without this page still travels with its caveats.

The classification schema the category and treatment fields draw on is versioned separately and published under the same license: the CryptoTaxEdge Classification Standard.

7. Changelog

• 2026-08-12 · v1.2. The Study A single-run category-accuracy head-to-head table was retired from this page, in both directions, because measured run-to-run flip rates (2.1–17.4% by band on byte-identical input) are wider than the deltas at the top of that table; it is preserved unchanged in the archive. Silent-wrong, stability, disagreement, and refusal results are unchanged. No dataset file changed; no row was added or removed.
• 2026-08-12 · v1.1. The Study A per-row labels were ungated and published under CC BY 4.0 as bakeoff-labels.json and bakeoff-labels.csv, replacing the email-request gate. Section 5 was rewritten to state what is open, what stays closed, and why. No number on this page changed; no row was added or removed.
• 2026-08-12 · v1.0. Initial publication of this hub: study descriptions, both hash lists, adjudication rubric summary, aggregate tables with CIs.
• 2026-08-06 · pre-run set re-curations (Study B, all before any model call, all disclosed): (1) a rate-limit response initially read as an empty transaction list was fixed with a retry; (2) a stale GMX v2 router dominated by reverted bot traffic was replaced with the active router from the protocol's official deployments repository; (3) plain approvals and order cancellations, outside the closed taxonomy, were globally excluded. One mid-run harness fix on the engine-reference caller is disclosed on the methodology page; model calls were untouched.
• 2026-08-05 · Study A artifacts frozen (enrichment packets, raw outputs, scorecard). 2026-07-22 · bake-off protocol pre-registered before execution.

This page reports measurements, not tax advice. Classification of any specific transaction depends on facts and elections; verify with a qualified tax professional before filing. All figures as of August 2026.