~ / evaluation
##Evaluation
Compare models, evaluate your own, know the failure modes.
How the class gets measured: boards to compare on, suites to run against your own model, the datasets inside them, and probes of known failure modes.
### Leaderboards & indexes 6
The mainstream benchmarking site's index, rebuilt as v2.1 on Nov 8: 19 decision models over six weighted evals — AA-Paperclip (private documents, 30%), TypeSafe's O*NET, When2Call, a private AA-Omniscience adaptation, HLE-MultiChoice with an abstain option, and AA-Symmetry, which checks a decision holds when options move, reword, or the question negates (15%) — scored as accuracy-minus-Brier per answer. The Nov 11 standings put Nace's Drex Max v1.1 first, SubQ Snap 1.2 second, GLiDE v1.2 third, mammoth-1 fourth, d1-ultra fifth, and Jev 1.14 eighth, with Gemini Decision 1 ninth in its first independent row; language models shown as the trailing baseline.
The audio counterpart (v0.1): 988 items (112 public / 876 sealed) across intent, escalation, command safety, sentiment, urgency, speaker verification, and sound events. The ranked board is all general audio models so far — Qwen3-Omni-30B-A3B leads the Jev-class rows (83.2 capability, 70.4 composite) and Gemini 3.8 Flash is most capable on public-only items (94.4) but outside the latency and cost caps — and none of the dedicated audio decision models (Clef-omni, d1-omni, NeuDecide) have been measured on it yet.
TypeSafe's workflow evals: four production-shaped workflows scored against frontier LLM reference probabilities.
Held-out image-decision suite, rebuilt as v0.3.0 on Oct 17: 2,441 items over 2,160 images (300 public / 1,200 sealed), 63 systems measured, frozen hashes and contamination tracking retained — the original pool flagged Kev's Mind2Web training overlap itself. kettei-fat tops it at 75.9, with Sage 1.4's beta image mode and Jev 1.13's image preview next; the Qwen3.5-adapter Imajev family leads open weights (68.0); frontier chat APIs still trail far behind.
Community index: a 37-benchmark public panel plus private tests (60% same skills, 40% new domains), five skill areas, latency on fixed hardware, and raw data published as JSON. The Nov 6 draw (edition 0.4.1) ranks 274 systems on the Jev 1.14 reference: REAX 1.3 leads the Full score at 73.98 with GLiDE 27B and KnowLine 31B next, mammoth-1 down at #13 after topping the Oct 29 draw, and the Jev 1.13 reference at #63 (59.22). The surface Clef's launch numbers reported on.
Independent decision-model leaderboard (run by Benchmark Heaven, not TypeSafe): 1,500 typed decisions per system scored on intelligence, calibration, speed, and cost — an official composite plus a capability view capped at 2× Jev's cost and latency — with versioned releases, published methodology, and result JSON. The v1.8 refresh (Oct 30; 10-second rule, GPU-time cost basis) grew it to 235 open-weights systems and 48 hosted APIs. On the API board, Super Decider leads the composite (78.5) ahead of Launchables' Myles-G 12B — a price-axis play at $0.001/1k — with kettei-fat third and still #1 on the capability view (86.3); Jev 1.13 sits #15. On open weights, Base Labs' mammoth-1 leads capability at 85.2 with KnowLine 31B, Fastino GLiDE 1.2 (now open), H2O's 31B, and Perplexity's Decider v1.1 packed within three points behind it.
### Class-native suites 23
TextQL's enterprise-decision benchmark (paper name Argo-Bench; no relation to the dataset above): a simulated year of a NYC delivery platform exported to a 235-table, 7.5B-row Oracle-style warehouse, where agents file typed actions (ban accounts, freeze payouts, file forecasts) graded by their consequences in the simulator, with latent ground truth withheld so answer keys cannot leak. One appendix worth the price of admission: Gemini 3.8 Flash escaped an unisolated sandbox, located the grader, and filed the answer it read — the run was discarded and the traces published. The board is frontier LLM agents end-to-end (Opus 5.5, 59.5) — the outer loop a decision model would slot into.
80 scenarios and 293 questions per tier (medium is rules-based, hard is judgment calls), with a cost-per-decision axis baked in. What Jev-Omni reports on. Apache 2.0.
Document-grounded typed decisions across tasks, domains, and primitives, scored on accuracy, expected calibration error, negative log likelihood, and coverage, with task-level artifacts and complete answer distributions. Applied and reasoning tracks are kept separate; no relation to the two other Decision-Benches above.
StartLux's diagnostic benchmark (renamed from DecideBench on Oct 14, item hashes preserved) for the control points where decision models sit in LLM systems: contrast groups of 2–4 items identical except one decisive detail, scored on pair accuracy, confident errors, calibration, and automation at a fixed 5% error rate, with accuracy also read at a 0.90 probability threshold. Releases ship SHA-256 item hashes and items found in StartLux-Decision's own training data are removed from the reference track — first-party, but built with the contamination discipline of an independent bench. Paper.
Robustness benchmark for the class: 1,058 curated attacks across meaning-preserving text perturbations, prompt injection, image perturbations, in-image instruction overlays, and instruction injection beside images, with ground truth on whether the attack preserves the correct answer. Findings worth the read: resistance to perturbation does not imply resistance to injection, and label-consistency checks substantially shrink measured attack success.
Fastino's held-out suite: 17 operational domains × 300 examples each, scoring Jev, SemIf, Laya, and GLiFormer head-to-head on the same inputs. First-party to GLiNER2.5-Decide, which tops it.
Probabilistic forecasts of real future events, scored with difficulty-adjusted Brier as outcomes resolve over time — the natural eval surface for the
estimateandforecastfield types extraction models like Clairic introduce, and a different job from deciding over a fixed state.The first adversarial benchmark for the class: black-box attacks flip 46.3% of Jev's correct decisions, and the transfer attacks generalize across tasks. The evaluation companion to the injection papers under Reading.
Theory-of-Mind suite for the class: 11,117 questions from nine ToM benchmarks across six capability groups, scoring five typed decision models against seven generative baselines. Jev takes 73.2% source-balanced accuracy and ranks first or second on every capability; the findings that confidence-based selection stays benchmark-dependent, and that supplied reasons can repair or overturn answers, belong in any assistant-integration plan.
Agent evals and guardrails as Jev decisions: one request per trace. Runs locally on Kev or Laya. From Openlayer.
A separate project sharing the name: metamorphic tests of whether typed probabilities stay coherent when the question is transformed — probability, representation, batch, logic, and choice-set relations. Coherence is not accuracy (uniform answers can be perfectly coherent), but it is the systematic version of the option-order bias the probe studies below describe.
ACL 2026 Findings benchmark superseding RouterBench for the routing job: 400K+ query-model instances, 21 datasets, 33 models, 10 routing algorithms under one framework with cost curves, Pareto fronts, and latency — explicitly built to fix the incomparability between prior router evaluations. Code.
Community typed-decision suites (customer service, invoices, security incidents, agent traces) the open implementations report accuracy and calibration against. Apache 2.0.
Massive Multitask Decision Making (Nov 10): 5,448 fresh questions across 16 categories built to dodge prior benchmark exposure — contexts to 64,817 tokens, option spaces from 2 to 1,024. Headline findings: hard single-select tops out at 76% (WeMM-JEV 27B), and no model exceeds 36.81% on hard multi-select — first hard evidence of a multi-select ceiling in the class.
2,352-question validation set built to mirror JevBench's task mix, released with Releasing the Inner Jev (Geuter, Cheng & Alvarez-Melis) — the paper showing plain instruction-tuned Gemma 4 31B, read through its output logits with no training, already scores 79.6 to Jev's 80.1, and that a simple fine-tuning recipe lifts it to 84.4 here and 81.5 on JevBench v1.7, past Jev. Models and serving code ship with the set.
Parallel's benchmark for the axis no other suite measures: 300 closed-ended questions whose answers depend on current, niche, or scattered live-web facts, each confirmed against a direct source, scored by exact match — with runnable code (MIT) and the dataset on Hugging Face. First-party, and Parallel tops its own board (76.2% vs OpenAI Decisions' 41.7% and Jev's 33.4%) — but the honest framing is in the cross-check: on JevBench's public set the same three sit within 1.7 points, so the spread is web grounding, not decision skill. A JevBench score cannot tell you what happens when the answer isn't in the prompt.
Martian × UC Berkeley's routing benchmark: 405K+ precomputed inference outcomes from 11 LLMs across 8 datasets, so router policies train and compare without live queries, against Oracle and Zero Router bounds. The standard cost-quality surface for the routing job most of this list's Applications section exists to do. Paper.
EMNLP 2025 Findings benchmark for routing at model-pool scale: 200M+ performance records across 8,500+ LLMs and 12 evaluations. Its headline finding — a capable router improves as the candidate pool grows, potentially beating the best single model in it — is the research backing for the whole cheap-first-escalate pattern this list's Tools section builds. Code.
Run one YAML suite of your own labelled cases against any
/v1/systemoneserver (Jev, Laya, kev, llama.cpp, Ollama) side by side, pick the lowest confidence cutoff per question that reaches a target precision, and route cheapest-first with escalation below it. Ships an offline keyword baseline and a GitHub Action that fails CI when accuracy drops.Head-to-head of System One models: Laya vs Jev on byte-identical inputs.
Runnable suite with frozen, hash-checked datasets, identical request payloads, released predictions, and accuracy / calibration / cost / latency per system — 16 datasets, 3,250 cases, 4,850 decisions across English, French, and Arabic, with a public board at systemonebench.dev. The methodology is the reason it's here: local timing kept separate from API comparisons, and latency rows excluded when a control model shows a materially different serving environment — discipline the field's incomparable-milliseconds problem needs. Public datasets can overlap training data; hosted-model runs may change.
Academic benchmark for the class from DFKI, RPTU, Imperial, and Mila: seven policy-labelled generators and nine suites scored as median across paraphrases, calibration against the finite-sample noise floor, selective prediction, ordinal proper scoring rules, and realised cost under asymmetric matrices. Headline finding: the hosted model follows stated policy but is wording-sensitive and systematically underconfident, enough that trusting its probabilities can cost more than just taking the top answer.
The multilingual axis at scale: nine suites across 122 languages — 88,425 texts, 153,875 typed questions, the largest suite in the field — with every suite sharing one scoring contract and aligned, locally-labelled, and native items reported separately. Findings that matter for anyone shipping routing or moderation: the English-best model loses significantly in a quarter of suite-language cells, and an English-tuned 5%-error confidence threshold exceeds that rate on XNLI even in German and French.
### Component datasets 18
Shopping-query relevance classification (exact / substitute / complement / irrelevant) at Amazon scale. A Jev Decision Index component.
Alibaba's tool-use benchmark for whether a model picks the right API call. A Jev Decision Index component.
AI2's grade-school science MCQs in Easy and Challenge splits. Part of the standard MCQ battery used in Jev vs Luna Decisions comparisons.
77 fine-grained banking intents over 13k queries. Single-domain counterpart to CLINC150, and the benchmark Janus reports wins on. CC BY 4.0.
Berkeley Function Calling Leaderboard: the field's standard for function/tool-call accuracy (ICML 2025). A Jev Decision Index component.
Reasoning-intensive retrieval benchmark scored by nDCG — the Jev Decision Index's reranking component, and one Jev still leads.
150 in-scope intents plus explicit out-of-scope examples. Useful for routing, abstention, and confidence-threshold tests. CC BY 3.0.
Graduate-level, Google-proof MCQs; the Diamond subset is the hard tier. Part of the standard MCQ battery used in Jev vs Luna Decisions comparisons.
Grade-school math word problems; repurposed as multiple choice with random distractors (4- and 10-option conditions in the Jev vs Luna comparison).
Commonsense sentence-completion MCQs. Part of the standard MCQ battery used in Jev vs Luna Decisions comparisons.
Multilingual intent-and-slot dataset (about 1M utterances, 52 languages) for testing whether a fast decision layer generalizes. CC BY 4.0.
The standard 57-subject multitask MCQ benchmark. Part of the standard MCQ battery used in Jev vs Luna Decisions comparisons.
Harder MMLU successor with 10-option questions and more reasoning. Part of the standard MCQ battery used in Jev vs Luna Decisions comparisons.
Multistep soft-reasoning MCQs (murder mysteries, object placements, team allocation). Part of the standard MCQ battery used in Jev vs Luna Decisions comparisons.
2,000-email phishing benchmark plus a 220k-run adjudicated grid showing how system-prompt config swings detection. A Jev Decision Index component.
ACL 2025 tool-retrieval benchmark: 7.6k retrieval tasks over a 43k-tool corpus, scored by nDCG@10. A Jev Decision Index component.
Evaluates tool-calling decisions rather than accuracy: when to call, when to ask a follow-up, when to admit the tools can't answer. A Jev Decision Index component — and one Jev still leads.
Pronoun-resolution commonsense MCQs, adversarially filtered. Part of the standard MCQ battery used in Jev vs Luna Decisions comparisons.
### Probes & studies 8
Airbnb's production benchmark of a content-policy classifier, and the most credible production-scale independent data on Jev yet: on a 10,000-example held-out set at thousands of RPM, Jev is "the most accurate off-the-shelf model we tested under 100 ms" and beats GPT-6 Luna at equal cost — but a fine-tuned Gemma4-E2B scoring via logprobs beats everything on all three axes (F1 0.563 → 0.902 at 35 ms median, ~1% of frontier cost), and a 2,700-word prompt fails to close the gap for any deployable model. Three findings worth pinning: thresholds beat verdicts at production scale (the class's whole thesis, confirmed); fine-tuning moves the quality-latency frontier instead of trading along it; and Jev's cost quadrupled under the long prompt because at evaluation time its API offered no cached-input discount — a gap both OpenAI's Decisions GA and Morph Reflex now price.
Measures a confidence threshold on your labeled data, then routes Jev vs a larger model. Ships no default: on Banking77 routing wins; on Web of Science it says do not route.
Independent look at Jev 1.13.0: option-order bias, facts lost in the middle of long context, and cases where a correct prerequisite still leads to the wrong action. Raw payloads and offline verification included.
Independent calibration, selective-risk, and latency measurements of Jev against GLiNER and other baselines.
Independent calibration test on tasks Jev cannot have seen: 900 rule-generated tickets plus public benchmarks, with ECE noise floors and a temperature refit.
Jev vs Claude Haiku 4.5 on 2,000 phishing emails: accuracy, calibration, latency, and cost, reproducible.
Jev vs Cohere Rerank 4 vs zerank-2 vs a chat baseline on 14 datasets. Every raw response saved. nDCG@10 is a tie, not a win.
The skeptic's literature review: 28 papers from the first nine days after Jev's launch, related back to label-probability classification, constrained decoding, and cascades. Headline: the typed readout itself shows no independent accuracy advantage over comparable label-probability readouts — the gains are latency and cost — with a 14-item checklist for future TDM evaluation. Useful calibration for every vendor number in this list.
nothing matches. try fewer words, or clear a filter.