Shopping-query relevance classification (exact / substitute / complement / irrelevant) at Amazon scale. A Jev Decision Index component.

github.comPythonApache-2.0pushed jul 280 forksadded oct 1

Alibaba's tool-use benchmark for whether a model picks the right API call. A Jev Decision Index component.

github.comadded oct 1

AI2's grade-school science MCQs in Easy and Challenge splits. Part of the standard MCQ battery used in Jev vs Luna Decisions comparisons.

allenai.orgadded oct 7

Audio-decision counterpart to Image JevBench, for the multimodal entrants (Clef-omni, d1-omni, NeuDecide).

benchmarkheaven.comadded oct 10

77 fine-grained banking intents over 13k queries. Single-domain counterpart to CLINC150, and the benchmark Janus reports wins on. CC BY 4.0.

github.comPythonCC-BY-4.0pushed apr 29 202257 forksadded sep 21

Berkeley Function Calling Leaderboard: the field's standard for function/tool-call accuracy (ICML 2025). A Jev Decision Index component.

gorilla.cs.berkeley.eduadded oct 1

Reasoning-intensive retrieval benchmark scored by nDCG — the Jev Decision Index's reranking component, and one Jev still leads.

arxiv.orgadded oct 1

150 in-scope intents plus explicit out-of-scope examples. Useful for routing, abstention, and confidence-threshold tests. CC BY 3.0.

github.compushed jun 1 202147 forksadded sep 20

80 scenarios and 293 questions per tier (medium is rules-based, hard is judgment calls), with a cost-per-decision axis baked in. What Jev-Omni reports on. Apache 2.0.

hugging face datasetapache-2.0pushed sep 221.8k downloadsadded sep 22

TypeSafe's workflow evals: four production-shaped workflows scored against frontier LLM reference probabilities.

evals.typesafe.aiadded sep 20

Fastino's held-out suite: 17 operational domains × 300 examples each, scoring Jev, SemIf, Laya, and GLiFormer head-to-head on the same inputs. First-party to GLiNER2.5-Decide, which tops it.

hugging face datasetapache-2.0pushed oct 82.1k downloadsadded sep 25

Graduate-level, Google-proof MCQs; the Diamond subset is the hard tier. Part of the standard MCQ battery used in Jev vs Luna Decisions comparisons.

github.comJupyter NotebookMITpushed sep 30 202460 forksadded oct 7

Grade-school math word problems; repurposed as multiple choice with random distractors (4- and 10-option conditions in the Jev vs Luna comparison).

github.comPythonpushed jan 21 2024203 forksadded oct 7

Commonsense sentence-completion MCQs. Part of the standard MCQ battery used in Jev vs Luna Decisions comparisons.

rowanzellers.comadded oct 7

Held-out image-decision suite: 684 items (228 public / 456 sealed) with frozen hashes and contamination tracking — it flagged Kev's Mind2Web training overlap itself. Jev-Omni, decider-vision, and Reflex top it; frontier chat APIs score near zero.

benchmarkheaven.comadded sep 26

Community index aggregating ~43 decision-model evals — intent classification, function calling, tool retrieval, reranking — plus latency. The benchmark surface Clef's launch numbers report on.

hugging face spacepushed oct 12added oct 1

Independent decision-model leaderboard (run by Benchmark Heaven, not TypeSafe): 1,500 typed decisions per system scored on intelligence, calibration, latency, and cost, with versioned releases, published methodology, and result JSON. Two boards — open weights and hosted API — across ~75 systems; the hosted board currently ranks Sage #1, d1 #2, Mercury Decide #3, Jev #4.

benchmarkheaven.comadded oct 10

Agent evals and guardrails as Jev decisions: one request per trace. Runs locally on Kev or Laya. From Openlayer.

github.comPythonMITpushed oct 110 forksadded sep 23

Community typed-decision suites (customer service, invoices, security incidents, agent traces) the open implementations report accuracy and calibration against. Apache 2.0.

hugging face datasetpushed oct 1343k downloadsadded sep 21

Multilingual intent-and-slot dataset (about 1M utterances, 52 languages) for testing whether a fast decision layer generalizes. CC BY 4.0.

github.comPythonpushed nov 28 202260 forksadded sep 20

The standard 57-subject multitask MCQ benchmark. Part of the standard MCQ battery used in Jev vs Luna Decisions comparisons.

github.comPythonMITpushed may 28 2023122 forksadded oct 7

Harder MMLU successor with 10-option questions and more reasoning. Part of the standard MCQ battery used in Jev vs Luna Decisions comparisons.

github.comPythonApache-2.0pushed mar 1858 forksadded oct 7

Multistep soft-reasoning MCQs (murder mysteries, object placements, team allocation). Part of the standard MCQ battery used in Jev vs Luna Decisions comparisons.

github.comPythonMITpushed aug 10 202413 forksadded oct 7

2,000-email phishing benchmark plus a 220k-run adjudicated grid showing how system-prompt config swings detection. A Jev Decision Index component.

hugging face datasetpushed apr 182k downloadsadded oct 1

Head-to-head of System One models: Laya vs Jev on byte-identical inputs.

github.comPythonMITpushed oct 102 forksadded sep 23

ACL 2025 tool-retrieval benchmark: 7.6k retrieval tasks over a 43k-tool corpus, scored by nDCG@10. A Jev Decision Index component.

github.comJavaScriptApache-2.0pushed dec 22 20255 forksadded oct 1

Evaluates tool-calling decisions rather than accuracy: when to call, when to ask a follow-up, when to admit the tools can't answer. A Jev Decision Index component — and one Jev still leads.

aclanthology.orgadded oct 1

Pronoun-resolution commonsense MCQs, adversarially filtered. Part of the standard MCQ battery used in Jev vs Luna Decisions comparisons.

winogrande.allenai.orgadded oct 7

Measures a confidence threshold on your labeled data, then routes Jev vs a larger model. Ships no default: on Banking77 routing wins; on Web of Science it says do not route.

github.comPythonMITpushed oct 5added sep 20

Independent look at Jev 1.13.0: option-order bias, facts lost in the middle of long context, and cases where a correct prerequisite still leads to the wrong action. Raw payloads and offline verification included.

github.comPythonMITpushed sep 17added sep 20

Independent calibration, selective-risk, and latency measurements of Jev against GLiNER and other baselines.

github.comPythonApache-2.0pushed sep 175 forksadded sep 20

Independent calibration test on tasks Jev cannot have seen: 900 rule-generated tickets plus public benchmarks, with ECE noise floors and a temperature refit.

github.comPythonMITpushed oct 5added sep 23

Jev vs Claude Haiku 4.5 on 2,000 phishing emails: accuracy, calibration, latency, and cost, reproducible.

github.comPythonpushed sep 191 forksadded sep 23

Jev vs Cohere Rerank 4 vs zerank-2 vs a chat baseline on 14 datasets. Every raw response saved. nDCG@10 is a tie, not a win.

github.comPythonMITpushed sep 25added sep 20