~ / serving-runtimes
##Serving & runtimes
Run one locally, or turn any open model into one, without new training.
Run one, or turn any open model into one. Same request shape, no new training.
### Serve any open model 22
Turns any LLM into a Jev-style decision model: typed decisions with probabilities, no training, reading answers from the next-token distribution with cyclic shifts and prior correction to strip positional and label bias. From Nokia Applied Research, Apache 2.0. Its self-distilled Tacit models are listed under Models.
DiffusionGemma plus vLLM: an inference method rather than new weights, with native image inputs and a hosted API. Ranked 3rd on JevBench v1.2, just under Jev itself.
Decision-model inference engine from the OneJev authors, now with the class's first serving paper (arXiv 2610.42148, NUS): exact state forking — each question reuses the shared state's computation in a separate forward pass, giving up to 9.5× over vLLM and SGLang with answers bit-identical to the question alone, and
flide verifychecks that on your own model. Serves 30+ model families through/v1/systemonewith per-model speed tables published (Clef 1.3× through Kev ~7.9× and OneJev ~6.8×; multi-request batching, session caching for agents, FP8). The email triage demo is the class's best throughput number: 10,000 emails × 8 questions in 28 s on one H200. Apache 2.0,pip install flide, ~318 stars.Self-hosted drop-in on GLiFormer 400M, MIT. Speaks the TypeSafe request shape (JevBench drove it with the official adapter); ranked in v1.2.2.
Educational Jev-like visual inference experiment on Apple Silicon: shared context, direct candidate scoring, and local visual demos.
Jev-style parallel constrained decisions for any MLX model on Apple Silicon. Scores every allowed answer per schema field in one pass, so the output is valid by construction.
Native decision-model serving since PR #29818 (Oct 2):
llama-serverexposes/v1/systemoneover GGUF checkpoints — Julia-1, Laya, Kev-4B, Lev, OpenJev 27B, with vision via Clef and later additions — onellama serve -hf ggml-org/Kev-4B-GGUFaway on a laptop or CPU box. The inference stack underneath Ollama, now speaking the wire itself. Launch postAdapts local language models into Jev-compatible decision engines with Choice / Score / Noul outputs, via prefill-only binary inference. Apache 2.0.
Native decision-model support since 0.35.0:
/v1/systemoneon localhost speaks the Jev API shape, so the official TypeSafe SDK works by repointing it.ollama pullserves Nimble 9B, Tev1, d1, Clef and Clef-Flash, and pplx-decider, with base64 images on the multimodal ones; the honest limits are published too — a 64 KiB request cap, no streaming, and a confidence score that measures distribution peakedness, not calibrated correctness. The launch post demos Pac-Man on Nimble at ~91 ms per decision on a laptop.Ollama for decision models:
ollaya pull/serve/runmanage Laya, decider, NLI, and GLiClass locally behind a wire-identical TypeSafe/v1/systemone(and/v1/decisions) endpoint. A separate project from Ollama's own native support above, built before it and worth keeping for the multi-model management.System One-style layer on open weights, ranked on JevBench. Its cautionary detail: 72% vs 21% on the same answer-judging items with option order flipped.
Open, Jev-compatible System One decision server on DiffusionGemma. Apache 2.0.
TypeSafe
/v1/systemoneon Qwen 35B MoE via SGLang. Prefill once, then first-token logits per question. No extra training.Open recreation on Qwen3.5. Prefills the state once, then scores every question in parallel from next-token logits. Serves the TypeSafe request shape. Browser demo on WebGPU. #3 on Image JevBench's original pool; on v0.3.0 it's #13 under the Jev-class caps (53.3).
Runtime-defined semantic decisions from direct option logits. CUDA, MLX, and WebGPU, plus committed benchmarks. Repo recently renamed from SemIf.
Native decision-model serving since v0.5.21 (Oct 2, merged upstream Sept 25): both a
/v1/decisionsendpoint that converts any Jinja-templated chat model into a decision model and a/v1/systemoneendpoint speaking the exact TypeSafe wire — official SDKs work by repointing the base URL. Any served generation model qualifies (validated on Qwen3.8-27B and Qwen3.5-35B-A3B, sub-100 ms decisions), no decision-specific training required. At ~38.6K stars the largest runtime the list covers; the stack xor pins its serving bundle to and openjev-sglang wraps, now first-class.Turns any open model into a classifier / Jev endpoint. From Featherless, Apache 2.0.
Turns any open LLM into a System One classifier: batched single-token choice inference, TypeSafe SDK compatible. Doom and wikiracing demos on Qwen3-8B.
Apple's Neural Engine as a free local System One decision engine on macOS: "Jev at home."
Mode's native vLLM serving for Jev and decision models — prefill-only letter-logit scoring inside vLLM rather than through an adapter shim, ~954 stars and the default route for teams already on vLLM. Apache 2.0.
Upstream vLLM serves
/v1/systemonedirectly: decision models with a supported read strategy run as pooling models and answer choice / noul / score from label log-prob mass (--logprobs-mode raw_logprobs), so the shared state rides prefix caching like any chat prefix. What Clef-Flash and H2O's shim build on, as an official endpoint rather than a per-model hack; distinct from the vLLM Semantic Router listed under Applications.Non-autoregressive local drop-in, sub-15 ms decisions, Apache 2.0.
nothing matches. try fewer words, or clear a filter.