Jev and open alternatives¶

What Jev is¶

Jev (TypeSafe, typesafe/jev, served through OpenRouter) turns a model into a typed decision engine: send a text (the state) and questions of type noul (yes/no), choice or score, get typed answers with probabilities, with no text generation. SelfJev copies the interface, not the model: selfjev serve answers Jev's requests at Jev's paths (/v1/systemone, /api/alpha/decisions, /v1/decisions), so Jev clients work by changing the base URL and the key (API, how it works).

What TypeSafe discloses (public web, read 2026-09-23):

• "Transformer-based, but it is not a large language model", with a "new model architecture, parallel sampler": all outputs come from one pass.
• Questions run "in parallel and in isolation against the same state". That is the shape of our shared-prefix tree.
• Limits: 64K tokens per request; 32K for the state plus the longest question. Rate limit 250K tokens/s.
• Training: "Reinforcement Learning for Calibrated Decisions (RLCD)", no details. Their benchmark's reference answer is the average of GPT-6 Astra and Fable 5.1.
• Not disclosed: size, base model, data. They "cannot prove the price is unsubsidized".

Sources: typesafe.ai/blog/introducing-system-one-models-and-jev, docs.typesafe.ai/models.md, marktechpost.com/2026/09/19/typesafe-ai-releases-jev/.

What we measured about it:

Our reading (2026-09-25 update: what Jev's behaviour implies and what to build next): the gap is mostly training, not architecture. 4B → 8B, a bigger adapter and stock → tree each moved the dev benchmark by ≤ 1.3 points, while 10K verified target-task questions moved eval2 by +5.5. Our default model, selfjev-4b, is now 1.4 points behind on eval2 (95.8), level on eval_llm (93.1, p = 0.52) and ahead on the dev benchmark (83.8). Jev's speed is mostly the hardware: on one H100 our Qwen3 4B tree is faster than Jev inside the machine in every cell but 4,096 tokens × 16 questions (2026-09-26, speed); Jev's per-token cost is still ≈ 6× lower than ours, so a smaller model or more GPUs per request, and the end-to-end difference is network distance. selfjev-4b's own serving engine is not timed on a GPU yet.

The open field and the shared yardsticks (survey 2026-09-30)¶

About 50 model cards read from HF other=classification (sorted by downloads, likes and trending, created since 2026-09-15), plus the benchmarks they cite. Nothing was run for the survey. Session: Claude competitors, JOURNAL 2026-09-30.

Shared yardsticks. These are the only places where our numbers and theirs can meet:

Overlap check (exact + 8-gram, data/all.jsonl.gz vs the benchmark states): JevBench public-231 0 / 231, typed-decisions test 0 / 400. We never trained on either. On the Nimble suite, MNLI's train split is in our hf_nli.

Measured head-to-head (2026-09-30): the same requests, on one L40S (protocol, recipes and caveats). Eight open models, each served with its card's recommended recipe, and selfjev-4b-vision, all asked exactly the questions Jev gets (scripts/eval/score_systemone.py); failed requests count as wrong. Public sets through their own harnesses. Generated by scripts/eval/competitors/matrix.py:

• Our frozen sets: selfjev-4b-vision is first among the open 4B models on eval2 and eval_llm; every paired test against a 4B favours it (closest: Plumb-4B on eval2, 79 vs 52 questions only one of the two gets right, p = 0.023). openjev (27B, non-commercial) beats it on eval2 (96.8, 30 vs 72, p = 4e-5) and is level on eval_llm (92.6, p = 0.063), 3× slower on the same GPU. Jev on the same requests: eval2 97.2, eval_llm 92.5. These sets are ours (same authors and judges as our training data), so the shared benchmarks below matter more for a public claim.
• Through Jev's request shape we lose points against our own evaluation (eval2 94.7 vs 96.1, eval_llm 90.7 vs 92.5): multilabel becomes one yes/no per candidate (86.1 vs 90.8 on eval2). The native multi question is our edge.
• On the shared public benchmarks we are mid-pack, not first. JevBench public-231 83.5 (zero-shot) vs Plumb 89.6, jpt-4b 87.9, openjev 87.4, imajev-4b 85.7, decider-4b 83.1, Mica 82.7, kev-4b 75.8 (Jev 86.6 on the board). typed-decisions 66.1: below openjev (71.1), imajev (68.5), Mica and decider (68.0), kev (67.0) and Jev (72.7), above Plumb (61.4); jpt-4b's 78.8 is in distribution (its card). Nimble 13 macro 74.8 vs Jev 76.0 and Nimble-9B 74.8 on their runs. Our weakest question type there is score (60.9).
• Images: on the 4 datasets nobody trained on, a tie (85.1 vs imajev-4b 85.2, jpt-4b 83.8, openjev 82.6). Our 94.0 on the other 6 comes from fine-tuning on them.
• Speed: second among the 4B models on this workload (eval2 in 100 s with selfjev 0.4.2, measured 2026-10-05; 166 s with 0.4.1 and 242 s before; jpt-4b 90, kev-4b 109, decider-4b 145, Mica 162, Plumb 471, imajev 489, openjev 753). End to end with 4 requests in flight, not a controlled latency benchmark (engine report).
• Every card number we could reproduce matched (JevBench: decider, kev, Mica, Plumb, jpt; typed-decisions: Laya 36.0 vs 36.2), so the harnesses measure what the cards measure.

Accuracy matrix, published numbers only (%). B = run by the board's maintainers, S = self-reported by the model's author, M = measured by us on 2026-09-30 with the benchmark's own harness, † = the author used the public items for model selection or tuning, F = fitted on the benchmark's train split. JevBench public-231 totals come from the per-tier numbers on each card where the card gives no total.

Eikos-4B was scored on eval2 on 2026-09-24 through our option mapping: 92.8 (reports/eikos_4b/eval2).

What the survey and the head-to-head say about our claims.

• Architecture. The shared prefix is common now: Mica shares one prefill across questions, kev continues each question from the shared state, ArseneLupin has a shared_prefix mode, decider and Eikos lean on vLLM prefix caching. What remains ours: each candidate is scored from its own description inside one tree pass (no A–Z letter slot, no 26/52-option chunking) and multilabel is native. Most others read letter logits; several cap options at 16–26.
• Text and images in one model is not unique: jpt-4b/9b, imajev-4b, openjev, vjev-vision, AutoJev-27B and decider-2b-vision take images too. imajev-4b leads Image JevBench (76.39). Our image test is mostly in distribution (JOURNAL 2026-09-30 02:05: no transfer to new image tasks); on held-out image datasets we tie imajev-4b (above). Image JevBench's public items are not downloadable, so only its maintainers can score us there.
• Real data. Most cards report on their own sets or on public items they tuned against (†). Maintainer-run numbers (B) are rare and worth more: sealed accuracy on JevBench is far below public for everyone (Jev 36.7 sealed vs 86.6 public on v1.4.2.2).
• Laya is the most liked but the weakest here: JevBench board #93, Decision Index 6.04, a ≈ 320-token state window.

Open "Jev-like" models (survey of Hugging Face model cards, 2026-09-24)¶

Nothing was downloaded or run for the survey (Eikos-4B was scored later, below); claims are each model's own, on its own benchmark, and not comparable with ours.

• Their claims: AutoJev-27B 84.6 vs Jev 82.8; kev-4b beats Jev in distribution but trails out of domain (0.817 vs 0.857); kev-0.8b trails everywhere; decider-35b-a3b JevBench hard 0.676, decider-4b 0.541.
• The same pattern as ours: sub-1B trails, 4B and up is where the quality is.
• "JevBench" public items exist and several cards report on them: the only shared yardstick. Scoring our tree on it is an open idea.

Laya (reviewed at commit 970dc8c)¶

• A bidirectional encoder (ModernBERT-large 421M, or mmBERT-base 322M multilingual), one sequence per question with every option and the state; a new 2-layer transformer head scores each option's [MASK]. Trained with a GRPO-style policy gradient on proper-scoring-rule rewards, which it calls RLCD.
• Documented limits: the state is truncated to ≈ 320 tokens by default (≈ 768 multilingual, up to 8K opt-in), options share a 192–256-token budget, base checkpoints near chance zero-shot on its own set, over-confident before temperature fitting (ECE 0.466 → 0.081).
• Its Jev latency figure (236–276 ms p50) is slower than we measured (143–178 ms).
• Verdict: the faster and multilingual option (≈ 33 ms on a T4), not the more accurate one on long, trap-heavy texts. Our closest measured analogue, jina 0.6B listwise, scores 73.3 on eval2. Scoring Laya on eval2 is open (no API cost; a small GPU box, never the laptop).

Eikos-4B (caiovicentino1/Eikos-4B, MIT)¶

• A full fine-tune of Qwen3.5-4B distilled from GLM-5.3-Flash on finance and rules data; options as letters, softmax over the letter logits.
• vLLM ≥ 0.30 with --enable-prefix-caching --mamba-cache-mode all shares the state across questions (the card reports that vLLM 0.11 lost 3–6 points on batched long shared documents).
• Own-harness claims: JevBench public hard 72.1 (Jev 73.0 on the official leaderboard), ECE 0.033.
• Evidence that the Qwen3.5 base works for typed decisions. Zero-shot through our option mapping it scores 92.8 on eval2 (reports/eikos_4b/). Qwen3.5-4B has been the base of our best models since 2026-09-25, selfjev-4b included.

Qwen3.5 and Qwen3.8 as bases¶

• Qwen3.8 has no small dense model: 27B (48 linear-attention + 16 full layers), a 180B MoE and a 2.4T MoE.
• Qwen3.5 has 0.8/2/4/9/27B; the 4B has 32 layers (24 Gated DeltaNet + 8 full attention) and 262K positions. A tree mask cannot branch its linear-attention layers in one pass. First a forked native cache did it (challengers); now our Qwen3.5 tree (src/selfjev/engine/tree.py) runs those layers level by level from copied states, in training and serving (tree scorer).