Compared with open Jev-like models¶
Measured 2026-09-30. Eight open "Jev-like" decision models from Hugging Face and SelfJev were each served with their card's recommended setup on the same GPU (one NVIDIA L40S) and asked exactly the requests TypeSafe's Jev receives. A request a model cannot answer counts as wrong. Jev is shown for reference.
SelfJev's test suites. Text Decisions (1,991 questions) and AI Response Review (946), published as selfjev-decision-bench, and photos:
| model | size | images | licence | Text Decisions | AI Response Review | held-out images | Text Decisions time (s) | AI Response Review time (s) |
|---|---|---|---|---|---|---|---|---|
| Jev 1.13 (TypeSafe API, reference) | ? | no | paid API | 97.2 | 92.5 | — | — | — |
| openjev | 27B | yes | CC-BY-NC-4.0 | 96.8 | 92.6 | 82.6 | 753 | 329 |
| SelfJev-4B Vision | 4B | yes | Apache-2.0 code | 94.7 | 90.7 | 85.1 | 100 | 41 |
| SelfJev-4B Vision, version 0.4.1 | 4B | yes | Apache-2.0 code | 94.7 | 90.7 | 85.1 | 167 | 68 |
| Plumb-4B | 4.2B | no | Apache-2.0 | 93.4 | 84.1 | — | 471 | 164 |
| jpt-4b | 4.5B | yes | CC-BY-NC-4.0 | 92.5 | 85.0 | 83.8 | 90 | 40 |
| imajev-4b | 4B | yes | Apache-2.0 | 91.5 | 84.9 | 85.2 | 489 | 155 |
| decider-4b | 4.2B | no | Apache-2.0 | 90.8 | 82.8 | — | 145 | 52 |
| Mica-v0.1-4B | 4.2B | no | Apache-2.0 | 89.6 ² | 84.1 ² | — | 162 | 69 |
| kev-4b | 4B | no | Apache-2.0 | 88.4 | 74.7 | — | 109 | 34 |
| Laya | 0.4B | no | Apache-2.0 | 45.4 | 46.4 | — | 23 | 11 |
- Every model gets the same Jev-shaped questions; select-all questions become one yes/no per option, which costs SelfJev 1.4–1.8 points against its native evaluation (96.1 and 92.5). Held-out images: the 4 image datasets of the SelfJev image test that none of these models trained on (772 questions). Times: each suite end to end, 4 requests in flight; SelfJev 0.4.2 was measured 2026-10-05 on the same GPU model, and the row for version 0.4.1 is the same weights on the engine before it. ² Refuses inputs over 8,192 tokens (2 % and 1 % of the questions), counted wrong.
- No other model trained on these suites, but they come from the same authors and judges as SelfJev's training data, so they favour SelfJev.
- SelfJev is the most accurate open model at 4.5B parameters or less on both suites (paired tests, p ≤ 0.023) and ties imajev-4b on held-out images; the 27B openjev is higher on Text Decisions. Among the 4B models only jpt-4b is faster on Text Decisions, and kev-4b and jpt-4b on AI Response Review.
Protocol, per-model setups, paired tests and raw reports: reports/competitors.
Version 0.4.2 against 0.4.1¶
Version 0.4.2 (2026-10-05) changed the engine, not the model. The same weights on each version's engine; accuracy is SelfJev's native evaluation, where a select-all question stays one question:
| version 0.4.1 | version 0.4.2 | ||
|---|---|---|---|
| Text Decisions end to end, 1,991 questions, 4 requests in flight (L40S) | 167 s | 100 s | 1.7× faster |
| AI Response Review end to end, 946 questions (L40S) | 68 s | 41 s | 1.7× faster |
| One request, 2,048-token text, 1 question (A10G) | 584 ms | 434 ms | 1.3× faster |
| One request, 2,048-token text, 1 question (L40S) | 243 ms | 146 ms | 1.7× faster |
| One request, 2,048-token text, 1 question (H100) | 166 ms | 99 ms | 1.7× faster |
| One request, 32,000-token text, 1 question (A10G) | 22,019 ms | 6,721 ms | 3.3× faster |
| One request, 32,000-token text, 1 question (L40S) | 9,146 ms | 2,739 ms | 3.3× faster |
| One request, 32,000-token text, 1 question (H100) | 4,033 ms | 1,055 ms | 3.8× faster |
| Accuracy, Text Decisions | 96.1 | 96.2 | unchanged (a few near-tie questions differ) |
| Accuracy, AI Response Review | 92.6 | 92.6 | unchanged (a few near-tie questions differ) |
| Accuracy, Image test | 90.5 | 90.5 | unchanged (a few near-tie questions differ) |
Every text length and question count on three GPUs: question sweep. What changed in the engine: report.