Speed and cost¶

Tree vs stock pairs (same A10G, bf16, unmerged LoRA)¶

End-to-end p50, one request at a time. Sources: reports/bench/gpu_{tree_4b,stock_lora_4b}_bf16/ and …_long_bf16/.

The stock model pays pairs × text tokens. The tree pays for the text once plus a short branch per question and candidate, so adding questions barely changes its time. With one binary question there is nothing to share.

End to end vs Jev (2026-09-23)¶

Same decisions-API requests from a Mac in California, one at a time, endpoints in random order. Jev through OpenRouter (edge 12 ms away); ours, tree 4B + LoRA, on one AWS A10G in us-east-1 (71 ms away). p50 over 10 timed rounds; choice questions with 3 options. Full tables with p95 and server-side times: reports/latency/summary.md.

How Jev stays flat. A fit on its 400 requests gives 132–137 ms fixed + 2.2–2.6 ms per 1,000 input tokens: about 400K tokens/s of marginal speed for one request, against ~6K tokens/s for our tree on the A10G. Every architecture must touch each token, so the flat curve means a very small cost per token: consistent with a ~0.5–1B-parameter model (or an MoE with ~1B active) on H100/B200-class GPUs, or a larger model split over several GPUs. Jev does read the whole text: 97.2% on round-2 questions whose evidence is at the end of a text over 4K tokens (up to 17.6K).

The same model on an H100 (2026-09-26)¶

The same sweep, same day, same client: Jev through OpenRouter (11 ms away) against tree_4b_combo merged on vLLM 0.30 on one p5.4xlarge spot (1× H100 80 GB, ≈ $2.54/h, us-east-2, 61 ms away), in bf16 and with vLLM's dynamic FP8 (--quantization fp8). The A10G and L40S columns are the earlier sweeps of the same model. p50 ms, wall at the client / server inside the box (Jev: inside OpenRouter). Raw rows: reports/latency/requests_h100.jsonl. The model's weights, its vLLM code and the sweep script (scripts/latency_sweep.py) are at tag archive/pre-cleanup-2026-09-27.

• Inside the machine the H100 is 2–5× faster than Jev's server time, at every size with one question and up to 2,048 tokens with 16; the one slower cell is 4,096 tokens × 16 questions (189 vs 134 ms).
• End to end we trail by 10–100 ms, and that is the network: 61 ms to Ohio against 11 ms to OpenRouter's edge. Served from a point as close as theirs, this model beats Jev's latency.
• FP8 changes nothing: a 4B on an H100 is bound by per-request overhead at these sizes, not by arithmetic.
• Per-token cost, one question, server-side: ours ≈ 20 ms fixed + ≈ 15 ms per 1,000 text tokens (≈ 68K tokens/s); Jev 132–137 ms fixed + 2.2–2.6 ms per 1,000 (≈ 400K tokens/s). Jev's marginal cost per token is still ≈ 6× lower, so it is a smaller model or more GPUs per request, but at request sizes up to 4K tokens the fixed costs decide.
• Accuracy is kept: eval2 through vLLM on the H100 scores 94.42 in bf16 (5 of 1,991 decisions differ from the transformers run's 94.48) and 94.48 with FP8 (25 differ); all 1,991 questions take 16–18 s (A10G: 144 s).
• Cost, GPU fully busy (reports/latency/throughput_h100.json, at $2.63/h): 294 requests/s and $0.0025 per 1,000 at 8 tokens × 1 question (Jev $0.016); 105 requests/s and $0.007 at 512 × 1 (Jev $0.038); 5.9 requests/s and $0.124 at 4,096 × 16 (Jev $0.265). 2.1–6.5× cheaper than Jev in every cell on spot; at $6.88/h on demand still below Jev everywhere except within 10% at 4,096 × 16.
• So the speed gap was hardware, not architecture. For this model speed is a deployment question (GPU class and placement); the default selfjev-4b still has to be timed on its own engine (below).

Serving optimizations (2026-09-24)¶

A controlled experiment on one A10G, R1 LoRA, 2,048-token text × 16 questions × 3 candidates, new document each request, model resident, network excluded (conclusions):

• vLLM keeps quality: 81.53% on the dev benchmark vs 81.50% for merged transformers.
• Merging the LoRA into bf16 weights costs a little precision: unmerged 81.68% vs merged 81.50%.
• The compact format moves repeated instructions into the shared root: 38.5% fewer branch tokens (2,119 → 1,303). It is 22.6% faster on transformers but only 7.4% on vLLM, below the predeclared 15% gate, and it adds CLINC over-rejection (94.67 → 91.67%, all nine changes to none). Experimental only.
• Cache matters: with the document root already cached, R1 vLLM answers new questions in 409 ms; a fully repeated request takes 185 ms. Those are different workloads from a new document.
• Faster GPUs (L40S, H100, Blackwell, A100) could not be launched in three regions that day; the L40S (2026-09-25) and H100 (2026-09-26) numbers on this page came later.

Qwen3.5 on vLLM (2026-09-25, L40S)¶

qwen35_4b_tree (the same architecture as selfjev-4b) merged into Qwen3.5-4B and served by vLLM 0.30 (the code is now src/selfjev/engine/vllm.py: selfjev merge, then selfjev serve --engine vllm), next to the Qwen3 tree (tree_4b_combo) on the same GPU, with Jev in the same sweep from California. Accuracy through vLLM is unchanged: eval2 95.58% (transformers 95.58%), and 94.53% for the Qwen3 tree (94.48%). Server-side p50 (ms; Jev: time inside OpenRouter):

Both models run at about the same speed per token (17–24K tokens/s); the difference is how much of the text each has to recompute. vLLM's prefix cache shares an attention model's text at any 16-token boundary, so the Qwen3 tree computes the text once and each candidate only adds its own tokens. For Qwen3.5 the cache has to store the recurrent state itself (tens of MB), and vLLM sets its block to 528 tokens for that; every candidate prompt recomputes the text after the last block boundary plus its question:

mamba_block_size does not change this in vLLM 0.30; caching the state in bf16 halves the block to 272 tokens, with mixed effects (256 tokens 407 ms, 1K 744 ms, 4K 689 ms). Sources: JOURNAL 2026-09-25 18:05, reports/latency/requests_qwen35.jsonl, the probe script reports/qwen35_4b_tree/vllm/cache_probe.py (its log is not in git).

What serves selfjev-4b now. The fix is to serve Qwen3.5 with its own tree, as in training: TreeServer (src/selfjev/engine/tree.py) computes the text once, each question once and each candidate's own tokens, the same work as the Qwen3 tree. Since the 2026-09-27 cleanup it is the default engine of selfjev serve, eval and bench (vLLM stays available with --engine vllm). It matches standalone sequences in a CPU test (tests/engine/test_tree.py) and, on GPU, the forked-cache engine's answers: the same selfjev-4b weights score the same on both engines: eval2 95.68 vs 95.78, dev benchmark 83.78 vs 83.75, eval_llm 93.13 vs 93.13 (99.8%, 99.8% and 100% of decisions identical) (JOURNAL 2026-09-27 11:55). But its latency has not been measured on an NVIDIA GPU, including the L4 (g6.xlarge) that selfjev deploy aws picks by default: there are no benchmark numbers for it yet. The end-to-end test (2026-09-28: an L40S, the adapter unmerged for fine-tuning, round trips including ~70 ms of network) saw 0.2–0.3 s for one warm request with six questions, 39 s for the very first request (kernel compilation, now done at startup), and 16 concurrent requests, batched together, returning after 18.6 s cold and 4.4–6.4 s warm while a fine-tuning job shared the GPU: requests in one batch all wait for it. selfjev bench on a GPU box is the first step. For reference, the forked-cache engine it replaced took 156 and 252 ms for one question at 512 and 2,048 tokens, and 341 and 508 ms for 16 questions (in-process p50 on the L40S, reports/qwen35_4b_tree/bench.json).

Current model on Apple Silicon (2026-09-28)¶

The current selfjev-4b adapter merged into Qwen3.5-4B runs through TreeServer on a local Apple M5 Pro (20-core GPU, 48 GB unified memory) with PyTorch MPS in bf16. The CLI still defaults to CUDA; this is a lower-level engine experiment, not a validated Mac server. Each cell uses one request with synthetic repeated text and three options per question. Times include tokenization and scoring but exclude HTTP/network. Median of 10 timed calls after two warm-ups, with MPS synchronized at both ends. Raw samples · reproduction script.

flash-linear-attention is not installed for MPS, so the gated recurrent operation uses the correct but slower PyTorch reference path. The NVIDIA figures above belong to an older Qwen3 model on vLLM; their difference from these Mac times does not isolate the hardware effect. No controlled NVIDIA latency measurement exists for the current model.

Cost vs Jev¶

Jev charges $0.042 per million input tokens and bills about 372 tokens of overhead per request plus about 113 per extra 3-option question. Our cost assumes the A10G ($1.006/h) is fully busy (batched throughput):

A g5.xlarge left on all month (≈ $734) beats Jev only above about 7 requests/s sustained, for 512-token requests.

On an L40S ($2.242/h, fully busy; reports/latency/throughput_*_l40s.json), per 1,000 requests:

Other backends, for scale¶