Journal: what is running, what happened when¶

Current state (every test result, conclusions, dead ends, open ideas to claim) lives in experiments.md. This file is the chronological part: jobs in flight and a dated log, newest first. Times are PDT (the user's clock) unless marked UTC. Rules for every session: AGENTS.md.

In flight (add a row when you start, delete it when you are done)¶

Spend so far (real cost, BYOK upstream included)¶

The tree and custom-model GPU runs and the unknown boxes are not in this table yet: their owners should add them.

Log¶

2026-10-05: batch cc_veryhard_v1 written: 436 very hard questions by Claude Code subagents (Claude Code typed-batch, user request)¶

• What: one in-session subagent per question type (.claude/agents/selfjev-writer.md; started on Opus 5.5, switched to Sonnet 5.5 by the user after 7 texts; no API), every question with 2+ traps aimed at the slices where the vision default trails Jev on eval2 (multi-positive, multi-turn, double negation, numeric, hypothetical, distractor, …) and at the weak eval_llm use cases (verify, score). State length drawn log-uniform 64–8,192 tokens. Test-diagnosis-motivated; writers saw only trap names and slice scores.
• Result (data/batches/cc_veryhard_v1/raw/, all lines pass selfjev.data check): binary vhbin 65 texts / 215 q (49% true; stopped by the user), multiclass vhmc 32 / 100, multilabel vhml 47 / 121 (stopped at ≥ 100 q, user). 40 of 144 texts are LLM evaluation. Removed vhml-0039 (two wrong labels, rewritten correctly as vhml-0040).
• Writer-flagged doubts, left in: vhml-0009, vhml-0015 q2, vhml-0028, vhml-0042 q2, vhml-0047 q3, vhmc-0010 q3, vhmc-0025 q1/q4; vhmc-0004 is 31% short of its drawn length. Multilabel is thin on double negation, hypothetical, paraphrase and injection (4 texts).
• Not done: no external judge (user, 2026-09-29); not built into data/all.jsonl.gz yet (grow_batch.sh build needs judge answers or --unverified). Cost $0 (Claude Code session).

2026-10-05 17:45 PDT: the question sweep again with 0.4.2 on A10G, L40S and H100; website and docs show 0.4.1 next to 0.4.2 (Claude competitors, user request)¶

• Run: selfjev bench --lengths 512,2048,8192,16384,32000 --questions 1,5,10,25,50 (the 2026-09-30 grid) with selfjev 0.4.2 on g5.xlarge, g6e.xlarge and p5.4xlarge spot in parallel. Summary with the 0.4.1 numbers next to it: reports/bench/selfjev4b_qsweep_v042_summary.md (tables by scripts/eval/qsweep_summary.py, new); raw: reports/bench/selfjev4b_qsweep_v042_{a10g,l40s,h100}/. The 0.4.1 folders and summary keep their names.
• One question, p50, 0.4.1 -> 0.4.2: A10G 241 -> 158 ms (512 tokens), 584 -> 434 (2K), 22.0 -> 6.7 s (32K); L40S 145 -> 93, 243 -> 146, 9.1 -> 2.7 s; H100 146 -> 94, 166 -> 99, 4.0 -> 1.1 s. 1.3–3.8x.
• Not faster: many questions on a short text. 25 or 50 questions on 512 or 2,048 tokens: 1.0–1.1x on the A10G and the L40S (L40S 512 x 50: 672 ms vs 644), 1.1–1.3x on the H100. The branches dominate there and their attention still uses a [branch x row] mask (idea in experiments.md).
• Memory: 32K text 15.5–18.0 GiB reserved (18.5–21.1); 50 questions add at most 2.1 GiB allocated (1.4 before).
• Also measured: images on the L40S (time_images.py): text 73 ms, photo + 1 question 97, photo + 5 questions 116 (136 / 163 / 167); the 8-bit build on the A10G under the 7.2 GiB cap: 558 ms at 2K, 1,767 at 8K, out of memory at 16K as before (reports/engine_2026-10-05/quant_bench; 4-bit from the morning check: 454 / 1,565 / 3,193 ms).
• Published (new columns and rows that show speed and accuracy side by side):
• comparison table (README, docs, website, 5 Hugging Face cards): a second speed column (AI Response Review time for every model, from each model's 2026-09-30 run) and a row "SelfJev-4B Vision, version 0.4.1" (167 s / 68 s on the box that timed 0.4.2); on eval_llm kev-4b (34 s) and jpt-4b (40 s) are faster than our 41 s;
• docs and website comparison pages: a table "Version 0.4.2 against 0.4.1" (both suites end to end, one request on each GPU at 2K and 32K tokens, accuracy on three test sets), from matrix.py --versions;
• website: the homepage chart reads the new sweep and shows each GPU's 0.4.1 time under its bar (fainter bar); the hardware page has the new A10G table, a per-GPU 0.4.1 / 0.4.2 table, the accuracy rows and the quantized table;
• docs/speed.md: the sweep section rewritten with 0.4.1 and 0.4.2 rows; vision card and README.pypi.md image timings; the 4-bit and 8-bit Hugging Face cards' memory and speed tables (and their install line, stale since 0.4.0).
• website install lines said selfjev==0.3.0 (quickstart, API, fine-tuning) and its label v0.3: now ==0.4.2 and v0.4; the Ollama page installs from PyPI. These are hand-written: grep ==0.4.2 and v0.4 in website/ at the next release. README.pypi.md's new timings reach the PyPI page with the next release.
• Checked live (commit a320efe): selfjev.dev (comparison, hardware, homepage chart, quickstart), the docs site (comparison, speed), the eight Hugging Face READMEs; CI and the docs build passed.
• Cost: ≈ $1.63.

2026-10-05 12:00 PDT: selfjev 0.4.2: the faster engine, released; speed updated where it is published (Claude competitors, user request)¶

• Checks before the tag (commit e485511, which holds the engine of the 01:45 entry): e2e 21 / 21 on a g6e.xlarge (fine-tuning jobs: supervised 4.4 min, RLCD 4.0, photos 3.2; 16 concurrent requests 1.2 s; bursts 137 ms per 1K tokens, 214 with 0.4.1); quantized serving under the 7.2 GiB cap on an A10G: 4-bit eval_llm 91.86 vs 91.97 (1 / 0), compact challenge 100 / 100 in 4-bit and 8-bit, 4-bit 16K text 7.2 -> 3.2 s and 6,157 -> 5,694 MiB (reports/engine_2026-10-05/quant). The e2e does not start a quantized server, so that check was added.
• Release: pyproject.toml 0.4.2, tag v0.4.2 = 386c62b (release workflow: build, smoke test, trusted publishing to PyPI). On PyPI at 11:50 PDT; CI and the docs build passed; the five Hugging Face READMEs show 100 s; checked live: selfjev.dev (comparison, hardware note, homepage note) and the docs site (comparison, landscape, speed).
• Speed cell 166 -> 100 s (Text Decisions end to end, 4 in flight; matrix.py wall() now reads reports/engine_2026-10-05/http/eval2/new-run1): matrix.md, public_table.md, card_section.md, README.md, both cards in weights/, docs/comparison.md, website/content/comparison.md, and the five Hugging Face READMEs (publish_comparison.py --push-hf); the text now says only jpt-4b is faster among the 4B models. Prose: landscape, speed, tree_model, experiments, reports/competitors/README.md.
• Not re-measured, still 0.4.1's numbers: the three-GPU latency sweep behind the website's hardware chart and hardware page (A10G, L40S, H100; a note there says 0.4.2 is faster) and the "136 ms text, 163 ms image" lines of the vision card and README.pypi.md (scripts/eval/time_images.py). Re-measuring needs the three boxes again (≈ $2.5). Done the same day: 17:45 entry.
• Cost: ≈ $1.26 (e2e ≈ $0.74, quantized check ≈ $0.52).

2026-10-05 01:45 PDT: TreeServer 40 % faster end to end, same answers; not released yet (Claude competitors, user request)¶

• What: src/selfjev/engine/tree.py rewritten without retraining (the two speed ideas of the 2026-10-04 teardown, S1 and S2). Profiled and A/B-tested on one L40S with scripts/eval/engine_bench.py (new: several versions of tree.py on one loaded model). Report.
• End to end, the competitors' protocol (selfjev serve, Jev's requests, 4 in flight, same box, two passes): eval2 167 / 169 s -> 100 / 101 s; eval_llm 68 / 68 -> 41 / 42 s. On 2026-09-30: jpt-4b 90, kev-4b 109, decider-4b 145, Mica 162. reports/engine_2026-10-05/http/.
• One request (selfjev bench, L40S, p50, 1 question; released = the 2026-09-30 sweep): 512 tokens 145 -> 89 ms, 2K 243 -> 145, 8K 1,173 -> 619, 16K 3,129 -> 1,312, 32K 9,146 -> 2,720 (3.4x); reserved memory at 32K 20.2 -> 15.5 GiB. reports/bench/selfjev4b_flat_l40s.
• Same answers (release adapter, paired with reports/engine_0.4.1/ and reports/images_v1/test): eval2 96.18 vs 96.08 (1 / 3), eval_llm 92.60 vs 92.60 (0 / 0), images 90.46 vs 90.51 (1 / 0), dev benchmark 83.98 vs 84.07 (6 / 3, p = 0.51); through Jev's requests 94.68 vs 94.78 and 90.80 vs 90.59. In process, 3 to 4 of 1,793 decisions differ from the released engine; it differs from itself by 3 to 4 between batch sizes. reports/engine_2026-10-05/{eval2,eval_llm, eval_images_v1,test}.
• The changes (324 eval2 requests in process, 1 / 4 per forward: 82.3 / 85.0 s -> 51.4 / 49.5 s): (1) the text attends causally on the fast kernel, only branch tokens get a mask: 71.3 / 73.0; (2) hidden states are one flat row of real tokens, projections once per token, a yes/no question is one segment: 66.9 / 63.4; (3) DeltaNet: the convolution once over the flat row with the first tokens of each segment corrected, q / k normalized once before the heads are repeated, a level only gathers and runs the kernel: 53.9 / 53.3; (4) outputs written in place, norms without fp32 copies: 51.4 / 49.5.
• What the profile said (it corrected the teardown's estimate): on the released engine the Gated DeltaNet path was 58 % of the time and the MLPs 15 %; building the T x T mask on the CPU 10 %. What is left: the fla kernel (≈ 0.65 ms per call whatever its size, 72 calls per forward), MLPs, attention.
• Did not help: more than ~4 mixed-length requests per forward (16: 52.1 vs 49.5 s; a length-aware batch plan is the next lever); cuDNN refuses the channels-last conv on the box; shifted multiply-adds are no faster than conv1d.
• Tests: tree = standalone sequences (scores and gradients) still holds; the gradient tolerance went from 1e-4 to 5e-4 of the largest gradient (measured 7e-5 to 9e-5 with the new norms); 106 CPU tests pass.
• Not done (needs the user): no commit, no PyPI release; the published speed cell stays 166 s (0.4.1) in matrix.py (wall()), the comparison pages, README and model cards.
• Cost: ≈ $2.66 (g6e.xlarge 1 h 26 min).

2026-10-04 PDT: jpt-4b and decider-4b taken apart: why they are fast, where they are accurate, what we can take (Claude competitors, user request)¶

• What: read both cards and both serving stacks (llm2jev 0.6.1, MIT; decider-ai 1.4.0, Apache-2.0), compared them with our competitor runs by slice, and split our own eval2 time with a fit of our latency sweep. No GPU, $0. Report; numbers from scripts/eval/competitors/workload.py (new).
• Speed is not token count. eval2 through Jev's requests (647 requests, 3,491 questions): our tree 1,128,029 tokens (leaves 13.6 %), jpt-4b's prompts 1,002,614. A fit of reports/bench/selfjev4b_qsweep_l40s (28 cells, median error 5 %) applied to those requests gives 154 s one request at a time (measured 166 s with 4 in flight): fixed 74 ms per request 48 s (31 %), per token 65 s (42 %), dense-mask attention over the text 35 s (23 %), per question 6 s. The 145 ms floor at 512 tokens is the same on an L40S and an H100, so it is not GPU compute. Modeled, not profiled.
• Why they are faster: both read one token per question (softmax over option letters; k options cost nothing extra), both run the text through plain causal attention, jpt on vLLM (continuous batching), decider on CUDA graphs captured at start-up for 89 (batch, length) shapes plus a cache fork for texts of 768+ tokens.
• Accuracy: on our tests we lead both on every type and almost every trap (the exception of size: nota, 90.7 vs jpt 94.9, 2 / 7, n.s.). jpt leads on JevBench hard by family: long_policy 74 vs 53, judge_hard 76 vs 59, probability 90 vs 70 (families of 10 to 19 items); its data has 1,289 long policy documents and program-generated rule arithmetic. decider's specific methods: replay rows trained toward the previous model's own probabilities (v2.1: undid v2's 1-point loss on old tasks, kept the hard-set gain), twins with computed labels (decider/data/rules.py), abstention rows, selection on held-out generated families.
• Ideas added to experiments.md (unclaimed, none approved): S1 causal attention for the text, S2 fixed cost per forward (profile first), S3 one-token readout at Proposed answer:, A1 replay toward the release's own probabilities, A2 generated twins with computed labels.
• Cost: $0.

2026-10-04 PDT: Fastino GLiDE surveyed: a hosted Jev clone on the same System One API (Claude, user request)¶

• What: read docs.fastino.ai (llms-full.txt: GLiDE, primitives, /v1/systemone, models, pricing). Nothing called; no cost.
• Product: POST https://api.fastino.ai/v1/systemone, model fastino/GLiDE, Jev's request shape (state string / object / array, named questions of type noul / choice / score). Docs ship a TypeSafe → GLiDE migration guide and an agent skill for it. Differences from Jev: X-API-Key, Score score is the argmax level index (Jev's float is their expected_level), every instructions must be a non-empty string, extra top-level fields are a 422, retry 425 / 503.
• Limits: 40,000 tokens per rendered question (state + one question), ≤ 255 options / levels, single-label only (no multi-label primitive), one state per request, no streaming, no fine-tuned GLiDE deployments. Confidence: noul |2p − 1|, choice / score top1 − top2.
• Model: "thinking decision model": a fast assessment, then extra reasoning on uncertain questions, folded into the probabilities. Size, base, data, latency not disclosed; hosted only (no weights). Our reading, not stated: a decoder LLM (40K context rules out the DeBERTa family) that reads answer-token probabilities, generating reasoning only on low-margin questions. Their own examples fit: a confident noul reports output_tokens 1; a choice + score request with a 0.5-margin score reports 252. They advise a 300 s read timeout.
• Claim: Decision Index 0.2.1 64.81 vs Jev 57.91 (our landscape table has Jev 57.91 too), all 5 areas, 31 / 38 benchmarks. We have not run Decision Index.
• Price: $0.15 / 1M input tokens, output free (GLiNER-2.5-Decide $0.03).
• Vs ours: selfjev serve already answers the same /v1/systemone shape, so GLiDE is a direct competitor of both Jev and us, and the same harness (scripts/eval/score_systemone.py) can score it on eval2 / eval_llm. Ours has no reasoning step; the parked cascade idea in experiments.md (defer low-margin questions to a stronger scorer) is the same adaptive compute idea. Next (needs the user's OK and a price): GLiDE on eval2 + eval_llm via the API.

2026-10-04 PDT: GLiNER2.5-Decide surveyed (fastino, 340M): how it works and how it differs from ours (Claude, user request)¶

• What: read the model card, configs, the GLiNER2 paper (arXiv 2507.18546) and fastino-ai/GLiNER2 @ 55656fb. Nothing downloaded or run; no cost.
• Architecture: DeBERTa-v3-large (24 layers, 1024 hidden, 340M), fine-tune of gliner2-large-v1 (checkpoint 45500). The config says architecture: span: the GLiNER2 span model, not the 2.5 boundary model the name suggests. Input is one bidirectional sequence ( [P] task ( [L] label1 [L] label2 … ) ) [SEP_TEXT] text; each [L] token's last hidden state goes through an MLP (1024 → 2048 → 1) to one logit. Single-label: softmax, argmax. Multi-label: sigmoid ≥ threshold (0.4 in the card, code default 0.5). Training loss: BCE per label. Several heads share one forward pass.
• Output outside the list: impossible; the answer is labels[argmax], an index into the list given. No abstention: single-label always returns one label, and multi-label returns the top label when none passes the threshold (inference/runtime.py _extract_classification_result), so it never returns an empty set. Add other/none.
• Context: max_position_embeddings 512 (relative positions), paper trains up to 2,048 tokens; classify_text_long chunks at 384 tokens (overlap 64) and max-pools the logits per label.
• Data: not disclosed for Decide. Base GLiNER2: 254,334 examples (135,698 real: news, Wikipedia, law, PubMed, arXiv; 118,636 synthetic), all labelled by GPT-4o. Paper zero-shot classification avg 0.72 (GPT-4o 0.84).
• Their benchmark: fastino/fast-decisions (17 English domains × 300, synthetic-looking, Apache-2.0): Decide 60.2, Decide-1B 59.6, JevK5 57.6, SemIf (Qwen3.5-4B) 56.4. Latency (blog): 38.3 ms V100, 167.3 ms 48-vCPU Xeon at 64 tokens.
• Vs ours: same constrained scoring idea (score only the listed options, no generation); they get it from 12× fewer parameters by using short texts and a bidirectional cross-encoder. Closest measured analogue on our sets: Laya (0.4B ModernBERT) 45.4 on eval2 vs our 94.7 (reports/competitors/matrix.md). Expected, not measured: Decide well below us on eval2 / eval_llm (long, trap-heavy texts, many questions per state, images).
• Next (open idea, needs a GPU run and the user's OK): score Decide on eval2 / eval_llm, and selfjev-4b on fast-decisions (5,100 rows), for the first two-way comparison.

2026-10-02 17:30 PDT: 0.4.1 speed vs the competitors, its accuracy unchanged, and a fine-tune on 300 photos of a new task (Claude images, user request)¶

• Speed (the competitors' protocol: selfjev serve merged, eval2 through Jev's request shape, 4 in flight, one L40S g6e.xlarge): 166 s and 167 s on two passes, 242 s before the batching fix (JOURNAL 2026-10-01 15:50). Still behind jpt-4b 90, kev-4b 109, decider-4b 145, level with Mica 162; ahead of Plumb 471, imajev 489, openjev 753. reports/competitors/eval2/selfjev-4b-vision-api-w4-v041; matrix.py reads it; the speed cell is updated in matrix.md, public_table.md, card_section.md, README.md, the two model cards in weights/, docs/comparison.md, docs/landscape.md and website/content/comparison.md (matrix.md not regenerated whole: it would add the unpromoted selfjev-verdict-json* rows of another session).
• The 0.4.1 engine (DeltaNet batch buckets, q / k l2norm in torch) scores like the release, paired against reports/images_v1/: eval2 96.08 vs 96.13 (1 / 2, p = 1), eval_llm 92.60 vs 92.49 (1 / 0), images 90.51 vs 90.41 (2 / 0); through Jev's requests 94.78 vs 94.68 (2 / 0). reports/engine_0.4.1/.
• Image fine-tuning demo (table): 300 satellite photos of hurricane damage (150 per class; scripts/data/hurricane_photos.py, rows the frozen test does not use, none byte-equal to a test photo) through the API flow (scripts/eval/image_finetune_demo.py: folder -> upload_file(rows) -> job -> wait_fine_tuning_job). Frozen test, hurricane (100 q): release 61.0 -> 79.0 after 1 epoch (2 optimizer steps, 1.8 min; 26 fixed / 8 broken, p = 0.003) -> 86.0 after 5 epochs (10 steps, 2.8 min; p = 2e-5). The other 9 image tests 92.1 -> 91.9 / 91.6 (no family significant; -2 on eurosat, fashion, snacks at 5 epochs); eval2 96.1 unchanged.
• Change from it: one epoch on a small file is 2 steps, so epochs now defaults to enough for 10 optimizer steps (training/finetune.MIN_STEPS, at most 10 epochs; large files still 1); the API's epochs is optional. On master, not on PyPI yet.
• Cost: ≈ $1.96 (g6e.xlarge 1.05 h).

2026-10-01 15:50 PDT: batched requests no longer stall; e2e 21 / 21; selfjev 0.4.0 on PyPI (Claude images, user request)¶

• Cause (profiled on a g5.2xlarge, replaying the e2e's 20 bursts in process from a cold Triton cache): fla's in-kernel q / k l2norm autotunes per input size (constexpr NB = rows / 65,536, rows = sequences × length × heads), so a batch whose texts add up to a new row count waits for a new tuning. 32-request bursts: 19–29 s with the kernel, 10–11 s (their compute) with the norm in torch; smaller bursts within 5 %; warm_kernels 80 -> 42 s. One request never stalled (its rows stay under one block). Fix 6b43d94: engine.tree.l2norm (same math: fp32, eps 1e-6). Gotcha: fla stores autotune results in Triton's disk cache (cache_results), so a probe in a process that already served traffic finds them tuned: my first l2norm probe on the e2e box (14:10 entry) wrongly cleared it.
• e2e of 6b43d94 (report, g6e.2xlarge): 21 / 21. 16 concurrent requests 1.4 s (9.5 s, 19 s before); bursts over 4K tokens 214 ms per 1K tokens, slowest 261 (the check: < 3× the median); 8 photos at once 1.0 s (2.3 s); jobs supervised 5.4 min, RLCD 5.4, photos 2.3; the photo job on a random 150 beans photos (3 classes) + 50 text rows: held-out beans 30 / 32 -> 31 / 32.
• Release: selfjev 0.4.0 on PyPI (tag v0.4.0, f2cc778, release workflow: build, smoke test, trusted publishing): everything since 0.3.0 (Ollama engine, quantized serving, row uploads, wait_fine_tuning_job, the image fixes) but not this stall fix, which is on master (6b43d94) for the next release. 0.4.1 (2026-10-02, tag v0.4.1, 034e938, user request): the stall fix, on PyPI.
• Cost: ≈ $1.57 (profiling ≈ $0.60, e2e ≈ $0.97); the whole task ≈ $3.68 of the ≈ $4.75 approved.

2026-10-01 14:10 PDT: SDK, server and fine-tuning end to end on a GPU, text and images: 40 of 41 checks; one open stall (Claude images, user request)¶

• Why: the user asked to make sure the SDK, the endpoint and fine-tuning work well on text and images.
• Fixed (92ba607, 49075ef, a666743): upload_file takes rows, images as Path / bytes / PIL, each row checked before sending; wait_fine_tuning_job (sync, async); photos read upright (EXIF orientation: phone photos reached the model sideways); an image the processor refuses is a 422, not a 500; a training file with an unreadable image is a 422 naming the line at upload (it failed an hour into the job), checked off the event loop; selfjev deploy aws leaked the key pair when a launch had no capacity; fine-tunes from --init (every job) default to lr 5e-5 (continuing the release at 1e-4 lowered it in the chain pilot; images v1 used 5e-5). Docs: docs/api.md, README.pypi.md.
• scripts/aws/e2e.py now also checks images (photos sent as files, text + image parts, a bad image, 8 concurrent, the async client), a fine-tune on a folder of photos plus text rows, and bursts of mixed traffic (each must end < 5 s).
• Run 1 (49075ef, report): 20 / 20. Ticket answered right (refund 0.993, spam 0.011, team billing); photos 45 / 48 right on 24 held-out images v1 photos, 305 ms per request through the SDK from California; 8 photos at once 2.3 s; jobs: supervised 8.1 min, RLCD 5.0 min, photos 2.6 min.
• Run 2 (a666743, report): 20 / 21; the new burst check failed (below). The photo job's 32 / 32 -> 29 / 32 on held-out beans (both runs) was the test's data, not training: the first 150 beans photos of the file are one class, so the model learnt "always angular leaf spot" (it did: validation loss 0.078 -> 0.018). The script now samples photos at random.
• Open: batched requests are slow. One request ≈ 0.2 s on the server; 16 at once 9.5 s (19 s in run 1), bursts of 8 ≈ 1–1.7 s, one burst 13.9 s. Not GPU kernels: fla autotunes per batch size (0.4–1.1 s per new size, measured), now padded to powers of 2 and tuned at start (bucketed_rule, warm_kernels: a first single request 1.5 s -> 0.3 s, start-up +≈ 25 s), but bursts stayed slow; fla's l2norm, depthwise conv1d and masked SDPA at batch shapes all run in ms on the box, and packing 32 trees takes 13 ms (CPU). Next: profile TreeServer.score_requests on a GPU (merged and unmerged LoRA; the e2e server is unmerged because of --fine-tuning).
• Cost: ≈ $2.11 (two g6e.2xlarge runs). Verdict: SDK, server, images and fine-tuning over HTTP work end to end; concurrency is the one gap.

2026-10-01 15:15 PDT: verdict / JSON-only retrain: the public benchmarks move, our suites dip slightly (n.s.); not promoted yet (Claude competitors, user request)¶

• Why: separate the two batches of the 07:00 retrain: same recipe, but only verdict_json_v1 (no typed-decisions import).
• Run: scripts/data/build_verdict_mix.py verdict_json_only_v1 verdict_json_v1 (3,697 new + 3,697 text replay + 3,000 images = 10,384 q); scripts/train/verdict_json_box.sh verdict_json_only_v1, 257 steps on one A10G (g5.2xlarge, ≈ 31 s/step; no g6e capacity). The first try ran out of memory at 16K tokens in the backward pass (24 GB; 1.4 GB reserved but fragmented); PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True fixed it. Validation 92.9 -> 93.0 -> 93.2 (confidently wrong 28 -> 35). Adapter runs/verdict_json_only_v1/run/adapter (not in git). Reports reports/verdict_json_only_v1/, logs in reports/verdict_json_only_v1/logs/.
• Paired vs the vision release (McNemar; only-new-right minus only-old-right):

Jev on the same sets: eval2 97.2, eval_llm 92.5, JevBench 86.6 (hard 73.0), typed-decisions 72.7. - Verdict: the typed import was the main cost (via Jev requests −21 -> −9 on eval_llm). The verdict / JSON data alone transfers to both public benchmarks (JevBench ties Jev's 86.6; typed-decisions +4.6 out of distribution), but every one of our six sets still moves down 0.3 to 1.0 point (none significant alone). Not promoted; weights/selfjev_4b_vision stays the default. User decision (15:30 PDT): stop this line here; the adapter stays local at runs/verdict_json_only_v1/run/adapter in case the public benchmarks are shown again. Cost: ≈ $4.60 (box).

2026-10-01 07:00 PDT: retrain on the verdict / JSON + typed-decisions batches: not better, not promoted (Claude competitors, user request)¶

• Run: selfjev finetune --init weights/selfjev_4b_vision --lr 5e-5 --soft-weight 0.5 --max-length 16384 --batch-tokens 16384 --grad-accum 2, 1 epoch, 426 steps (17 s/step) on one L40S, mix runs/verdict_json_v1 (7,802 new + 7,802 text replay + 3,000 images; scripts/train/verdict_json_box.sh). Validation 89.2 -> 93.0 (best, step 300) -> 92.85 (last); confidently wrong 54 -> 29. Adapter runs/verdict_json_v1/run/adapter (not in git). Reports reports/verdict_json_v1/.
• Paired vs the vision release (McNemar): eval2 95.58 vs 96.13 (14 / 25, p = 0.11); eval_llm 91.54 vs 92.49 (9 / 18, p = 0.12); dev benchmark 83.64 vs 84.07 (p = 0.086); images 90.36 vs 90.41 (p = 1). Through Jev's request shape (as the competitors): eval2 93.97 vs 94.73 (15 / 30, p = 0.036), eval_llm 88.48 vs 90.70 (9 / 30, p = 0.0011).
• Diagnostics: JevBench public-231 84.8 vs 83.5 (hard 69.4 vs 66.7; 9 / 6, p = 0.61); typed-decisions 78.5 vs 66.1 (in distribution now: its train split is in the data).
• Verdict: the targeted data moves the targeted benchmarks (typed-decisions a lot, JevBench hard +3 items, not significant) and costs our own suites, significantly through Jev requests. Not promoted; weights/selfjev_4b_vision stays the default. Likely culprit: the typed-decisions import (≈ 4B-teacher labels, 2-way noul choices, 4,105 of the 7,802 new rows). A verdict / JSON-only retrain would separate the two (≈ $5). Cost: ≈ $5.40 (box).

2026-10-01 04:05 PDT: two training batches: planted wrong verdicts + JSON records (Luna), and typed-decisions train re-judged (Claude competitors, user request)¶

• Why: the public-benchmark diagnosis (JOURNAL 2026-09-30, reports/competitors/README.md): 78 % of our JevBench hard misses are the planted "surface answer" (a wrong verdict written into the text), and the typed-decisions gap is mostly invoice JSON records whose fields must be compared. Test-diagnosis-motivated; writers saw only abstract trap definitions.
• verdict_json_v1: two new traps in data/hardcases/BRIEF.md and gen_hardcases.py (planted_verdict: a person's or system's verdict the facts contradict, right in about a third; json_record: the state is JSON records to reconcile, serialized as the server does). gen_hardcases.py --batch verdict_json_v1 --model openai/gpt-6-luna --openai --round3 --traps planted_verdict,json_record: 1,917 texts, 4,434 questions; Astra agreement 95.9 % (json 96.0, verdict 95.8); strict build 4,095 kept; moderation 1 flagged (a forklift insurance claim, kept).
• typed_decisions_train_v1: LocalLLaMA/typed-decisions train split (Apache-2.0, rev d0e2f0c4; test never read) via scripts/data/import_typed_decisions.py through the server's Jev mapping; its labels are a ≈ 4B teacher's, so Astra re-judged all 6,000: 76.7 % agreement (invoice 72.5), 4,599 kept with a NON-strict build (agreeing questions of a text kept). One Astra batch stuck at 998/1,000 for 2.5 h was cancelled; its answers were still delivered. Typed-decisions is in distribution for us from now on.
• data/all.jsonl.gz: 103,224 questions. scripts/train/jev_soft_targets.py: rows without a Jev answer (3) train on the label only. Mix (scripts/data/build_verdict_mix.py): 7,802 new + 7,802 replayed text (a third of the > 4K texts) + 3,000 replayed images = 18,604; val 1,200 text + image val + 892 new.
• Cost: $30.72 (quoted ≈ $32). Next: continued fine-tune from weights/selfjev_4b_vision (lr 5e-5, soft 0.5, 1 epoch) on AWS, then the usual evals + the two public benchmarks as diagnostics (scripts/train/verdict_json_box.sh).

2026-10-01 02:40 PDT: chain layout (a verdict line per option, one branch per question): no gain, dead end (Claude images)¶

• What: layout "chain" (Qwen35Scorer.layout, tree.segments, --layout chain, selfjev.json next to the adapter): a choice / select-all question becomes one branch with a line per option, each line read where it ends and seeing the lines before it (one forward pass, as today; binary questions unchanged). Motivated by a test-set diagnosis (scripts/eval/jev_gap.py -> reports/audit_2026-09-30/jev_gap.md: vs Jev, multi_positive 4 / 21, p = 0.0009, most of the eval2 gap). Exactness: tests/engine/test_tree.py::test_chain_layout_matches_full_sequences.
• Pilot (paired): both arms from the text release on the same 15,000 questions (scripts/train/sample_text_mix.py 15000 0.333, sha256 2fc73d91…), lr 1e-4, 1 epoch, same seed; only the layout differs. Leaves' best-by-validation was step 0 (the release itself), so both arms are compared at their trained checkpoints (leaves step 464, 2609f99d…; chain best = step 450/457, e3f9dd0d…). Reports: reports/chain_pilot/{leaves,chain}/.

Order: reversing the options changes the chain's answer on 2.6% of eval2's choice / select-all questions. Training: 6% slower per step despite fewer tokens (the DeltaNet state goes line by line). - Verdict: dead end. No gain on select-all (the target), slightly worse elsewhere, order-sensitive. Also: continuing the release at lr 1e-4 on a 15K slice lowers it (both arms below the release; leaves' best checkpoint was step 0). - Incidents: the leaves box's driver evaluated step 0 (best-by-validation); I swapped it to step 464 on the box, a pkill pattern matched my own ssh command and killed one eval, and the box was terminated before its rerun finished, so its eval2 / eval_llm were rescored on the chain box (same adapter, sha checked). Cost ≈ $10.50. - Code removed (user request, 2026-10-01): the layout was dropped from the engine, CLI and tests; it remains in git history.

2026-10-01 01:25 PDT: selfjev-4b-vision on Ollama: GGUF release + --engine ollama (Claude Ollama, user request)¶

• What: a user asked for an Ollama model or plugin. Built selfjev serve --engine ollama (src/selfjev/engine/ollama.py, no torch, pip install "selfjev[ollama]"): the challenger-state-first-v1 prompt as a raw /api/generate call per option, one token, top-20 logprobs, z = logprob(yes) - logprob(no); Ollama's prefix cache shares the state. Images go as [img-N] + images. Server verified end to end (text and image requests) on the box. Docs: docs/ollama.md.
• Model: merged selfjev-4b-vision-merged -> llama.cpp convert_hf_to_gguf.py (bf16, plus --mmproj) -> llama-quantize. Published Jwuthrich/selfjev-4b-vision-GGUF: Q8_0 4.6 GB, Q4_K_M 2.8 GB, mmproj. ollama pull hf.co/Jwuthrich/selfjev-4b-vision-GGUF:Q4_K_M shows vision. Ollama 0.35's own safetensors import needs MLX and failed on Linux, so the GGUFs come from llama.cpp. Two FROM lines (model + mmproj) in a Modelfile give the projector.
• Numbers (selfjev eval --engine ollama, data/ova/ files, reports/ollama/): eval2 bf16 95.63 / Q8_0 95.73 / Q4_K_M 95.68 (tree engine 96.13); eval_llm 92.71 / 92.71 / 91.86 (tree 92.49); images Q8_0 89.41 (tree 90.41). Quantization costs ≈ 0 (Q4 -0.8 on eval_llm); the ≈ -0.5 to -1.0 vs the tree engine is there at bf16 too: the path, not the quantization. Cause not isolated; one known difference: the trained prompt tokenizes the space after "Question:" alone, Ollama merges it into the next word.
• Mistake caught: the first sweep used data/eval2.jsonl and data/eval_llm.jsonl (no option list in the question), not the data/ova/ files the reference scores use. Kept as reports/ollama/raw-* (95.1-95.3, 91.8-92.0); the table uses ova-*.
• Incident: five parallel Python evals on the 4 vCPU / 30 GB box made sshd stop answering (TCP up, no banner) for 20+ min, rebooted from AWS, which dropped the image eval and the shutdown cap (re-armed by hand); re-ran with three, then one process. Keep a 4-vCPU box to <= 3 client processes.
• Not done: bf16 and Q4 on images; speed against the tree engine (one smoke run: 0.2 s per question); the ollama.com library push (needs the user's Ollama account key: ollama create from the two GGUFs, then ollama push).
• Token: a failed upload printed the HF token into this session's tool output once; not written anywhere in the repo.

2026-10-01 PDT: pre-quantized 4-bit and 8-bit checkpoints on Hugging Face, and the website (Claude quant, user request)¶

• Published: Jwuthrich/selfjev-4b-vision-4bit (3.3 GB) and -8bit (4.8 GB), public, with model cards. Anonymous check: both public, model.safetensors sha256 equals the local file (runs/hf-quant/SHA256SUMS). Built on a g5.xlarge by selfjev quantize --adapter weights/selfjev_4b_vision --bits 4bit|8bit, then reloaded with selfjev eval --quantized-model DIR under the 7.2 GiB cap: compact challenge 100 / 100, 4-bit eval_llm 92.0 (= the on-the-fly build's 92.0): reports/quant/prequantized/. Not re-scored on eval2 after saving (the on-the-fly numbers stand).
• Code: selfjev quantize, --quantized-model REPO_OR_DIR on serve / classify / eval / bench (no bf16 base download, no merge). Not on PyPI yet (0.3.0 has no --quantize): the cards say to install from GitHub master until the next release.
• Website: the hardware page has an 8 GB section with the quality and memory tables (the old "Why not an 8 GB machine?" section is replaced), and the Hugging Face panel lists both builds with Decision Bench scores. Peak memory is in GiB (MiB / 1024); the first journal entry's 4.6 / 6.2 were MiB / 1000.
• Process: my first CLI edit lost quantized_model in TreeServer(...) (the call had been reformatted), so the first two verification runs loaded bf16 and failed; merge_and_unload and a None revision in the report header failed next. Fixed, re-verified. The 8 GB claim for 4-bit rests on the allocator cap, not on a physical 4060.
• Cost: g5.xlarge 07:32–08:15 UTC (0.73 h) ≈ $0.74 + transfer; this task's total with the first runs ≈ $3.1 of the $4.

2026-10-01 PDT: 8-bit and 4-bit selfjev-4b-vision for an 8 GB GPU (Claude quant, user request SELA-003)¶

• Why: a user with an RTX 4060 (8 GB) asked for a quantized build and its accuracy next to the full model. The bf16 merged weights are ≈ 9 GB, so they cannot load on 8 GB.
• What: --quantize 8bit|4bit on selfjev eval|serve|bench (engine/qwen35.py: merged_checkpoint + load_quantized, bitsandbytes LLM.int8 / NF4; extra quant). The adapter is merged in bf16 first (merging into quantized weights is lossy), written once to ~/.selfjev/quantized and reloaded quantized. 8 GB simulated on an A10G by capping the allocator at 7.2 GiB. Summary table: reports/quant/summary.md; raw reports reports/quant/{4bit,8bit,bf16_compact_challenge_v1}.
• Accuracy (Decision Bench, 3,657 questions): bf16 95.9 pooled, 8-bit 95.3, 4-bit 95.0. eval2 96.1 / 95.2 / 94.7, eval_llm 92.5 / 92.0 / 92.0, compact challenge 100 / 100 / 100 (bf16 compact rerun on the A10G). 4-bit costs 1.4 points on eval2, mostly multilabel exact match (90.8 bf16 → 86.1).
• Memory: the 4-bit model peaks at 3.5 GiB (2K-token text), 4.5 (8K), 6.0 (16K): fits 8 GB. The 8-bit model peaks at 4.9 / 5.9 GiB and runs out of memory at 16K tokens and on a long eval2 text under the 7.2 GiB cap; its 95.2 on eval2 comes from a rerun with the cap lifted. Latency (A10G, 2K text, 1 question, p50): 4-bit 604 ms (bf16 584), 8-bit 824 ms.
• Verdict: 4-bit (NF4) is the build for an 8 GB card: ≈ 1 point below bf16 pooled, same speed on an A10G, texts up to 16K tokens. 8-bit is 0.3 points better but needs ≈ 10 GB for long texts. Not yet published to Hugging Face; the first start needs the 8 GB bf16 download plus a CPU-side merge and 16 GB host RAM. A pre-quantized checkpoint repo would remove both (open idea).
• Process notes: aws_launch.sh rejects fractional hours (user data arithmetic), so the shutdown cap was set by hand over SSH (shutdown -h +113); the first loads thrashed a 16 GB host because the bf16 CPU copy stayed alive during the quantized load (fixed: freed first); two relaunch races duplicated runs on the 8-bit box and cost ≈ 25 min.
• Cost: g5.xlarge × 2: 4-bit box 05:57–06:44 UTC (0.8 h), 8-bit box 05:57–07:25 UTC (1.5 h) ≈ $2.3 (approved ≤ $4). Boxes terminated, SGs and key pairs deleted.

2026-10-01 PDT: 200 × t4g.nano download both HF repos, round 2 (Claude, user request)¶

200 t4g.nano (us-east-1) each downloaded every file of Jwuthrich/selfjev-4b and Jwuthrich/selfjev-4b-vision (33 files, 461 MB) over HTTPS. Final batch hf-dl-test4: 192 of 200 OK in 5.8 to 9.0 s each; 5 failed with Connection reset by peer (no retry in the script); 3 had no console line when read. Two earlier batches (hf-dl-test2, hf-dl-test3, 200 each) failed with No space left on device because /tmp is RAM-backed on AL2023; fixed by using /var/tmp. Tested HF's handling of about 200 parallel clients: fine apart from the resets. Cost about $0.0042/h × 800 box-runs × under 7 min, under $0.40. All 700 instances (including the first 100) are terminated; no SG or key pair created.

2026-09-30 21:30 PDT: can JevBench public / hard and typed-decisions train selfjev? (Claude images, user question)¶

• JevBench public (fstandhartinger/jevbench @ bb05a33, datasets/public): 231 items, original 72 (MIT, rubric reviewed) + easy 48 + hard 111 ("authored with rationale; cross-model review before any system ran"). The board ranks on sealed items too (24 held-out private, 146 imported, held-out hard); training on public items is allowed with disclosure (their smalljev note). Ours (reports/competitors/jevbench/selfjev-4b-vision/tiers.txt): easy 100, original 98.6, hard 66.7; weakest hard families temporal_numeric 0.333, long_policy 0.526, judge_hard 0.588, tradeoff 0.5 (small n each). Not for training: 231 items are 0.3% of our data, and training on them would spoil the clean public number while the sealed board score is what ranks. Use as a diagnostic: its weak families match eval2's (temporal, numeric, judging).
• typed-decisions (LocalLLaMA/typed-decisions, Apache-2.0): train 1,200 cases x 5 questions = 6,000 decisions (customer service, invoices, security incidents, agent traces; noul / choice / score, no select-all), test 400 cases. Gold = "the mean of three samples from a teacher of roughly 4B-class capability"; teacher self-agreement 0.735 is its ceiling. Ours zero-shot 66.1 (Jev 72.7; score questions 60.9). Usable only re-labelled: the train split is legitimate, but its labels are a weak model's, which AGENTS.md rules out as training targets; a verified batch (our Astra judge, ~$20-35) would be allowed. Effect: large on typed-decisions itself (in-distribution; the card then lists us under "fine-tuned on train"), little expected on eval2's gap (select-all questions, distractors), which it lacks.

2026-09-30 PDT: 100 × t4g.nano HF download smoke test (Claude, user request)¶

100 t4g.nano (us-east-1, tag Name=hf-dl-test) each curl-downloaded the 4 files of Jwuthrich/selfjev-4b-vision (adapter, about 230 MB) and shut themselves down (terminate-on-shutdown). All 100 were terminated within 100 s of launch. Console output was empty, so per-box download success is unverified. Cost about $0.42/h × under 2 min, under $0.05. No SG or key pair created; nothing left running.

2026-09-30 19:50 PDT: images v2 (diverse VQA data): +0.8 on unseen image tasks (not significant), costs text: not adopted (Claude images)¶

• Run: from the vision release (images v1), lr 5e-5, 1 epoch, 575 steps (≈ 3 h, one L40S), on runs/images_v2 (scripts/data/build_images_v2.py): 15,991 new image questions (VQAv2 8,000: 3,200 yes/no + counts + typed answers; A-OKVQA 4,991; COCO "which can be seen" multilabel 3,000) + 3,000 images-v1 rows + 12,000 replayed text questions with a third of the texts over 4K tokens. Validation 89.2 -> 90.7% (loss 0.207 -> 0.182, best = last step). Reports: reports/images_v2/; adapter runs/images_v2/adapter (not in git).
• Unseen image tasks: held-out QA test (eval_images_heldout, 1,027 q) text-only 88.3, vision v1 88.7, v2 89.5 (vs v1 24 / 16, p = 0.27; vs text-only 26 / 14, p = 0.08): AI2D 84.0 -> 86.7, CV-Bench 88.7 -> 90.7, ChartQA, VizWiz, RealWorldQA, DocVQA flat; v1's 4 held-out sets 84.3 -> 85.1 (19 / 13, p = 0.38). Pooled +43 / -29: a lean, not proof. The vision default itself is no better than text-only there (88.3 -> 88.7, p = 0.59): v1 learnt no transfer.
• Trained tasks kept: v1's 6 datasets 94.2 -> 94.5.
• Text gate fails vs the default: eval2 96.13 -> 95.23 (11 / 29, p = 0.006), dev benchmark 84.07 -> 83.55 (29 / 47, p = 0.05), eval_llm 92.49 -> 92.39 (=). Losses are hard / very hard text questions (26 of 29), mostly binary traps; eval2's > 4K-token slice is fine (97.3 vs 97.8), so the shortened long-text replay is not the cause. Likely: less text than image (12K vs 18K, v1 was 1:1) and a second fine-tune stacked on v1.
• Not adopted; the default stays the vision release (images v1). Also found: TextVQA and DocVQA are at ceiling in the held-out test (wrong options from other images are easy to rule out): a harder version would take wrong options from text in the same image. Run incidents: this Mac's public IP changed mid-upload (the driver now follows it in the SSH rule); 2 of 5,000 A-OKVQA rows had an empty option (the builder now skips them).

2026-09-30 16:35 PDT: tokenizer fix (quadratic in options) and DecisionBench retry, stopped again (Claude competitors, user request)¶

• Fix: src/selfjev/engine/qwen35.py entry() tokenized instruction + "\nProposed answer: " + answer + tail once per candidate, and the instruction lists every option (server option transform): O(N^2). Now head = tokens(instruction + "\nProposed answer:") once and head + tokens(" " + answer + tail) per candidate; ":" always ends a pre-token, so ids are identical (0 mismatches on 54,681 real questions from eval2, eval_llm, hf, the image set and 30K training rows; new test test_entry_branches_equal_full_string_tokenization). tree.split_branches uses os.path.commonprefix. CPU per question: 64 options 82.8 -> 4.2 ms, 255 options 1,414.6 -> 28.0 ms (50x). Training ids unchanged (same path).
• DecisionBench retry (g6e.2xlarge, 51 min, ≈ $1.93): GPU now at 100 % instead of idle, answers equal to the pre-fix run on 2,903 / 2,906 rows (batching numerics). Stopped at 3,460 / 23,900 rows: the tree engine does ≈ 21 rows/min on many-option tasks (route 113 options on average, tool_route 64, canonical_entity 60), ≈ 7 h more on an L40S, beyond the approval. 65 / 637 canonical_entity rows exceed 32,768 tokens (options in the question and in every leaf). Partial rows kept for a resume; per-task numbers in reports/competitors/README.md.
• Verdict: tokenization is no longer the bottleneck; the tree engine's GPU speed on 60–120-option questions is. A full DecisionBench needs ≈ 7 h L40S (≈ $16) or a faster engine/GPU: the user's call.
• Dropped (user, 2026-09-30 16:45 PDT): no further DecisionBench runs.

2026-09-30 15:45 PDT: a held-out image generalization test, and where image fine-tuning time goes (Claude images)¶

• Generalization test data/ova/eval_images_heldout.jsonl (gitignored, 160 MB, sha256 7865bfc8…, scripts/data/build_eval_images_heldout.py): 1,027 questions over 842 images from 7 question-answering sets no training mix may use, photos and documents, human labels: TextVQA 150, VizWiz 127, DocVQA 150, ChartQA 150, RealWorldQA 150, AI2D 150, CV-Bench 150 (COCO images removed). Open answers become a choice among the answer and 3 wrong ones of the same shape; numbers are x0.5 / x0.75 / x1.25 / x1.5, never within 5% and never over 100 for a percentage (a first build had easy distractors: a title for a date question, 166 for a percentage; rebuilt). Images up to 1,024 px (documents need them). Their sources overlap none of the planned v2 training sources (COCO / Visual Genome). No UI-screenshot set yet (ScreenQA is not on the Hub). Together with the 4 held-out sets of eval_images_v1 this is the number to raise.
• Training speed (measured on images v1's mix): preparing an image is ~5 ms, 0.05 s of a 17 s step; the replayed text is ~95% of the tokens (26K text vs 1.2K image tokens per step), and texts over 4K tokens are 11.5% of the replayed questions but 64% of their tokens. So the lever is the replay, not image code: scripts/data/build_images_v1.py LONG_FRAC keeps that share of the long texts (⅓: ~1.7x faster). eval2 has 9.3% of its questions over 4K tokens: check that slice when using it. (I had guessed image prep was the bottleneck; it is not.)
• v2 sources checked (stream fine): A-OKVQA train (4 human options), VQAv2 (lmms-lab mirror, validation), GQA (train_balanced_instructions), COCO 2017 objects (detection-datasets/coco). Not scored on a GPU yet; cost $0.

2026-09-30 14:15 PDT: head-to-head vs 8 open Jev-like models on one L40S, and selfjev-4b-vision on the shared public benchmarks (Claude competitors, user request)¶

• What: each competitor served with its card's recipe (pinned revisions), asked exactly Jev's requests on our frozen sets (scripts/eval/score_systemone.py, failures count as wrong); public sets through their own harnesses (JevBench runner, typed-decisions, lev's Nimble export). Protocol, recipes and caveats: reports/competitors/README.md; table generated by scripts/eval/competitors/matrix.py -> reports/competitors/matrix.md, embedded in landscape.md.
• Our frozen sets (same requests): selfjev-4b-vision eval2 94.7 / eval_llm 90.7 vs the 4B field: Plumb-4B 93.4 / 84.1, jpt-4b 92.5 / 85.0, imajev-4b 91.5 / 84.9, decider-4b 90.8 / 82.8, Mica 89.6 / 84.1, kev-4b 88.4 / 74.7, Laya 45.4 / 46.4. Every paired test vs a 4B favours us (closest Plumb on eval2: 79 / 52, p = 0.023). openjev-27B (FP8, non-commercial) beats us on eval2 (96.8, 30 / 72, p = 4e-5), level on eval_llm (92.6, p = 0.063). Jev: 97.2 / 92.5.
• Jev's request shape costs us 1.4–1.8 points vs our native evaluation (eval2 96.1 -> 94.7): multilabel as one noul per candidate (86.1 vs 90.8). Images through the API = native (90.6 vs 90.4, p = 0.73).
• Shared public benchmarks (zero-shot; 0 overlap with our data): JevBench public-231 83.5 (hard 66.7) vs Plumb 89.6, jpt 87.9, openjev 87.4, imajev 85.7, decider 83.1, Mica 82.7, kev 75.8; Jev 86.6 (board). typed-decisions 66.1 vs openjev 71.1, imajev 68.5, decider / Mica 68.0, kev 67.0, Plumb 61.4, Laya 36.0; Jev 72.7 (jpt 78.8 is in distribution). Nimble 13 macro 74.8 (Jev 76.0 and Nimble-9B 74.8 on their own runs). Mid-pack, not first.
• Images: on the 4 held-out datasets nobody trained on, 85.1 vs imajev 85.2, jpt 83.8, openjev 82.6 (a tie); our 94.0 on the other 6 is in distribution.
• Speed: eval2 end to end with 4 requests in flight: ours 242 s vs jpt 90, kev 109, decider 145, Mica 162, Plumb 471, imajev 489, openjev 753 (Laya 23).
• Harness checks: every card number we re-measured matched (JevBench: decider 83.1 vs 82.7, kev 75.8 = 75.8, Mica 82.7 vs 83.1, Plumb 89.6 = 89.6, jpt 87.9 = 87.9; typed-decisions: Laya 36.0 vs 36.2).
• DecisionBench stopped, no result: 2,925 / 23,900 rows; the server was CPU-bound tokenizing (qwen35.py entry() re-tokenizes the question + full option list once per candidate: quadratic), ≈ 11 rows/min on many-candidate tasks. Follow-up task suggested to fix it. Image JevBench public items are not downloadable (only its maintainers can score).
• Cost: ≈ $7.35 (g6e.xlarge 3 h 53 min). Verdict: best open 4B on our own sets and tied on held-out images, but mid-pack on the shared public benchmarks and slower than most 4B peers; the public "best" claim is not supported. Next: fix the tokenization, then DecisionBench + a JevBench board request (outward: user's OK first).

2026-09-30 10:35 PDT: many questions on one document: selfjev-4b latency, memory and throughput on A10G, L40S and H100 (Claude qsweep, user request)¶

• Why: a reviewer asked how latency, memory and throughput scale with the number of typed decisions on one document (1, 5, 10, 25, 50 on a 24 GB GPU). The current model had no NVIDIA timing on its own engine (TreeServer).
• Run: selfjev bench --lengths 512,2048,8192,16384,32000 --questions 1,5,10,25,50 (3 options per question, synthetic text, one request at a time, in process), weights/selfjev_4b_vision merged, bf16, on g5.xlarge (A10G 24 GB), g6e.xlarge (L40S) and p5.4xlarge spot (H100) in parallel. Summary: reports/bench/selfjev4b_qsweep_summary.md; raw: reports/bench/selfjev4b_qsweep_{a10g,l40s,h100}/.
• Memory is flat in questions, set by document length: 1 → 50 questions adds ≤ 1.4 GiB allocated (weights 8 GiB); 32K tokens peak at 14.2–15.0 GiB allocated, 18.5–21.1 reserved. All 25 cells fit on the 24 GB A10G (22.5 GiB usable), no OOM. Identical on the three GPUs.
• Latency, 1 → 50 questions (p50 ms): 2K text: A10G 584 → 1,945, L40S 243 → 894, H100 166 → 404; 32K text: A10G 22,019 → 27,760, L40S 9,146 → 11,544, H100 4,033 → 5,206. 50 questions cost 1.3–6.1× one; each extra question 3–24 ms (H100), 10–49 (L40S), 25–117 (A10G). One 50-question request is 8–40× faster than 50 one-question requests.
• Throughput within one request (2K text, 1 → 50 questions): A10G 1.7 → 25.7 decisions/s, L40S 4.1 → 55.9, H100 6.0 → 124.
• Fixes on the way: (1) on H100, PyTorch picks cuDNN attention, which failed to load (CUDNN_STATUS_SUBLIBRARY_LOADING_FAILED) and crashed the first request: TreeServer now calls torch.backends.cuda.enable_cudnn_sdp(False) (A10G/L40S never pick it). selfjev serve on an H100 would have hit this. (2) selfjev bench records an OOM cell instead of crashing, adds cuda_peak_reserved_mb, stops filling state_encodes / state_tokens_encoded with pair-count placeholders (nulled in the A10G/L40S JSON, which ran before that fix) and no longer crashes writing bench.md (max_batch_size missing from TreeServer's meta; the files were regenerated from bench.json). (3) scripts/aws/aws_launch.sh takes SPOT=1.
• Limits: synthetic text; one request at a time (concurrent batching not measured); 32K exceeds the 16K training length (timed, accuracy not validated); A10G 32K cells have 3 samples. Cold start 17–52 s load + 39–75 s first request.
• Cost: ≈ $2.15. Verdict: the shared-prefix tree makes extra questions on one document nearly free in memory and cheap in time; a 24 GB card serves 32K-token texts with 50 questions. Next: concurrent-request throughput.
• Docs and website: docs/speed.md gets a section and loses its "no NVIDIA timing for the current model" claims; the homepage hardware explorer (website/lib/evidence.ts, components/hardware-explorer.tsx) now reads these three bench.json files (512–32K tokens × 1–50 questions, plus peak memory) instead of the archived Qwen3 tree's vLLM logs; website/content/hardware.md has the A10G table, aws.md the A10G line.

2026-09-30 10:10 PDT: survey of the open Jev-like classifiers on HF and the shared benchmarks (Claude competitors, user request)¶

• What: read ≈ 50 model cards from huggingface.co/models?other=classification (by downloads, likes, trending; created since 2026-09-15) and the benchmarks they cite. Nothing downloaded or run; $0. Matrix and conclusions: landscape.md.
• Shared yardsticks: JevBench public-231 (Benchmark Heaven, MIT, not TypeSafe's; live board, sealed half run by the maintainers), LocalLLaMA/typed-decisions (400 cases), Decision Index 0.2.1 (38 public benchmarks), Nimble 13-subset suite (3,880), DecisionBench (23,900), Image JevBench (684). We have no number on any of them yet.
• Overlap check: data/all.jsonl.gz vs JevBench public-231: 0 exact, 0 with > 20 % 8-grams; vs typed-decisions test: 0 / 400. MNLI train is in hf_nli (Nimble suite has MNLI dev).
• Field: board-run leaders are Cygnet 73.7, Winnow-12B 73.2, Jev 72.1, JevK5-4B 71.9, Plumb-4B 71.6, decider-4b 71.3 (JevBench v1.5.4). Images: imajev-4b leads Image JevBench (76.39), jpt-4b 69.55. Laya: JevBench #93, Decision Index 6.04.
• Verdict: the shared prefix and text + images in one model are no longer unique (Mica, kev, ArseneLupin share the prefix; jpt, imajev, openjev, vjev-vision, AutoJev take images). Per-candidate description scoring and native multilabel are. Any "best" claim needs our numbers on the shared boards: next step is a GPU run (needs the user's OK).

2026-09-30 09:46 PDT: website defaults to the vision release (Codex vision website, user request)¶

• What / why: updated website/ for SelfJev 0.3.0 and the image-and-text default. Primary model downloads point to Jwuthrich/selfjev-4b-vision; the merged vision release is linked with its text-only vLLM limitation, and the previous text-only adapter remains secondary. Verified both public HF repositories (including both merged weight shards) and the PyPI 0.3.0 release before linking.
• Evidence: default text scores now read reports/images_v1/eval2/report.json (96.13%), eval_llm/report.json (92.49%), and test/report.json (84.07%), displayed at the site's existing one-decimal precision. Previous reports/selfjev_4b_treeserver/* scores are explicitly labelled text-only release. Image scores from reports/images_v1/images_v1_images/report.json: 90.4% across 2,002 questions, 94.2% for six trained kinds on held-out photos, 84.3% for four unseen kinds. Baseline from reports/images_v1/base_selfjev_4b_images/report.json: 75.5% overall, 70.4% / 83.5% respectively. Group aggregates are regenerated from predictions into website/data/scores.json; the site states no statistically significant transfer to unseen kinds. Timing remains 136 / 163 / 167 ms from reports/images_2026-09-29/timing.txt, explicitly measured before the vision fine-tune.
• Guides: pinned PyPI 0.3.0 installs, state=[Path("cat.jpg")], image JSONL training rows, vision model manifest, and Docker/Runpod LFS paths. Preserved the existing visual style; clarified the research timeline and release labels.
• Validation: scores snapshot regenerated (212 reports); production build and TypeScript passed; export check passed (15 HTML pages, 495 links); all 93 unique external destinations returned HTTP 200. Browser checked at 390 px (homepage and seven changed guide/research routes, no document overflow), plus desktop image section; no browser console errors observed. Code blocks scroll within their panels. git diff --check passed. Reports unchanged, so no ledger/eval2 regeneration needed; docs/findings.md and docs/leaderboard.md already reflect the vision release.
• Dev preview fix: the running server retained the old scores snapshot read with fs at module load and returned 500 after image groups were added. Changed the snapshot to a tracked JSON import so Next.js refreshes it; localhost:3001 now returns HTTP 200 with the vision links and all image scores. Rebuilt and rechecked the export successfully.
• Cost / verdict: $0; local website work only, no model jobs or paid resources. Complete; no commit made by this session.

2026-09-30 09:45 PDT: selfjev 0.3.0 on PyPI, the vision merged checkpoint on Hugging Face (Claude images, user request)¶

• PyPI: selfjev 0.3.0 (image states, image fine-tuning, the vision adapter by default) published by .github/workflows/release.yml on tag v0.3.0 (commit 6299e0d, run 36744769159: build, smoke test and trusted-publishing upload all green); wheel and sdist on pypi.org. Local checks first: twine check --strict passed; a clean venv imports it without torch, turns image bytes into a data URL and the CLI starts. PyPI README gained an images section and the vision release's numbers.
• Hugging Face: Jwuthrich/selfjev-4b-vision-merged, commit 8c2cd74356f857b1e8903eef825ad9445f066993: selfjev merge --adapter Jwuthrich/selfjev-4b-vision on a g5.2xlarge, the original multimodal checkpoint with the 152 LoRA-target tensors replaced (merge_meta.json: adapter sha256 5b619930…); both shards' sha256 on the box (036f8c81…, cfcfdc09…) equal the Hub's LFS hashes; the page shows no config error. The card says which release to use: the adapter + default engine for images, this one on the vLLM backend for text. The HF token went to the upload process over ssh stdin only (never written on the box). Not benchmarked through vLLM.
• The adapter card (Jwuthrich/selfjev-4b-vision, commit 55fb32c, and weights/selfjev_4b_vision/README.md) now installs from PyPI 0.3.0 instead of GitHub.

2026-09-30 08:30 PDT: images v1 is the default, published as Jwuthrich/selfjev-4b-vision (Claude images, user request)¶

• Hugging Face: new public repo Jwuthrich/selfjev-4b-vision, commit 10ff7de14dc9e1fcf0bf949fefed1553e6ea8543: adapter_model.safetensors (sha256 5b619930…, the adapter every reports/images_v1/ number was scored with; LFS hash checked after upload), adapter_config.json (the run's, minus the null task_type HF's page parser rejects: JOURNAL 2026-09-28 13:08), model.json and a model card with the text, image and speed numbers from their report files. The page shows no config error. The text-only repo Jwuthrich/selfjev-4b and the merged checkpoint are untouched (the merged one stays text-only).
• Default switch: weights/selfjev_4b_vision/ (same four files) is DEFAULT_ADAPTER; HUB_ADAPTER is the new repo (a pip install with no checkout downloads it); the Dockerfile and selfjev deploy aws pull that folder. The served model is still named selfjev-4b. weights/selfjev_4b stays (text release; its assets/ feed the README, website and card). AGENTS.md, README(s), weights/README.md, docs/api.md, findings (headline), leaderboard updated. vs Jev: eval2 25 / 47 (p = 0.013), dev benchmark 168 / 121 (p = 0.007).
• Not done: the website (another session's) and the PyPI release (0.2.0 predates image support); the merged checkpoint for vLLM (needs a GPU box, and vLLM does not take images yet).

2026-09-30 02:05 PDT: images v1: one mixed fine-tune learns 6 image tasks (70.4 -> 94.2%), text holds, no transfer to new image tasks (Claude images)¶

• Data (scripts/data/build_images_v1.py, all gitignored): 11,344 image questions from 6 licence-safe HF datasets (pets CC BY-SA, fashion-MNIST MIT, beans MIT, rice CC0, EuroSAT MIT, TrashNet MIT; ~1,000 photos each, a multiclass among up to 12 options + a yes/no per photo) + 11,344 text questions drawn (seed 0) from selfjev-4b's own training rows with Jev's soft targets (runs/jev_all). Validation: 596 image + the usual 1,200 text. Frozen image test data/ova/eval_images_v1.jsonl (sha256 c2e44782…): 2,002 questions over 1,063 photos, ~100 photos from each training dataset's test part and 4 HELD-OUT datasets never trained on (hurricane damage CC BY 4.0, snacks CC BY 4.0, indoor scenes and painting style: research, test only). No test photo is in train or validation.
• Run: selfjev finetune --init weights/selfjev_4b --lr 5e-5 --soft-weight 0.5 --max-length 16384 --batch-tokens 16384 --grad-accum 2, 1 epoch, 622 steps, ≈ 3 h on one L40S (≈ 17 s per step, slower than planned: the user OK'd the full epoch). Validation (mixed) accuracy 85.9 -> 92.6, loss 0.316 -> 0.166; best = step 600. Adapter: runs/images_v1/adapter (not in git, sha256 in the reports' meta). Reports: reports/images_v1/ (base_selfjev_4b_images = the zero-shot baseline).
• Images (paired McNemar vs zero-shot selfjev-4b):

pets-100 (the earlier 100 questions): 87 (zero-shot 78, pets-only fine-tune 90). - Text gate (vs selfjev_4b_treeserver, same engine, paired): eval2 95.68 -> 96.13 (23 / 14, p = 0.19); eval_llm 93.13 -> 92.49 (10 / 16, p = 0.33; Jev 92.5); dev benchmark 83.78 -> 84.07 (49 / 39, p = 0.34). None significantly worse: the gate passes. The pets-only fine-tune without text replay had cost eval2 0.4; the replay removed that. - Verdict: fixed-label image data teaches exactly its tasks (large gains) and costs no text quality with a 1:1 text replay, but gives no significant transfer to unseen image tasks. Transfer probably needs varied open questions per photo (LLM-written, judge-checked), not more label sets. Not adopted as the default yet: the user decides. Cost ≈ $6.79.

2026-09-29 23:47 PDT: redesign the architecture walkthrough (Codex architecture UX)¶

• What / why: replaced the oversized SVG flowchart with a compact HTML walkthrough in the site's sans-serif typography: one shared input, three question rows, answers in place and a concise output strip. Removed obsolete SVG styles. Customer-support and AI-review example buttons, selectable decision explanations, and an optional replay make the shared-context idea explorable. Updated the architecture guide's link wording.
• UX: no automatic looping; replay finishes and restores its control. Reduced-motion preference bypasses replay and disables transitions. Mobile stacks the input, questions and output without horizontal scrolling; decision explanations appear below the questions. Examples and output are explicitly illustrative, not inference or new benchmark data.
• Validation: npm run build (including TypeScript), node scripts/check-export.mjs (15 pages, 484 links), and git diff --check pass. Browser checked on desktop and at 390 × 844 (page width 390); verified example switching updates outputs, question selection updates explanations, replay disables while running and restores all answers afterward, and no browser warnings/errors.
• Cost / verdict: $0; no model jobs, paid resources, report edits or score regeneration. Ready on the local dev site; this session made no git commits. In-flight row removed.

2026-09-29 23:35 PDT: simplify image fine-tune card (Codex image copy)¶

• What / why: changed the result label to “SelfJev + image fine-tune” and replaced the photo count with “Fine-tuned on images”, as requested. Preserved 90% and the Oxford-IIIT Pet test scope from reports/images_2026-09-29/pets_ft_pets100/report.json.
• Evidence boundary: the 25-dataset figure describes the discovery catalog (JOURNAL 2026-09-29 19:55; docs/image_datasets.md), not this model's completed training. The separate in-flight images_v1 recipe uses six training datasets (scripts/data/build_images_v1.py). Did not describe the 90% model as trained on 25 datasets.
• Validation / cost / verdict: fetched the running Next.js homepage on port 3001 and verified both new strings and removal of both old strings; git diff --check passes. $0, no inference, reports unchanged, no commit. Done.

2026-09-29 23:18 PDT: announce image input on the website (Codex website images)¶

• What / why: added a homepage announcement and Images section in the existing dark/olive/orange style: copyable Python Path("cat.jpg") example, mixed text/image input explanation, shared frozen vision root, and image fine-tuning. Linked the model, API and fine-tuning guides to docs/api.md; the homepage and training guide link docs/image_datasets.md.
• Evidence: L40S warm medians 136 ms text / 163 ms image / 167 ms image with 5 questions, copied from reports/images_2026-09-29/timing.txt. Pet-photo accuracy 73% / 78% / 90%, checked against the 100 predictions in reports/images_2026-09-29/base_pets100/report.json, selfjev_4b_pets100/report.json, and pets_ft_pets100/report.json. 2,220 training photos from the 2026-09-29 20:40 entry above. Explicitly limited to Oxford-IIIT Pet and 100 test questions; the 90% fine-tune is separate from the text-trained release. No images_v1 results cited.
• Validation: cd website && npm run build passes (including TypeScript); node scripts/check-export.mjs passes: 15 HTML pages, 484 links. Built export inspected in the browser at 390 × 844: document and section width 390 px, readable stacked cards, functioning Copy button; desktop layout checked too, no browser errors/warnings. git diff --check passes.
• Cost: $0; local website build and UI checks only, no model inference or paid jobs. No reports changed, so no ledger or scores snapshot regeneration needed. Verdict: ready for review; no commit or push. In-flight row removed.

2026-09-29 20:40 PDT: images: 163 ms per request, and 2,220 pet photos lift the test from 78% to 90% (Claude images)¶

• Speed (L40S, scripts/eval/time_images.py, 20 photos, warm, reports/images_2026-09-29/timing.txt): text 1 question 136 ms; image 1 yes/no 163 ms (10 ms CPU resize, 153 ms model); image 12-way 165 ms; image with 5 questions 167 ms. The photo is encoded once, so more questions on it are nearly free. The first image request loads the vision tower: 4.1 s. Before the fix the same 100 questions took 411 s (fp32 tower, non-cuDNN Conv3d), after it 16 s. Through the SDK from the Mac (ssh tunnel, 2 questions): 284-355 ms, text 272 ms.
• Fine-tune on images (selfjev finetune --init weights/selfjev_4b --lr 5e-5 --epochs 1, data/pets_train.jsonl: 4,218 train / 222 validation questions from 2,220 Oxford-IIIT Pet train photos, description style "persian", not the test's "a Persian cat"): 35 steps, ≈ 15 min, no memory problem. Validation accuracy 90.5 -> 94.1, loss 0.296 -> 0.171.
• Test (eval_pets100, test-split photos, no overlap with training), reports/images_2026-09-29/:

Fine-tuned vs zero-shot: 13 only-fine-tuned / 1 only-zero-shot, McNemar p = 0.0018; vs base p = 0.0002. - Text after image training (eval2, no text rows mixed in): 95.68 -> 95.28 (10 / 18, p = 0.19, not significant); binary 96.7 -> 96.7, multiclass 96.8 -> 96.5, multilabel 91.4 -> 89.8. Mixing text rows into the image file is the obvious guard. - SDK demo (state=[Path(...)], plain selfjev-4b server): Persian 0.996, Siamese 0.950, Bengal 0.994 on their own photos, "is there a cat" 0.989-0.991: reports/images_2026-09-29/sdk_demo.txt. - Caveats: 100 questions (61 / 18 / 21 by family) is small; the 37-breed part has 18. One dataset, one run, one lr. data/eval_pets.jsonl is only the pets test split: no other visual task is measured. - After the run: selfjev serve's warm-up also sends one image (a 1 × 1 PNG, 64 tokens), so the vision tower loads before /health answers instead of on the first image request (the 4.1 s); an engine that refuses images still starts (tests/server/test_app.py). Not timed on a GPU. - Cost: ≈ $1.12. Verdict: image input and image fine-tuning work end to end and are fast; text cost of image training is within noise here. Next: more datasets (docs/image_datasets.md) mixed with text rows, full 1,853 pets questions.

2026-09-29 19:55 PDT: fine-tuning on images: code, datasets, GPU run waiting for the user's OK (Claude images)¶

• Fine-tuning takes images: encode_items keeps an image as a data URL in the shared root (tree.Root), score() makes its pixels per batch (a corpus of pixel tensors does not fit in memory), the tower stays frozen (no grad), so selfjev finetune, rlcd and the fine-tuning endpoint (upload rows with an image or a list of parts as state) train the language LoRA on photos. tests/engine/test_tree_image.py::test_training_on_images_matches_full_forward_gradients: every language-model gradient through the tree equals HF's full multimodal forward (tiny random model, CPU); tests/server/test_finetuning.py::test_image_rows_upload_as_parts. 98 tests pass. Not yet run on a GPU.
• Datasets: image_datasets.md: 25 HF classification sets checked (rows, classes, licence). scripts/data/hf_images_to_rows.py <hf id> turns any of them into rows (multiclass among 12 options + yes/no per photo); data/pets_train.jsonl (gitignored): 2,220 photos (60 per breed, Oxford-IIIT Pet train split) -> 4,440 questions.
• Not measured yet: the image latency after the bf16 tower + matmul patch embed fix (scripts/eval/time_images.py), and any image fine-tune. Driver ready; waiting for the user's price OK.

2026-09-29 18:35 PDT: first image scores, 100 random pets questions, zero-shot (Claude images)¶

• Data: 100 questions sampled (seed 0) from data/ova/eval_pets.jsonl (eval_pets100, sha256 e6eee7d0…; 61 cat breed 12-way, 18 all breeds 37-way, 21 "is it a cat"), one photo each. Human labels (Oxford-IIIT Pet). L40S, TreeServer path.
• Results (reports/images_2026-09-29/{selfjev_4b,base}_pets100):

Paired: 10 only selfjev-4b, 5 only base, exact McNemar p = 0.30: no significant difference at n = 100. The LoRA trained on text does not hurt the vision path; cat-breed identification is the weak spot (69%). - Speed problem, found and fixed in code, not re-timed on a GPU: 411 s for 100 questions (~4 s per image, vs 0.23 s per text question on eval2), which is also why the full run of 18:10 was still running at its 2 h cap. Cause: the vision tower was loaded in fp32 (_from_config default) and its patch-embed Conv3d ran without cuDNN (slow path). Fix: tower in the model dtype (bf16, rotary buffers stay fp32, as HF), and the patch embed as the exact matmul it is (kernel = stride = patch; qwen35.linear_patch_embed, test test_linear_patch_embed_equals_conv3d), so cuDNN is no longer needed. The 100-question scores above were made before this fix (fp32 tower). - SDK demo not run: OrbStack listens on port 8000 on the Mac, so the ssh tunnel to the box's server failed (404 from OrbStack). The SDK and server image paths are unit-tested (tests/client, tests/server/test_compat.py), not end to end. - Cost: ≈ $0.55. Next: time the fixed tower; full 1,853 questions; image training data if cat breeds matter.

2026-09-29 18:10 PDT: images as the state: engine and API work, image scores lost with the box (Claude images)¶

• What: state may now be an image (a base64 data:image/... URL, Prem's Jev-compatible convention) or a list of text and image parts; the SDK turns a Path, bytes or a PIL image into one (state=[Path("cat.jpg")]). Jev takes no images (TypeSafe docs: text / JSON only). The tree is kept: the image is the root, encoded once by the checkpoint's own vision tower (334M, frozen, loaded on the first image; Qwen35Scorer.visual), root positions are Qwen3.5's 3D M-RoPE (tree.root_rope), questions and options branch as for text. tests/engine/test_tree_image.py: tree = HF's own multimodal forward for every leaf (tiny random model, CPU). Training on images and --engine vllm refuse image states.
• Test set: scripts/data/build_eval_pets.py -> data/{,ova/}eval_pets.jsonl (gitignored, 114 MB each, rebuilt byte-identical on the box: sha256 803190c4…): Oxford-IIIT Pet test split (timm/oxford-iiit-pet@089695c, human labels, CC BY-SA 4.0), 1,853 questions / 1,567 photos: cat breed 12-way (1,183), all breeds 37-way (370), "is it a cat" (300).
• eval2 regression with the new code (L40S): 95.68, identical to selfjev_4b_treeserver (2 of 1,991 decisions differ, max score difference 0.43): reports/images_2026-09-29/selfjev_4b_eval2. A 5-question image smoke test: 5 / 5.
• Lost: the two full image runs (selfjev-4b, bare base) started 22:37 UTC; my completion poll waited for a marker that never came (the job likely died without a traceback) and I did not pull by hand, so the 2 h shutdown -h cap terminated the box with the reports. Lesson: pull each report as it lands and bound every wait.
• Box fixes found on the way: AutoImageProcessor needs torchvision even with backend="pil" (use Qwen2VLImageProcessorPil); cuDNN 9.24 cannot load libcudnn_engines_runtime_compiled on the DLAMI CUDA 13 box, so every cuDNN Conv3d fails: the vision tower runs with cuDNN off (its only conv is the patch embed).
• Cost: ≈ $3.72 (2 h g6e.xlarge). Verdict: image input works end to end; accuracy not measured yet.

2026-09-28 13:10 PDT: a 75-second video of the end-to-end run, made from its record (fork, user request)¶

• What: scripts/docs/e2e_video.py turns reports/e2e/<date>/ into selfjev_e2e.mp4 (1920×1080, 30 fps, 75 s, 3.0 MB, Git LFS) and poster.png: a page in the site's style (DM Sans, JetBrains Mono, charcoal and orange) animated by a JS timeline, captured frame by frame by headless Chromium (Playwright) and encoded by ffmpeg. Every answer, number and name on screen comes from transcript.jsonl and report.md: healthy 5:24 after launch, the six answers, identical answers on Jev's path, the error codes, both fine-tuning jobs in their queue order (8:07, then 5:05), the new models in /v1/models, 14 / 14, ≈ $0.81. Renders in about 3 minutes locally; no model, no paid API.
• Left out on purpose: concurrency timings (an open issue) and the jobs' validation numbers: the supervised job's validation fell from 1.00 to 0.88 (41 questions), so the default lr 2e-4 is probably too high when continuing from selfjev-4b on a small file; RLCD at 5e-5 kept 1.00. Worth a lower default for jobs that start from an adapter.
• Cost: $0.

2026-09-28 13:08 PDT: fixed Hugging Face adapter configuration warning (Codex HF config)¶

• HF's page parser rejected task_type: null in the published adapter config, although PEFT supports it. Removed the optional null field from weights/selfjev_4b/adapter_config.json and published only that file to Jwuthrich/selfjev-4b, commit ce0b0388d66ac3ee94f42a45a7c3109e7ccb79d9, using parent-commit protection. Omitting it preserves PEFT's default None and generic PeftModel wrapper; assigning a task enum would change the wrapper.
• Verified original/omitted-field configs deserialize identically. Tiny random 4-layer Qwen3.5 CPU fixture with nonzero LoRA weights loaded both variants into PeftModel with identical state tensors and bit-identical logits. Initial 2-layer test fixture had no attention q/v modules and was corrected to the hybrid 4-layer fixture; no full model run. All remote files except adapter_config.json retain their original blob IDs, including the weights.
• Read back the pinned published config; fresh HF page HTML no longer contains configError or the task_type warning. Evidence: reports/releases/selfjev_4b_config_2026-09-28/publication.json. Cost $0. No local git commit; verdict: published and verified.

2026-09-28 12:40 PDT: the product works end to end on a GPU: deploy, SDK, fine-tuning over HTTP (fork, user request)¶

• What: scripts/aws/e2e.py (new): selfjev deploy aws up --fine-tuning of commit 0a146ba on a g6e.xlarge (L40S), then through the SDK: health, auth, errors, models, every question type on a ticket with obvious answers, Jev's paths and names, 16 concurrent requests, a supervised and an RLCD job over HTTP (batch numdate_neg_v1 training rows, 508 questions) and their models. Every HTTP call recorded: report, transcript.jsonl, setup.log.
• Result: 14 of 14 checks passed. Healthy 5.4 min after launch; selfjev-4b answered the ticket right (refund 0.99, spam 0.02, team billing, urgency 1.98 of 2, topics [invoice]), identically through Jev's path; the supervised job took 8.4 min, the RLCD job 5.4 min, and both models were served at once and still answered right.
• Found and fixed: the first request took 39 s (kernel compilation): selfjev serve now warms up before /health answers. 16 concurrent requests all waited for one batch (18.6 s cold; 4.4–6.4 s warm with a training job on the GPU): open, see known issues. The script's teardown crashed because a plain uv run in the same checkout had removed boto3 mid-run; the box was terminated by hand 6 minutes later (SG and key pair deleted); the dev group now includes the deploy extra (pyproject change still uncommitted, next to the PyPI session's) and the script tears down first.
• Cost: ≈ $0.81 (26 min). A first attempt failed before creating anything (no AWS profile in the environment, $0).

2026-09-28 12:10 PDT: PyPI packaging for selfjev (Claude Code PyPI)¶

• What: the package is ready to publish as selfjev on PyPI (the name was free on 2026-09-28). Code license Apache-2.0, chosen by the user (LICENSE, license = "Apache-2.0"); the weights keep their own model-card licenses. Added a PyPI README (README.pypi.md, absolute links, no LFS images), URLs, classifiers and keywords. __version__ now reads the installed metadata, so pyproject.toml is the only version.
• Why: pip install "selfjev[serve]" && selfjev serve failed with no checkout. Now --adapter/--init fall back to the Hub: the missing default weights/selfjev_4b downloads Jwuthrich/selfjev-4b, and any other org/name that is not a local directory downloads from Hugging Face. huggingface_hub was added to the serve extra. A missing extra now exits with its pip install 'selfjev[…]' line instead of a traceback.
• Release: .github/workflows/release.yml. Pushing tag v<version> builds, checks the tag against the version, smoke-tests the wheel (it imports without torch and the CLI starts) and publishes via trusted publishing; workflow_dispatch publishes to TestPyPI. The PyPI/TestPyPI trusted publishers (environments pypi/testpypi) still need registering by the user.
• TypeSafe SDK drop-in (same session, user request): typesafe-sdk 0.7.2's TypeSafeClient() with only TYPESAFE_BASE_URL + TYPESAFE_API_KEY now works against selfjev serve. It already reached /v1/systemone; what failed was models.list() (/v1/models now also returns TypeSafe's models list) and Jev's looser question schema: optional instructions (a per-type default instruction), JSON instructions/criteria (JSON text, like state), a noul described on one side (the other side = not: <it>). A noul with neither instructions nor criteria is a 422. New tests: tests/server/test_typesafe_sdk.py (the real SDK against a live server, a dev dependency) and tests/server/test_compat.py. The TestPyPI job was dropped: only the PyPI publisher is registered.
• Checks: uv run pytest -q 92 passed, ruff clean, twine check --strict PASSED on sdist and wheel, clean-venv install OK. Cost $0. Nothing published and no commit.

2026-09-28 11:55 PDT: website-styled GitHub README prepared (Codex README)¶

• Rebuilt the root README around the site's charcoal/orange/olive palette, “Intelligence, decided” headline, concrete decision examples, and product-first navigation. New vector banner: docs/assets/readme/hero.svg; generator scripts/docs/build_readme_assets.py reads current TreeServer report scores. Reused the existing prefix-tree illustration.
• Added server-first quickstart with explicit adapter download and user-chosen server key, a typed SDK example, current TreeServer comparisons (95.7 / 93.1 / 83.8), links to both weight releases and Decision Bench, deployment-guide navigation, and a collapsed contributor section. Kept evaluation limitations and source links; moved the old run history to linked research pages.
• Validation: GitHub Markdown API render, browser visual check, loaded images and anchor navigation (no page overflow at the current 614px viewport); all 32 local link/image targets exist; report counts and headline values match source files; SDK snippet passes real schema validation with a mocked transport and no inference; Ruff check/format and git diff whitespace checks pass. CLI download arguments checked against installed HF help.
• Cost $0; no model jobs, cloud resources or benchmark reports. Changes are local; commit/push choice requested under the repository's no-commits-without-request rule. Verdict: ready to publish.

2026-09-28 11:32 PDT: current-model comparison and missing Jev score (Codex website benchmarks)¶

• Homepage and primary research chart now compare only current SelfJev and Jev. Earlier SelfJev and reranker runs remain in the expandable experiment archive. Removed the outdated two-SelfJev-versions caption.
• Fixed Jev AI-review source lookup: its recorded result lives in the canonical dataset and the audited weights/selfjev_4b/assets/chart-data.json export, not the report path previously assumed by the website. The site reads 92.49471458773785% from that export and checks question count/current-model score agreement before using it. UI shows SelfJev 93.1%, Jev 92.5%; raw source is linked, no inference/report mutation.
• npm run check passes (15 HTML pages, 473 links). Browser interaction verified both rows on AI response review. Cost $0; no model run or git commit. Verdict: complete.

2026-09-28 11:30 PDT: website links to published Hugging Face releases (Codex website HF links)¶

• Verified public HF API metadata for Jwuthrich/selfjev-4b, Jwuthrich/selfjev-4b-merged, and Jwuthrich/selfjev-decision-bench; read the published dataset card. Added shared release cards to the homepage/research page, a hero weights link, footer profile/dataset links, and quickstart download links. Fine-tuning docs now link the published adapter and its provenance/terms instead of stale pre-publication wording.
• Dataset copy correctly lists 3,657 questions across Text Decisions, AI Response Review and Record Reasoning; research clarifies that the 3,471-question development benchmark is separate. No score changes or new reports.
• npm run check passes (15 HTML pages, 476 links). Browser confirmed release targets and layout, 390px viewport has no horizontal overflow. Cost $0; read-only HF access, no publication or model run, no git commit. Verdict: complete.

2026-09-28 11:27 PDT: concrete typed decisions in the animated tree (Codex website examples)¶

• Replaced anonymous Question 01 / A-B-C nodes with one customer message about a duplicate charge. Binary refund question has one Yes candidate and yes/no readout; multiclass team routing has Billing/Support; multilabel topics has Payments/Refund/Login with two selections. Highlighted illustrative selections and displayed typed example values at the output. No invented probabilities or live inference claims.
• Branch paths now derive from the candidate lists, preserving animation for different branch counts. Updated accessible description, legend, and mobile root sizing. Files: website/components/architecture-tree.tsx, website/app/globals.css.
• Validation: production build and 432 link checks pass; desktop/mobile layout and pause/play checked, no horizontal page overflow. Cost $0, no model run, no report/score changes, no commit. Verdict: complete.

2026-09-28 11:20 PDT: three authored evaluation suites published as SelfJev Decision Bench (Codex dataset release)¶

• User selected all three authored suites. Published SelfJev Decision Bench, commit 459c0ab641ab2cd7faa3dcf86daf05c5ff33cf6e: Text Decisions 1,991 questions / 647 texts; AI Response Review 946 / 317; Record Reasoning 720 / 120. Exported exclusively from canonical data/all.jsonl.gz, with original IDs, questions, targets and metadata preserved; omitted the jev prediction field. No third-party public benchmark data or training rows included.
• Added scripts/data/export_decision_bench.py, dependency-free scripts/data/score_decision_bench.py, and docs/decision_bench_card.md. Release has three HF Parquet configurations, original-schema gzipped JSONL, scorer/exporter, provenance manifest and SHA256SUMS. Data card documents AI agreement limitations, development exposure, exact scoring, metadata/notes answer leakage, and separate reporting of the narrow programmatic challenge. Linked from docs/data.md.
• Validated every row against original frozen files, all Parquet/native round trips, all retained labels against both saved judge outputs, and zero exact text matches with non-test training rows. Scorer reproduces saved TreeServer predictions: 1,905/1,991 and 881/946; negative checks reject missing/duplicate IDs, wrong binary types and duplicate labels. No new inference or headline scores. Ruff and HF card validation pass.
• All 11 uploaded files fetched anonymously and byte-matched. Evidence: reports/releases/decision_bench_2026-09-28/ (publication, validation, manifest and hashes). Reuse license remains unset pending the user's CC BY 4.0 versus CC0 choice; the public card explicitly says so, and no license was assigned on the user's behalf.
• Cost $0; no paid calls, AWS resources, model jobs or git commit. Verdict: public evaluation release complete; license choice pending.

2026-09-28 11:19 PDT: animated architecture, context capacity and fine-tuning copy (Codex website animation)¶

• Rebuilt the homepage tree as a React/SVG component (website/components/architecture-tree.tsx): staged light travels document → questions → isolated candidates → readouts → typed answers; pause/play, reduced-motion CSS, accessible description, and a centered horizontal viewport on mobile. Original PNG remains available from the architecture guide.
• Added a homepage section about configurable context and adapting SelfJev to a use case, with fine-tuning workflow and documentation links. Verified TypeSafe's primary model docs on 2026-09-28: Jev allows 32K for state plus longest question, 64K for the complete request. Qwen3.5-4B's official config specifies 262,144 native positions. Website distinguishes this base capacity from SelfJev's 32,768 default and unvalidated full-window inference; it makes no tested long-context superiority claim. Sources: https://docs.typesafe.ai/models and https://huggingface.co/Qwen/Qwen3.5-4B/blob/main/config.json .
• Updated architecture, hardware and fine-tuning guides. Validation: npm run check passes (15 HTML pages, 432 links), desktop/mobile screenshots inspected, pause/play verified, no horizontal page overflow or browser errors. No model runs, new reports, or benchmark changes. Cost $0; no paid API/cloud resource, no commit. Verdict: complete.

2026-09-28 10:43 PDT: latency evidence and prefix-tree diagram on both HF cards (Codex HF visuals)¶

• Added a dedicated vector/raster architecture diagram: document → independent questions → candidate branches → probabilities, with attention ancestor visibility and inherited DeltaNet state. Describes TreeServer; vLLM execution is explicitly separate. Original header artwork remains.
• Added two latency figures and tables generated from raw samples: archived tree_4b_combo Qwen3/vLLM on A10G, L40S and H100 (24 cells, 10 warmed requests each, server-side time), and current SelfJev/TreeServer on M5 Pro MPS (4 cells, 10 synchronized local calls after two warmups, synthetic repeated text). Files: reports/latency/requests_combo.jsonl, requests_qwen35.jsonl, requests_h100.jsonl, mac_m5_pro_selfjev4b.json. NVIDIA/current-model gap and different timing boundaries are explicit; no direct Mac-vs-NVIDIA claim.
• New plotting module: scripts/docs/model_card_figures.py. Main generator now produces eight PNG/SVG figures, latency CSV/JSON with samples and configuration, and a chart-data manifest hashing 22 inputs. Reused on merged card with refreshed SHA256SUMS. Website files/artwork belong to the concurrent website session and were not edited here.
• Published adapter commit 5126b220cde8c8f31f85eaaefa27592e880377b2, merged commit f053919b6f498eb8b8667b97c0784adddef9e608. Ruff, metadata, example syntax, image/link checks and median recomputation passed. Visually inspected all three new figures, corrected one label overlap. Verified all 24 adapter-card and 26 merged-card files anonymously byte-for-byte; model artifacts unchanged. Evidence: reports/releases/selfjev_4b_merged_2026-09-28/latency_architecture_cards.json.
• Cost $0; no model jobs, new benchmark reports, cloud resources or git commits. Verdict: requested latency and architecture additions live on both model cards.

2026-09-28 10:40 PDT: simpler hardware copy and architecture illustration (Codex website architecture)¶

• Removed repeated Qwen-generation caveats from homepage hardware copy and website guides at the user's request. Raw timings are unchanged; the hardware methodology links the original model/runtime configurations and identifies the vLLM versus MPS measurement paths.
• Created website/public/images/selfjev-architecture.png with built-in image generation, showing shared document and question computation, isolated candidate paths, yes/no readout, and typed outputs. Integrated the picture into the homepage with descriptive alt text and a full-size link; linked it from the architecture guide. Prompt: website/design/selfjev-architecture-prompt.md.
• Validation: production build and export check pass (15 HTML pages, 428 links); browser confirmed loaded image and updated copy; 390px mobile viewport has no horizontal overflow. No model results changed and no inference run. Cost: no AWS/BYOK resources; built-in image generation, no separately reported price. Verdict: complete; no git commit.

2026-09-28 10:36 PDT: visual model card reused on the merged HF release (Codex HF card)¶

• Published the same five figures, benchmark tables and chart data to Jwuthrich/selfjev-4b-merged, commit 8588c5198d3697527ecbc74809bd718212810080. Updated full-checkpoint description, vLLM download/serving commands, model file table and source-adapter provenance. Reference scores remain explicitly attributed to TreeServer, not a new vLLM evaluation.
• Reusable converter: scripts/docs/build_merged_model_card.py; staging: runs/hf-merged-card/. Refreshed SHA256SUMS for README and new assets/reproduction files while retaining all original artifact hashes.
• Ruff, frontmatter, example syntax, local/remote file links and checksums validated. All 17 uploaded files fetched anonymously and byte-matched; model weights/config/tokenizer/provenance blobs unchanged. Hugging Face updated its generated .gitattributes. Publication evidence: reports/releases/selfjev_4b_merged_2026-09-28/visual_card_publication.json.
• Cost $0; no model inference, cloud resources or git commit. Verdict: merged model now has the shared visual card.

2026-09-28 10:33 PDT: hardware latency explorer and local M5 Pro measurement (Codex website)¶

• Added an interactive self-hosting comparison to the Next.js homepage for 8/512/2,048/4,096 text tokens and 1/16 questions. A10G, L40S and H100 values are medians from the existing tree_4b_combo Qwen3-4B vLLM sweeps (reports/latency/requests_combo.jsonl, requests_qwen35.jsonl, requests_h100.jsonl); the site labels their archived model and links the raw rows. Current-model NVIDIA latency remains unmeasured.
• With the user's explicit local-Mac authorization, measured the current selfjev-4b adapter through TreeServer on an Apple M5 Pro (20-core GPU, 48 GB unified memory, PyTorch MPS, bf16 merged adapter). One request, synthetic repeated text, three answer options, exact 8/512 text tokens, 1/16 questions, two warm-ups and 10 timed repetitions per cell, with MPS synchronization; local call time includes tokenization, excludes HTTP/network. Medians: 8 tokens 688/6,786 ms and 512 tokens 2,018/8,599 ms for 1/16 questions. Full samples/config: reports/latency/mac_m5_pro_selfjev4b.json; reproduction: scripts/bench_local_mps.py. PyTorch's reference gated-recurrence path on MPS is slower than the unavailable optimized kernel, so these Mac numbers do not isolate hardware differences from the earlier NVIDIA model/engine.
• Updated the hardware guide and docs/speed.md/docs/findings.md with scope and limitations. npm run build, uv run pytest -q tests/engine/test_tree.py (3 passed), and ruff checks passed; browser controls, Mac values, source and sizing links verified. Cost $0; no cloud resources or API calls. Verdict: site has a measured but clearly separated Mac row; Mac CLI serving and current-model NVIDIA latency still need work.

2026-09-28 10:31 PDT: full merged SelfJev-4B weights published (Codex HF merge)¶

• User approved the proposed $3 AWS job and refreshed SSO. Published Jwuthrich/selfjev-4b-merged, commit aed18d68e951f3cbbb84495c931f321fd5e03f10: 9,319,828,160 bytes of full merged weights, tokenizer/configuration, model card, source manifest, base LICENSE, merge provenance and file checksums.
• Existing merge command combined the pinned Qwen base with adapter dfbf2834d883987893ec305a6093a345fd79c77f903f60f2f914cbe3b6058d1b. Checked all 738 tensors for shape/dtype/finiteness; exactly 152 expected tensors changed. Two synthetic GPU smoke cases produced identical adapter/merged decision margins (4.5625 and -4.3125), not a benchmark or proof of general equality. No headline score changes.
• All checksummed Hub files verified anonymously from AWS and independently from the laptop. Evidence: release record, verification.json, merge_meta.json, SHA256SUMS, publication.json and MODEL_CARD.md in that directory. Updated weights/README.md and docs/deploy.md with the merged download link. The adapter card was left to the concurrent card-redesign session.
• AWS g5.2xlarge i-0d2a8155413fe9cd5, us-east-1, launched 17:18:50 UTC; termination requested 17:27:04 UTC. Verified automatic one-hour termination cap. A delayed shutdown required a force/skip-OS-shutdown request; instance now confirmed terminated. Security group sg-0001aea61abb2fb73, cloud key pair and local private key were deleted successfully.
• Compute estimate ~$0.17 at the live quoted $1.212/hour, plus storage/IP and outbound transfer (conservative total allowance ~$1.10, before free transfer/deduplication; final bill unavailable). No model jobs on the laptop, new benchmark evaluation or git commit. Verdict: full merged release is public and verified.

2026-09-28 10:25 PDT: visual HF model card published with audited comparisons (Codex HF card)¶

• Used Lev's public model card as a structural reference; created original dark/orange/olive figures and plain-language task descriptions. Published README, five PNG/SVG figures, CSV/JSON data, and reproduction source to Jwuthrich/selfjev-4b, commit 742f98f0cad6eae5399e34562e47c53bf8fbfe8f.
• Generator: scripts/docs/build_model_card.py; prose: scripts/docs/model_card.template.md. Reads canonical data/all.jsonl.gz plus saved predictions, hashes 18 inputs and checks matched question IDs, targets, types and candidate sets for report comparisons. Scores are generated from report correctness, never retyped from the ledger.
• Figures: shared-prefix architecture; current SelfJev/Jev on three suites; all 11 public-source slices; accuracy vs nominal backbone size for seven systems; six historical research checkpoints. Public-source-only result is 83.4% vs Jev 82.1% on 3,300 questions (not the full 3,471-question dev benchmark). Earlier 8B LoRA is 81.0% on those same questions. Evidence: reports/selfjev_4b_treeserver/test/report.json, reports/external/full/typesafe_jev-latest/report.json, reports/lora_8b/test/report.json; exact values in weights/selfjev_4b/assets/chart-data.json.
• Card labels adapted tasks, sampled options, training-source exposure, known overlap, AI labels, different training recipes and current-vs-historical engines. No S1Bench/Lev ranking or current-model latency claim. Follow-up protocol: docs/model_card_evaluation_plan.md; no new benchmark campaign launched.
• Validation: Ruff passes; all five figures visually inspected (fixed a footer overlap); frontmatter, local links and Python example syntax checked. All 15 published files downloaded anonymously at the commit and matched local bytes. Remote adapter SHA-256 unchanged. Cost $0, no inference, no git commit. Verdict: redesigned card live; independent external evaluation remains future work. Merged-weight release remains pending separately.

2026-09-28 10:14 PDT: merged Hugging Face release prepared, awaiting AWS access and budget (Codex HF merge)¶

• User requested a full merged model. Confirmed no merged checkpoint remains in runs/; existing selfjev merge preserves the pinned Qwen checkpoint layout and checks every LoRA target is replaced.
• Prepared runs/hf-merged-release/README.md and PLAN.md for Jwuthrich/selfjev-4b-merged. Card distinguishes source-adapter TreeServer scores from unmeasured merged-backend performance. Current HF account is Jwuthrich.
• Planned one g5.2xlarge (32 GiB host RAM), 1-hour automatic termination, proposed $3 budget including transfer/storage. AWS public published price is $1.212/h; live pricing verification failed because the connectly SSO session expired. Requires refreshed login and explicit priced approval under AGENTS.md before launch.
• No model computation, cloud resources or HF writes for this merged release. Cost $0 so far; no new benchmark report or git commit. Verdict: preparation complete; upload pending.

2026-09-28 10:03 PDT: publish SelfJev-4B under the user's new Hugging Face account (Codex HF upload)¶

• Verified the active account is Jwuthrich and uploaded adapter weights, PEFT config, manifest, and model card to https://huggingface.co/Jwuthrich/selfjev-4b, commit 2d2292461452bda3be49dc554e1a8e9dd1510072. Updated the card's download instructions to Jwuthrich/selfjev-4b.
• Anonymous Hub access verified public visibility and all four files. The three small files match local bytes; the 230,073,800-byte adapter's remote SHA-256 matches the local manifest: dfbf2834d883987893ec305a6093a345fd79c77f903f60f2f914cbe3b6058d1b.
• The earlier JulienHeysam/selfjev-4b repository was not deleted. No model jobs, paid services or local git commit. Cost $0. Verdict: requested weights and card published under the newly signed-in account.

2026-09-28 09:59 PDT: SelfJev-4B adapter and card published to Hugging Face (Codex HF upload, explicit user request)¶

• Uploaded the existing adapter, PEFT config, unchanged model manifest, and finalized model card to https://huggingface.co/JulienHeysam/selfjev-4b. The user explicitly instructed publication after the contractual concern was reported; this records that decision, not a finding that contractual permission was obtained.
• Hub commit 104f569298a621e65785ba7f52e7fe077aabd894. Anonymous Hub API confirms public visibility and all four files. Remote adapter is 230,073,800 bytes; SHA-256 matches the local manifest: dfbf2834d883987893ec305a6093a345fd79c77f903f60f2f914cbe3b6058d1b.
• Card source: weights/selfjev_4b/README.md; scores drawn from reports/selfjev_4b_treeserver/. No new training, eval, paid services, or local git commit. Cost $0. Verdict: publication completed.
• User subsequently asked how to sign in with another Hugging Face account. No repository was deleted or moved.

2026-09-28 09:56 PDT: theme-colored website code examples (Codex syntax, user request)¶

• Added Prism token rendering for shell, Python and JSON examples, including valid JSON request bodies inside single-quoted shell strings. CSS syntax variables match the existing dark/olive palette; keys, values, commands, numbers and comments have distinct colors. The homepage Python example uses the same renderer.
• Copy still receives the original source string; unknown/plain-text languages remain readable, and token text is escaped by React.
• Validation: npm --prefix website run check passed (production build, TypeScript, 15 exported pages / 417 links); all 20 fenced examples across ten docs pages match their source text exactly after HTML rendering. Browser confirmed shell/JSON key colors, Python tokens, Copy success feedback, and no API-page console errors. Browser clipboard readback returned an empty value, so byte-for-byte clipboard verification was unavailable.
• Cost $0; no model jobs, reports or score changes. Verdict: ready in the local website; no commit or deployment.

2026-09-28 09:54 PDT: clarify self-hosted API key; hold Hugging Face adapter release pending rights clearance (Codex API/HF)¶

• Traced authentication to src/selfjev/server/app.py and src/selfjev/client.py: the operator chooses SELFJEV_API_KEYS on the server; the client sends the same value as api_key / SELFJEV_API_KEY in a Bearer header. With no server keys, there is no auth. The AWS helper generates a key by default and saves it in its deployment record. Clarified the website's code example, quickstart and API reference, plus docs/api.md and docs/deploy.md.
• Audited the current adapter's manifest and hash: 230,073,800-byte LoRA, SHA-256 dfbf2834d883987893ec305a6093a345fd79c77f903f60f2f914cbe3b6058d1b, trained with 0.5 × checked labels + 0.5 × stored Jev output probabilities. TypeSafe's published [Master Customer Agreement §2.3(b)] (https://typesafe.ai/legal/mca) restricts using Service outputs for model distillation, training an imitator, or developing a similar product. No documented exception was found. Drafted weights/selfjev_4b/HF_CARD_DRAFT.md with exact provenance, usage and limitations; did not upload weights or create a Hugging Face repository. Public release awaits documented permission or a newly trained and reviewed adapter without Jev outputs.
• Validation: npm run check passed (15 HTML pages, 417 internal/source links); git diff --check passed; local docs route returned HTTP 200. No model jobs or paid services. Cost $0. Verdict: API-key question resolved; current weight publication not cleared.

2026-09-28 00:49 PDT: visitor-focused website language and live local refresh (Codex website, user request)¶

• Reframed the marketing and research pages around text decisions, AI response review, and broader text tasks; explained what accuracy measures before showing results. Simplified the architecture/demo and speed labels, curated the homepage comparison, and moved dataset IDs, technical methodology, and the full experiment archive into expandable sections. The experiment trail now explains lessons in plain language.
• Evidence remains read from the same report files in website/lib/evidence.ts; no scores, model reports, training data, or research conclusions changed. AI-label limitations and historical/current speed distinction remain visible.
• Fixed the non-refreshing local website: replaced this session's static serve out process with npm run dev at http://127.0.0.1:3000. Browser verification observed a source edit appear without reloading and Fast Refresh completion. Documented development versus static preview commands in website/README.md.
• Validation: TypeScript and production build passed; export audit passed for 15 HTML pages and 416 links. Browser checked task tabs and source links, expandable methodology, archive search, and desktop/mobile layouts (1440 / 390 / 320 px, no page overflow); no browser console errors. Development server left running.
• Cost $0; no model jobs or paid services. Verdict: ready for review, same visual direction with clearer explanations.

2026-09-27 23:51 PDT: Next.js marketing, evidence explorer, and practical docs (Codex website, user request)¶

• What / why: added website/, a static Next.js 16 site with an original dark/orange instrument design, interactive decision examples, shared-prefix architecture, a searchable 56-run benchmark explorer, historical H100 latency controls, and ten practical guides (quickstart, hardware, API, architecture, fine-tuning, Docker, AWS, Runpod, GCP, operations). The existing Zensical research notebook and deployment workflow remain intact. README links the new local site.
• Evidence: benchmark values read from report JSON at build time; per-test source links. Current engine headlines: 95.7 eval2 / 93.1 eval_llm (reports/selfjev_4b_treeserver/), separately labeled original engine 95.8 (reports/qwen35_4b_tree_scratch_jevall_/eval2/report.json). Historical latency comes from reports/latency/requests_h100.jsonl, medians over reps 1–10 with warm-up excluded; explicitly archived Qwen3/H100, never attributed to current TreeServer. No new model results.
• Hardware finding: current serving CLI is CUDA-only (no CPU flag); no validated full-model CPU minimum or timing. 24 GB VRAM / 4 vCPU / 16–32 GB host RAM is planning guidance; 16 GB VRAM is a documented, unproven workload-dependent floor. CPU 32 GB is only an experimental memory budget, not support. Runpod/GCP guides are manual and not GPU-tested.
• Validation: npm --prefix website run check passes: production build + TypeScript, 15 HTML outputs and 415 links (internal routes/anchors and linked repository source existence). Browser checked at 1440, 390 and 320 px: example changes, benchmark tabs and source URLs, search and empty state, latency controls, mobile navigation, active docs links and code copying. No page-level horizontal overflow on the checked mobile routes. git diff --check clean.
• Cost: $0 cloud/API; no training, evals, or model loads on the laptop. Verdict: ready for local review at http://127.0.0.1:3000 (static preview); public hosting unchanged, no commit. Build/run details: website/README.md.

2026-09-27 22:25 PDT: rerunning selfjev-4b's exact recipe moves the scores by about a point: single runs cannot settle ±1-point questions (fork, user request)¶

• What: selfjev_4b_repro, selfjev-4b's exact training data (80,093 rows, content-identical, same order and seed 13) trained with the restructured code (scripts/train/selfjev_4b.sh); 79,943 questions, 1,803 steps, best step 1,800 as before. Question: was selfjev_4b_v2's eval_llm drop the new code or batch 2?
• Result (reports/selfjev_4b_repro/), vs selfjev-4b: eval2 95.08 vs 95.78 (28 / 42, p = 0.12), dev benchmark 83.81 vs 83.75 (p = 0.92), eval_llm 91.97 vs 93.13 (16 / 27, p = 0.13); final validation 92.7 vs 93.7. vs v2 (same code,
• batch 2): eval2 95.08 vs 95.38 (p = 0.55), dev benchmark 83.81 vs 83.55 (p = 0.49), eval_llm 91.97 vs 90.49 (33 / 19, p = 0.07).
• Verdict: by the rule set before the results: no significant loss (p = 0.13), but eval_llm is 0.03 under the 92.0 bar (one question). Read: no evidence that the new training code differs; the same recipe run twice differs by about a point on eval2, eval_llm and validation, so selfjev-4b was a good draw. v2's eval_llm drop is part noise, part (not significant, p = 0.07) batch 2; batch 2 showed no gain anywhere. selfjev-4b stays the default (swapping to a rerun picked on test would be selection on test). Decisions on differences of about one point need two or more runs per arm (≈ $20 each). The adapter was not kept (moved to the Trash).
• Cost: ≈ $20.44 (547 min; the self-terminating driver pulled and terminated 9 min after the job ended).

2026-09-27 11:55 PDT: the new engine gives selfjev-4b's scores; v2's eval_llm drop is the model, not the engine (fork, user request)¶

• Why: selfjev_4b_v2 (JOURNAL 11:20) was scored by TreeServer, selfjev-4b by the archived forked-cache engine, and the engines had only been compared on 400 eval2 questions. Its eval_llm drop could have been the engine.
• How: weights/selfjev_4b re-scored by selfjev eval (TreeServer) on eval2, the dev benchmark and eval_llm on a g5.xlarge (A10G), compared question by question with its stored reports (L40S, old engine). Reports: reports/selfjev_4b_treeserver/.
• Result: eval2 95.68 vs 95.78 (1 / 3 questions, p = 0.62), dev benchmark 83.78 vs 83.75 (3 / 2, p = 1), eval_llm 93.13 vs 93.13 (0 / 0); decisions identical on 99.8%, 99.8% and 100% of questions (probability differences: median 0.0003–0.0006, p99 0.012–0.022). On the same engine, v2 vs selfjev-4b: eval_llm 90.49 vs 93.13 (9 / 34, p = 0.00017), eval2 95.38 vs 95.68 (35 / 41, p = 0.57), dev benchmark 83.55 vs 83.78 (60 / 68, p = 0.54).
• Verdict: TreeServer (what selfjev serve uses) serves selfjev-4b faithfully. v2 really is worse on eval_llm; what remains to explain it: batch 2's data, the restructured training code (v2 was its first GPU run) or run-to-run noise. A second seed of v2, or selfjev-4b's data trained with the new code, would separate them (≈ $21 each).
• Cost: ≈ $0.52 (31 min; a self-terminating driver: launch, evals, pull, terminate, delete SG and key).

2026-09-27 11:20 PDT: retraining selfjev-4b with batch 2 made it worse; not promoted (fork, user request)¶

• What: the selfjev-4b recipe from scratch (scripts/train/selfjev_4b.sh: new LoRA r64, lr 2e-4, texts up to 16K, 0.5 × label + 0.5 × Jev) on 83,581 questions: selfjev-4b's data plus batch mpos_distr_num_v1 (several correct answers, distractors, numbers; test-diagnosis-motivated). One L40S, 1,912 steps, best by validation loss = the last.
• Result (reports/selfjev_4b_v2/, vs selfjev-4b): eval2 95.38 vs 95.78 (33 / 41, p = 0.42), dev benchmark 83.55 vs 83.75 (p = 0.60), eval_llm 90.49 vs 93.13 (9 / 34, p = 0.0002). Confident mistakes on eval2 12 vs 11. The eval_llm loss is broad, not one family: hard and very hard verify (−9 questions), judge (−7), guardrail (−5) and very hard score multilabel (−3). Validation accuracy was lower at every late checkpoint (92.3–92.7 vs 92.9–93.7, same 1,198 questions).
• Verdict: fails the rule set before the results (eval2 ≥ 95.8 and eval_llm ≥ 92.6): selfjev-4b stays the default. More targeted data did not close the multi-positive, distractor and numeric gaps; one seed, so part of the difference may be run-to-run noise, but eval_llm's p = 0.0002 is not. The batch stays in data/all.jsonl.gz; the adapter was not kept (moved to the Trash with the run folder).
• Cost: ≈ $21.46 (g6e.2xlarge 9.6 h, incl. 23 min idle between the end of the job and the shutdown; polling every 10 min was too slow).

2026-09-27 02:20 PDT: only selfjev-4b on master, runs/ deleted, every doc page audited (fork, user request)¶

• Code (bfa0825, a195c38): Qwen3.5-4B only (the Qwen3.5-2B option and --base are gone), the unused forked-cache scorer is gone (TreeServer and vLLM serve), weights/qwen35_4b_tree left master (in history and at tag archive/pre-cleanup-2026-09-27; its reports stay), scripts that served only archived models are gone, and scripts/train/selfjev_4b.sh is the recipe (GPU preflight, train, evals).
• Bugs the audit found (e4d3c72): selfjev classify skipped the option lists selfjev-4b was trained with (the server listed them; now both do); an unhandled error was a plain-text 500 without x-request-id (now JSON with the id); .pre-commit-config.yaml had never been committed (a global gitignore excluded it). 82 CPU tests.
• runs/ (7.4 GB, 72 runs) moved to the macOS Trash; every run's train_meta.json is in reports/train_meta/ (runs/a/b → a__b.json), which the ledger now reads; its weights column names the kept adapter.
• Docs: three parallel audits of all 24 pages plus README, AGENTS, weights/ and the data catalog: stale paths, commands and claims fixed, archived models marked (with the tag's personal_jev / pjev names), the eval2 McNemar column now compares with selfjev-4b, spend recomputed from the table below (≈ $786 logged as of today), and the nav groups the archived model pages. Open: TreeServer latency has never been measured on a GPU.
• Cost: $0.

2026-09-27 01:42 PDT: on a GPU, TreeServer gives selfjev-4b's answers (preflight of the selfjev_4b_v2 retrain) (fork)¶

• Why: since the cleanup, selfjev serve and selfjev eval score with TreeServer (the shared-prefix tree, forward only); selfjev-4b's stored reports came from the archived forked-cache engine, and the restructured code had never run on a GPU. scripts/train/selfjev_4b.sh checks this before it trains.
• How: selfjev eval --adapter weights/selfjev_4b --data data/ova/eval2.jsonl --limit 400 on the L40S box i-0700e0bcadbf72ad6, compared question by question with reports/qwen35_4b_tree_scratch_jevall_/eval2/report.json.
• Result: 99.8% of the 400 decisions agree; largest per-question probability difference: median 0.0003, p99 0.0224. Latency was not measured.
• Cost: a few minutes of the retrain box (in its line). Verdict: the restructured engine is sound on GPU.

2026-09-27 01:40 PDT: repo cleanup and the product: selfjev package, SDK, server with Jev's API, fine-tuning over HTTP, deployment (fork, user request)¶

• Cleanup (ead0a73): the dead-end models (custom cross-attention, jina, T5Gemma, compact tree, option pointers, teacher distillation), one-off scripts and caches are gone (562 files changed, 170,519 lines deleted). Everything is at tag archive/pre-cleanup-2026-09-27, where the package is still personal_jev and the CLI pjev.
• Package (0bb21c7, 020cbc3, cce9061): personal_jev → selfjev 0.2.0 (CLI selfjev), split into core / engine / training / evaluation / server / data / deploy with tests mirroring it; ruff lint + format, pre-commit, CI.
• SDK and server (ac4f97e): SelfJev / AsyncSelfJev (httpx + pydantic only, retries, typed errors) and a FastAPI server with Jev's request and answers at /v1/systemone, /api/alpha/decisions and /v1/decisions (plus multi), cross-request batching, Bearer keys, /health, Prometheus /metrics. Contract: api.md.
• Deploy (b5abc75): selfjev deploy aws up|down|status|list|machines (NVIDIA DLAMI, systemd, API key, cost cap; default g6.xlarge $0.805/h) and a Dockerfile. deploy.md.
• Fine-tuning over HTTP (c431591): /v1/files and /v1/fine_tuning/jobs in OpenAI's shape, supervised or RLCD. Training files are Jev-shaped requests with answers; jobs run selfjev finetune|rlcd one at a time; a finished job's adapter is served next to selfjev-4b under its own model name. selfjev serve --fine-tuning.
• Scripts (b9bcc24, 1064ccd): into scripts/{data,eval,train,aws,docs}, shared code into selfjev.data.providers and selfjev.data.catalog.
• Checks: 83 CPU tests, ruff clean, the docs site builds, the SDK wheel installs without torch. Not run on a GPU yet: the server, the AWS deploy and fine-tuning jobs over HTTP (a g6.xlarge trial ≈ 1 h ≈ $0.80 needs the user's OK).
• Heads-up for other sessions: selfjev.data.providers now asks for max_tokens 8192 (was 6000; another session's edit, kept). Request bodies changed, so reruns of compare_external.py miss their cache and pay again.
• Cost: $0.

2026-09-27 01:20 PDT: mpos_distr_num_v1 built: 3,645 verified several-correct, distractor and number questions ($19.81) (fork)¶

• Why: on eval2 selfjev-4b still makes far more real errors than Jev on several-correct-answer questions (29 vs 13), near-miss distractors (29 vs 14) and numbers (28 vs 18). Test-diagnosis-motivated (the eval2 audit, reports/audit_2026-09-26/AUDIT.md): the writer saw only the abstract trap names, never test items.
• How: gen_hardcases.py --batch mpos_distr_num_v1 --round3 --traps multi_positive,distractor,numeric_reasoning (GPT-6 Luna, 1,671 texts from 8 to 8,192 tokens, 4,041 questions), then scripts/data/grow_batch.sh judge / build / finish: blind GPT-6 Astra judge, strict build with the overlap guard against every test set, OpenAI moderation, Jev.
• Result: writer = judge on 95.6% (distractor 97.4, multi_positive 95.4, numeric 94.5); 3,645 of 4,041 questions kept (train 3,282, validation 363; multilabel 1,471, binary 1,299, multiclass 875): 179 dropped where the judge disagreed, 217 more by --strict because another question of the same text did. Moderation flagged 0 of 1,671 texts. Jev gets 94.8% right. data/all.jsonl.gz now has 93,237 questions. Review: data/batches/mpos_distr_num_v1/review/JUDGE.md.
• Cost: Luna $1.87 + Astra batch judge $17.80 + Jev $0.14 = $19.81 (quoted ≈ $19; the judge ran ≈ $2 over my estimate).
• Verdict: data only; selfjev-4b does not include it. Using it takes one more retrain (≈ $21, ≈ 9 h), not started.

2026-09-26 21:40 PDT: selfjev-4b is the default model (fork, user request)¶

• The from-scratch retrain's best checkpoint (runs/qwen35_4b_tree_scratch_jevall/adapter, JOURNAL 21:25) is now weights/selfjev_4b (name selfjev-4b, model.json, adapter sha256 dfbf2834…, Git LFS): eval2 95.8, eval_llm 93.1, dev benchmark 83.8. README, AGENTS, docs (index, findings box 1, leaderboard, reproduce), weights/README.md and the code examples point to it; qwen35_4b_tree stays in weights/ as the previous default (best on the dev benchmark, 84.4).
• AWS: the retrain box's SG and key pair deleted after the SSO login; no selfjev-jev-* resource left in us-east-½, us-west-2.

2026-09-26 21:25 PDT: retrain from scratch on everything with Jev targets: eval_llm 82.1 → 93.1 (Jev 92.5), eval2 95.78, dev benchmark −0.7 from Jev's targets on emotions (fork)¶

• What: Qwen3.5-4B, new LoRA r64, lr 2e-4, 1 epoch (1,803 steps), pjev finetune on 79,943 non-test questions of data/all.jsonl.gz (every train and validation row of hardcases, hardcases_r3, hardcases_llm, hf, synthetic, llm_multilabel_v1, numdate_neg_v1, minus the 1,200-question validation sample), texts up to 16K tokens (--max-length 16384 --batch-tokens 16384 --grad-accum 2: the 8K default dropped 2,548 questions, mostly round-3 long texts; only 150 dropped now), target 0.5 × label + 0.5 × Jev. One L40S, scripts/jev_soft_box.sh scratch. Validation: 78.2 → 93.66 (best, step 1,800; qwen35_4b_tree 92.9 on the same code).
• Results (best checkpoint; the last, 3 steps later, scores 95.63 / 83.84 / 93.13):

eval2 calibration: Brier 0.0466 (original 0.0480), ECE 0.026, confident mistakes 11 (original 30, Jev 7), mean confidence when wrong 0.676 (Jev 0.670). Wrong questions 84 (original 88, Jev 55): multilabel 38 → 33, binary 31 → 33, multiclass 19 → 18. - Reading: - eval_llm, the clean check (not mined for our errors): +11.0 over the original and level with Jev (93.13 vs 92.49, p = 0.52), multilabel EM 86.8 vs Jev 81.3. This is the LLM-evaluation data (hardcases_llm, llm_multilabel_v1), which the original never saw; B (same data, gentle update) got 90.06. - eval2: best of our models, not significant (+0.2). The 16K texts fixed long-text errors (8 → 3); double negation, temporal and distractors improved; multi-positive and numeric did not. - Dev benchmark −0.7 (p = 0.13) is almost all hf_emotions_multilabel (−27 questions): Jev is weak on that subjective public set (68 real errors in the audit), and half-weight Jev targets pulled the model toward it. Next time: --soft-weight 0 on hf rows. - Files: reports reports/qwen35_4b_tree_scratch_jevall_[_last]/ (eval2, test, eval_llm), baselines on eval_llm reports/qwen35_4b_tree/eval_llm, reports/qwen35_4b_tree_sft_jevall__last/eval_llm; logs and train_meta reports/rlcd_jev_2026-09-26/; adapters runs/qwen35_4b_tree_scratch_jevall/adapter (best) and adapter_last (not in weights/). - Cost: ≈ $20.80 (L40S, 18:59–04:17 UTC, 9.3 h incl. a 25-minute 8K start that was restarted at 16K). The AWS SSO token expired before terminate-instances: the box was shut down from inside (terminate on shutdown); its SG and key pair still need deleting. - Repo note: after the 16:48 LFS commit (efe3a5c) and merge, 97 data files in this checkout were LFS pointers (e.g. data/eval2.jsonl, data/ova/*.jsonl), which broke eval2_summary.py; git lfs checkout restored them from the local store. - Verdict: a better model on the LLM-evaluation use cases and eval2, a worse one on emotions. Candidate new best, pending the user's call; the fix for the dev benchmark is Jev targets only where Jev is reliable.

2026-09-26 15:15 PDT: C: RLCD with a confident-mistake cost: 8 confident mistakes on eval2 (Jev 7), but only by being less sure overall; at equal coverage no better than B (fork)¶

• What: pjev rlcd --reward log=1,brier=1,spherical=1,confident_miss=5 from B's result (runs/qwen35_4b_tree_sft_jevall/adapter_last), the same 69,528 questions, Jev targets (0.5), lr 2e-5, KL 0.2, 1,230 steps, one L40S. confident_miss: −1 for every decision ≥ 0.9 sure that the target says is wrong, not differentiable, so only the sampled-report policy gradient can optimize it.
• Results (final checkpoint; best-by-validation was step 0, B itself):

• Reading:
• The cost does what a fine-tune cannot: confident mistakes 14 → 8 on eval2 (Jev 7) and 75 → 48 on the dev benchmark (Jev 253), mean confidence when wrong 0.690 (Jev 0.670), ECE 0.043 (Jev 0.041), at the same accuracy (vs B 9 / 10 and 17 / 17).
• The price is sharpness: Brier 0.0438 → 0.0518 on eval2 (worse than the start's 0.0480), because the model also backs off on answers it gets right. B has the best probabilities; C has Jev's caution.
• The first RLCD result that differs from fine-tuning, and only because its reward is a decision cost, not a proper score.

• Correction (same day, equal-coverage check): At equal coverage (acting on the same share of decisions), C is no better than B: its 8 confident mistakes at ≥ 0.9 come from being ≥ 0.9 sure on only 77% of decisions (B 88.8%, Jev 87.7%). A higher threshold on B gives the same trade for free. B ranks better than the original at every coverage; Jev ranks far better (8 mistakes at 92% coverage, B 20).

  mistakes among the k most confident decisions (eval2, 3,491)	70%	77%	85%	88%	92%
  qwen35_4b_tree	3	8	14	16	29
  B	4	5	9	13	20
  C	4	8	10	12	24
  Jev	3	3	4	7	8

• Files: reports/qwen35_4b_tree_cost_jevall__last/, logs and train_meta in reports/rlcd_jev_2026-09-26/, adapter runs/qwen35_4b_tree_cost_jevall/adapter_last.

• Cost: ≈ $15.90 (L40S, 7.1 h). Box terminated; SG and key pair deleted.
• Next: the from-scratch retrain (In flight) can get the same C stage afterwards; a smaller weight (confident_miss=2) may keep more of B's Brier.

2026-09-26 12:05 PDT: numdate_neg_v1 built: 549 verified double-negation, number and date questions ($3.08) (fork)¶

• Why: the eval2 failure audit (reports/audit_2026-09-26/AUDIT.md): our real errors vs Jev's on double negation 7 vs 1, temporal 28 vs 21, numeric 26 vs 18. Motivated by a test-set diagnosis: the writer saw only the abstract trap names.
• What: gen_hardcases.py --batch numdate_neg_v1 --round3 --traps double_negation,numeric_reasoning,temporal_reasoning (new --traps: only those focus traps, balanced, one per question on every tier), GPT-6 Luna through the OpenAI API: 264 texts, 643 questions, 8 to 8K tokens, tiers balanced. Blind Astra batch judge: author = judge 93.0% (double negation 97.9, numeric 91.0, temporal 90.8). Strict build + test-overlap guard: 549 kept (train 508, validation 41; binary 273, multiclass 169, multilabel 107). Moderation: 0 of 264 texts flagged. Jev column added: Jev right on 92.9%.
• data/all.jsonl.gz now 89,592 questions, all with jev.
• Cost: Luna $0.28 + Astra $2.78 + Jev ≈ $0.02 = $3.08.
• Next: the from-scratch retrain on every non-test question (80,093 incl. llm_multilabel_v1 and this batch), In flight.

2026-09-26 11:19 PDT: test-failure audit complete; eval2 labels are sound, our real gaps vs Jev are distractors and multi-positive; Jev on the multilabel batch ($2.72)¶

• Owner: Synthetic data generation strategy. The OpenRouter limit was raised by the user.
• Jev step of llm_multilabel_v1: $0.30, 0 errors; all 89,043 rows of data/all.jsonl.gz now have jev. Jev gets 86.1% of the batch's multilabel questions exactly right (eval2 multilabel EM: 94.2): the batch is hard for Jev too.
• Audit part 2 (scripts/audit_failures.py, Opus 5.5 blind, $2.42; total audit $4.92): all 968 failed test questions relabelled. reports/audit_2026-09-26/AUDIT.md.
• eval2: 114 of 119 failures are real model errors; 5 proposed label errors: eg-0024-q1 clear (Opus 0.9, = ours), four debatable Kimi-written binary questions phrased "Which is true about …?" (Opus 0.5-0.6). eval2 stays frozen; errata list in errata_candidates.jsonl.
• Our real eval2 errors (86): by trap distractor 33, multi_positive 28, temporal 28, numeric 26, paraphrase 17, exception 13, role_reversal 11; by type multilabel 37, binary 30, multiclass 19; very hard 43, hard 35, simple 8. Jev's real errors (50): temporal 21, numeric 18, distractor 13, multi_positive 13. Where we lose most to Jev: distractors (33 vs 13), multi-positive multilabel (28 vs 13), long texts (8 vs 2), double negation (7 vs 1), role reversal (11 vs 5). Temporal and numeric reasoning are weak for both.
• eval_llm (Jev only; ours not scored yet): 68 of 71 Jev failures are real; worst traps multi_positive 21, numeric 20, flawed_step 14, benign_lookalike 12, judge_injection 10.
• Dev benchmark: 317 of 778 failures (41%) look like label problems (both models wrong: 224 of 362); real errors sit in subjective public sets (emotion, tweet sentiment, SST-2, BoolQ): not a data-generation target.
• This is a test-set diagnosis: any data built from it is labelled as such, the generator only sees abstract pattern descriptions, and eval_llm (not mined for our errors) is the clean check.
• Next (proposed, needs OKs): retrain with llm_multilabel_v1 first (it already targets near-miss distractors and multi-positive), then a small temporal/numeric batch if those errors remain.

2026-09-26 09:14 PDT: llm_multilabel_v1 built: 7,570 verified LLM-oriented multilabel questions ($68.96, cap $70)¶

• Owner: Synthetic data generation strategy (a fork of it also worked on this batch by mistake, then handed it back; its parts are counted here). Why: multilabel is the largest gap (eval2 multilabel EM 90.1 vs Jev 94.2); our best model's wrong multilabel answers on eval2 mostly add a near-miss candidate (26 of 38).
• What: gen_hardcases.py --usecases train --multilabel --batch llm_multilabel_v1 (every question multilabel, 4-8 candidates, ≥ 2 near-miss negatives and ≥ 1 implicit positive per question, 0 / 1 / 4+ positives spread, LLM question menus per use case). Writers: Luna 2,460 questions (1,065 through the OpenAI API directly, fork's --openai), Gemini 3.8 Flash 1,726 sync + 4,652 through Google's Batch API (scripts/gen_gemini_batch.py, first live use, 3 jobs), Grok 4.7 448 (stopped on the user's call). Blind Astra batch judge on 9,286 questions: 89.7% agree (Gemini batch 92.0, Gemini sync 91.9, Grok 91.1, Luna 83.5).
• Kept (strict build, overlap guards): 7,570 questions (train 6,803 / validation 767), data/batches/llm_multilabel_v1.jsonl, now in data/all.jsonl.gz. Use cases 1,436-1,591 each; positives 0: 1,057, 1: 1,160, 2: 1,815, 3: 1,596, 4: 1,490, 5+: 452; 5.0 candidates on average. Multilabel training questions in the combined file: 15,776 → 22,579.
• Safety: moderation screen (free) flagged 254 of 4,364 texts; all read; 11 removed from the raw files and the raw outputs (self-harm method details, an assistant reply encouraging suicide, a graphic torture scene). Kept: harmful requests, injections, crisis messages without method details, insults, fantasy violence.
• Incidents: the OpenRouter key hit its monthly limit (403) and the account ran out of credits (402) mid-run; Gemini moved to Google's Batch API (half price, ≈ $2.3 per 1K questions vs ≈ $7 through OpenRouter) and Luna to the OpenAI API. The first Grok writer died when the fork stopped it; gen_gemini_batch.py collect crashed on a bare string (fixed by the fork); Gemini-batch rows are tagged synthetic:google-batch/.
• Cost: writing $26.60 (Gemini sync $12.01, Grok $5.21, Luna $1.14, Gemini batch $8.24) + Astra judge $42.36 = $68.96. Pending: Jev predictions for the batch (≈ $0.30), blocked by the OpenRouter limit.
• Next: retrain with it (AWS, needs a price OK) and compare multilabel EM on eval2 and eval_llm.

2026-09-26 09:10 PDT: Jev's probabilities as soft targets on all data: the fine-tune halves confident mistakes on eval2; RLCD adds nothing over it (fork)¶

• What (user request: RLCD with soft targets, "we kinda distill Jev", on all the data): from weights/qwen35_4b_tree, one epoch over 69,528 non-test questions of data/all.jsonl.gz (every train and validation row of hardcases, hardcases_r3, hardcases_llm, hf and synthetic, minus the model's 1,200-question validation sample and 2,446 questions over 8K; scripts/jev_soft_targets.py). Target: 0.5 × verified label + 0.5 × Jev's probabilities (--soft-weight 0.5: the label stays the argmax, so Jev never decides a label). A = pjev rlcd (log + Brier + spherical, 8 samples, sd 0.3, KL 0.2), B = pjev finetune (soft cross-entropy), both lr 2e-5, 1,230 steps, one L40S each (scripts/jev_soft_box.sh). No H100 capacity in us-east-2, us-east-1 or us-west-2.
• Jev on these questions: probability of the verified answer ≥ 0.9 on 77.3%, 0.5–0.9 on 14.8%, < 0.5 on 7.8%.
• Results, final checkpoints. Best-by-validation picked step 0 in both runs (hard-label validation loss and Brier penalize hedging); both boxes re-scored it and got the start's 95.58 / 84.44 exactly (kept outside the ledger: runs/rlcd_jev_start_recheck/).

• Reading:
• Soft targets deliver the hedging hard labels could not: confident mistakes on eval2 30 → 14 (Jev 7), mean confidence when wrong 0.755 → 0.708, Brier −9%, at the same accuracy (eval2 +0.15, p = 0.74: multiclass 96.8 → 97.5, binary 96.9 → 96.8, multilabel EM 90.1 → 90.3). The dev benchmark is flat (84.41 vs 84.44; 83 → 75 confident mistakes, Brier slightly worse).
• ECE rises (0.005 → 0.025) by design: the model keeps some doubt on answers it gets right, as Jev does (0.041).
• RLCD on the same targets moved less (KL anchor, noisier gradient) and beats the fine-tune on nothing but ECE: A vs B 4 / 10 on eval2 (p = 0.18), 25 / 22 on the dev benchmark (p = 0.77). Expected: its three rewards peak at the same target as the fine-tune's cross-entropy.
• Files: reports reports/qwen35_4b_tree_{rlcd,sft}_jevall__last/; logs, box logs, train_meta reports/rlcd_jev_2026-09-26/; adapters (not in weights/) runs/qwen35_4b_tree_{rlcd,sft}_jevall/adapter_last.
• Cost: A ≈ $16.20 (7.2 h), B ≈ $13.50 (6.0 h), Jev $0 (already in all.jsonl.gz). Boxes terminated; SGs and key pairs deleted, including those of the failed H100 attempts.
• Verdict: Jev's soft targets are the useful part; RLCD's sampling adds nothing when the reward is a proper score of the same target. Running (user request): C = RLCD from B's result with a 5× cost per confident mistake (--reward log=1,brier=1,spherical=1,confident_miss=5), the one reward a fine-tune cannot express.

2026-09-26 08:59 PDT: test-failure audit, part 1: on the public dev-benchmark rows, 43% of failures look like label problems ($2.50)¶

• Owner: Synthetic data generation strategy (user request). scripts/audit_failures.py: every test question that qwen35_4b_tree or Jev gets wrong (968 questions in 926 texts: eval2 119, dev benchmark 778, eval_llm 71 Jev-only), blind relabel by Claude Opus 5.5 (effort medium, never sees the label or any model answer), then sorted.
• Partial: 602 texts relabelled, all public-set rows of the dev benchmark; the OpenRouter key then hit its monthly limit (403) and the account ran out of credits (402), so eval2, eval_llm, the authored eval rows and 155 public rows are pending.
• Result (reports/audit_2026-09-26/AUDIT.md), 602 dev-benchmark failures: label right / probably wrong / ambiguous = ours only 82 / 21 / 7, Jev only 124 / 59 / 29, both wrong 71 / 180 / 29. Of 260 proposed label errors, 84 are clear (Opus confidence ≥ 0.8; mostly tweet sentiment 27, GoEmotions 21, AG News 11, emotion 10) and 176 debatable. Spot checks: genuinely subjective public labels ("i do feel stressed": sadness vs fear; an Iraq stock exchange story: world vs business).
• Verdict so far: when both models miss a public-set question, the label is usually the problem. Real errors of ours cluster in tweet sentiment (42), emotion (32), GoEmotions multilabel (23), TREC (18), CLINC (13): subjective public sets, not a data-generation target. The target-task sets (eval2, eval_llm) are the ones still to audit.
• Errata candidates for human review: reports/audit_2026-09-26/errata_candidates.jsonl; frozen files unchanged.

2026-09-26 01:50 PDT: the speed gap was hardware: on one H100 the 4B tree beats Jev's server time (fork 2 session)¶

• User-approved open idea D (spot first, then on-demand, then g7e). The spot p5.4xlarge (1× H100 80 GB) was fulfilled at the first try in us-east-2b at $2.544/h (spot placement score 9/10 that morning; every 2026-09-24 launch had failed). Box i-046f7d9cd8bde18bd, DLAMI Ubuntu 22.04 (driver 595.91.07), vLLM 0.30.0, torch 2.13+cu130. Setup to servers up: 9 min. Terminated 08:43 UTC; SG and key pair deleted; nothing of this session runs in AWS.
• What ran: tree_4b_combo merged (vllm_tree merge), served on vLLM in bf16 (:8002) and with --quantization fp8 (:8003, new flag on vllm_tree serve and scripts/eval_vllm.py), both resident (0.42 of the GPU each); the same latency sweep as before from the Mac through an SSH tunnel (scripts/latency_sweep.py sweep, 10 rounds, Jev in random order with ours; reports/latency/requests_h100.jsonl); then eval2 through vLLM in both precisions and the batched throughput benchmark. RTT: OpenRouter 11 ms, box 61 ms.
• Latency, p50 ms, wall at the client / server inside the box (Jev: inside OpenRouter):

• Inside the machine the H100 is 2–5× faster than Jev's server time in every cell but 4,096 tokens × 16 questions (189 vs 134 ms). End to end we trail by 10–100 ms: the 61 ms hop to Ohio against 11 ms to OpenRouter's edge.
• FP8 changes nothing: the 4B is overhead-bound on an H100 at these sizes. Fit, one question, server-side: ours ≈ 20 ms + ≈ 15 ms per 1,000 text tokens (≈ 68K tokens/s); Jev 132–137 ms + 2.2–2.6 ms per 1,000 (≈ 400K tokens/s). Jev's per-token cost is still ≈ 6× lower, so a smaller model or more GPUs per request; below 4K tokens the fixed costs decide.
• Accuracy of the serving paths on eval2 (reports/tree_4b_combo/eval2_vllm_h100{,_fp8}; transformers reference 94.48): vLLM bf16 94.42 (5 of 1,991 decisions differ, 3 / 2), vLLM FP8 94.48 (25 differ, 9 / 9; multilabel EM 86.6 vs 85.9). FP8 costs no accuracy. All 1,991 questions in 18 s (bf16) and 16 s (FP8); the A10G took 144 s.
• Throughput, GPU fully busy (reports/latency/throughput_h100.json, cost at $2.63/h in the table; actual spot $2.54): 294 requests/s at 8 tokens × 1 question ($0.0025 per 1,000 vs Jev $0.016), 105 req/s at 512 × 1 ($0.007 vs $0.038), 5.9 req/s at 4,096 × 16 ($0.124 vs $0.265). Cheaper than Jev in every cell, 2.1–6.5×, on spot. At the on-demand price ($6.88/h) multiply ours by 2.6: still below Jev everywhere except within 10% at 4,096 × 16.
• Verdict: the speed gap was hardware, not architecture. No speed-driven architecture work is needed; speed is a deployment question (H100-class GPU, placement near the client). The open work is accuracy (memo proposals A–C, E).
• Process notes: (1) pkill -f "<pattern>" inside an ssh host '<command>' kills the remote shell itself when the pattern appears in the command string, and vLLM's EngineCore child keeps the GPU memory after its parent dies: free the GPU by nvidia-smi --query-compute-apps=pid PIDs instead. (2) vLLM 0.30 aborts at interpreter exit after the throughput run ("terminate called without an active exception") once results are written; harmless.
• Cost ≈ $1.20 (spot 28 min $1.18 + Jev $0.02). Docs updated: findings 19–21, speed.md (new H100 section), ledger conclusion 11, open idea D done, landscape, next, the memo.

2026-09-26 01:34 PDT: Jev's prediction on all 81,473 questions; one dataset that grows by batches ($1.95)¶

• Owner: Synthetic data generation strategy. User: store Jev's probabilities once per row (no recompute, usable for fine-tuning); always use one dataset and grow it instead of creating a new one each time.
• Jev: scripts/jev_predictions.py --call answered the 33,707 texts not in any cache, $1.95, 0 errors; with the 16,543 cached questions every row of data/all.jsonl.gz now has jev (P(yes) or per-candidate probabilities, answer, confidence, correct, version typesafe/jev-1.13-20260917). Stored in data/jev/predictions.jsonl + data/jev/cache.jsonl.
• Jev vs our labels (from the stored rows; test sets for reporting only): hf 75.2, synthetic 99.0, round 2 95.7, round 3 96.4, hardcases_llm 94.9, eval 96.7, eval2 97.2, eval_llm 92.5, compact challenge 100.0. Checks: dev benchmark 82.7 and eval2 97.2 equal the reports. Caveat for distillation: on the public sets Jev disagrees with the human labels on a quarter of the questions, so a Jev teacher term would pull against the gold labels there.
• Growth: gen_hardcases.py --batch <name> writes data/batches/<name>/raw; scripts/grow_batch.sh <name> judge|build|finish judges blind, builds strict with overlap guards and a moderation screen, adds Jev predictions and rebuilds; build_all.py discovers data/batches/*.jsonl by itself. BoolQ curve rows moved out of the combined file (BoolQ is a held-out dev-benchmark family). AGENTS.md, README, docs/data.md, data/README.md and memory updated.

2026-09-26 01:23 PDT: Jev's prediction stored per question (jev field in data/all.jsonl.gz), 16.5K from cache ($0)¶

• Owner: Synthetic data generation strategy; the user wants Jev's probabilities stored once per row, to stop recomputing them and to use them for fine-tuning. scripts/jev_predictions.py writes data/jev/predictions.jsonl (id, P(yes) or per-candidate probabilities, answer, confidence, correct, Jev version, request hash); build_all.py joins it as jev.
• Free pass: 16,543 questions recovered from the caches of earlier Jev runs (dev-benchmark test, round-2 judge, eval2, eval_llm). Check: Jev on the dev benchmark 82.7% and eval2 97.2% from the stored rows, identical to the reports.
• Missing: 67,930 questions in 36,707 texts (hf train/val/cal, synthetic, round 3, hardcases_llm, BoolQ curve, eval val/cal, compact challenge, 6 changed eval_llm rows), estimated ≈ $2.12. Awaiting the user's price OK.

2026-09-26 01:18 PDT: one catalog and one file for all our data ($0)¶

• Owner: Synthetic data generation strategy; the user found the data scattered. scripts/build_all.py writes data/all.jsonl.gz (every original question once, 84,473 from 10 datasets, each row tagged dataset, own split kept; 85 MB, gitignored, rebuildable) and the generated catalog data/README.md: what each dataset is, used as (training / dev benchmark / frozen test), questions, texts, splits, types, sha256, and the derived files left out on purpose (hardcases_nb, ova, ptr, curve subsets, distill, teacher, fixtures).
• personal_jev.data reads .gz and accepts the dataset field, so load(["data/all.jsonl.gz"]) works.
• Linked from README, AGENTS.md and docs/data.md. Rerun the script after adding a dataset (edit its DATASETS list).

2026-09-26 01:10 PDT: LLM-evaluation data made safe: strict labels, moderation review, PII fix. Now 9,443 training / 946 test ($0)¶

• Owner: Synthetic data generation strategy; the user asked for these data to be safe. Supersedes the counts of the entry below. Details: llm_eval_data.md.
• Labels: --strict in build_hardcases.py and build_eval2.py drops a whole text when any of its questions was flagged by a judge (training −897 sibling questions, test −160).
• Content: all 4,732 sources through OpenAI moderation (free, scripts/moderate_texts.py); the 169 flagged texts were read. 11 more texts removed (13 in all): working attack code, weapon and hazmat smuggling tips, a step-by-step fraud plan, self-harm encouragement, self-harm method details. Raw files edited and tracked generation caches scrubbed.
• PII: 7 SSN-shaped numbers in issuable ranges rewritten to the never-issued group 00; no public figures.
• Result: data/hardcases_llm.jsonl 9,443 questions / 3,654 texts (sha256 b694f5a0…), data/eval_llm.jsonl 946 questions / 317 texts (sha256 6cfc3e16…). Jev on eval_llm 92.5% (verify 86.7, multilabel EM 81.3).

2026-09-25 23:45 PDT: first RLCD test: no gain in accuracy or calibration (fork session)¶

• What: pjev rlcd from weights/qwen35_4b_tree (reward log + Brier + spherical, 8 sampled reports per question, sd 0.3), validated on the fine-tune run's 1,200 validation questions. Logs and settings: reports/rlcd_2026-09-25/.
• Take 1, on 16K of the model's own training questions (lr 5e-5, KL 0.05): worse on every validation measure by step 100 of 504 (accuracy 92.74 → 92.05, cross-entropy 0.128 → 0.231, ECE 0.005 → 0.024, confidently wrong 15 → 40). Stopped. On questions it already fits, a proper score keeps pushing the given answer toward 100%, like a second fine-tuning epoch.
• Take 2, on 4,412 questions it never trained on (validation-split rows outside the 1,200-question sample), lr 1e-5, KL 0.2, against a plain fine-tune control on the same questions and settings. Both select their last step.

Dev benchmark: start 84.44 (ECE 0.0056, 83 confident mistakes), RLCD 84.36 (ECE 0.020, 118), control 84.50 (ECE 0.011, 91). Reports reports/qwen35_4b_tree_rlcd_fresh_/, reports/qwen35_4b_tree_sft_fresh_/. - Reading: - Our model is already better calibrated than Jev by ECE (0.005 vs 0.041); what Jev has is fewer mistakes and hedged ones (7 of 57 at ≥ 0.9 against our 30 of 99). With one hard label per question, a proper-score reward gives the same signal as cross-entropy, only noisier, so it cannot teach "be less sure here"; RLCD came out slightly more confident when wrong. - The Gaussian sampling of reports also biases the optimum toward overconfidence as the noise grows (CPU check, 70% base rate: sd 0.3 → 0.695, sd 0.6 → 0.717, sd 1.0 → 0.754); small at the sd used, so not the main cause. - Cost: ≈ $6.20 (L40S 2.8 h; the duplicate evals of identical checkpoints were skipped). - Verdict: RLCD v1 as implemented is not worth running on hard labels. For it to matter the reward has to carry something labels don't: soft targets (the judges' agreement, a teacher's probabilities) or a cost that punishes confident mistakes more than it rewards confident rights.

2026-09-25 23:40 PDT: LLM-evaluation data built: 10.4K training questions + frozen 1.1K test set eval_llm ($96.38)¶

• Owner: Synthetic data generation strategy. The user asked for ≈ 2K synthetic questions per use case of Jev's pitch ("score, judge, verify, guardrail, and detect jailbreaks of LLM prompts, reasoning traces, and/or outputs"). Before this, ≈ 6% of round-2/3 texts and 3% of eval2 were LLM artifacts; the dev benchmark's eval_agent_output slice (26 q) had the tree at 61.5 vs Jev 92.3. Write-up: llm_eval_data.md.
• Training: data/hardcases_llm.jsonl, 10,369 verified questions / 4,243 texts (train 9,333 / validation 1,036, sha256 cf46e0cf…); judge 2,040, guardrail 2,101, score 2,053, jailbreak 2,084, verify 2,091; ≈ 1K per length step 8 → 8K tokens. Writers Luna / Gemini 3.8 Flash / Grok 4.7 / DeepSeek V4 Flash via gen_hardcases.py --usecases train (brief data/hardcases/BRIEF_llm.md). Blind Astra batch judge: 92.3% agree (Grok 97.7, Gemini 96.9, Luna 93.1, DeepSeek 73.7). data/hardcases_llm/review/.
• Test: data/eval_llm.jsonl, 1,106 questions / 398 texts, frozen like eval2 (sha256 6f163506…): eval2's writers (Opus 5.5, Kimi K3, GLM 5.3), a third of the calls on LLM applications absent from training, judges Astra 94.5% and Claude Sonnet 5 95.1% (not Gemini 3.1 Pro: the user's call), kept only when both agree with the author. Jev scores 93.3% (verify 88.6, guardrail 93.8, score 93.8, jailbreak 94.7, judge 95.3; multilabel EM 83.2). data/eval_llm/REVIEW.md, review/SPOTCHECK.md.
• Safety scan of the kept texts: two working exploit snippets (lgf-0135 reverse shell, lgf-0251 netcat injection) removed from the raw file and scrubbed from reports/hardcases_llm/gen_cache/lgf.jsonl. 0 texts dropped by the 8-gram overlap guards.
• Not done: no model trained on it or scored on eval_llm yet. Next: baseline the best models on eval_llm, then retrain with hardcases_llm.jsonl (AWS, needs a price OK). Training on it ends eval_agent_output's held-out status.
• Cost $96.38 (approved ≈ $100): training writing $37.09 + Astra $37.95; test writing $15.55 + judges $5.76 + Jev $0.03.
• Fixes: build_eval2.py takes --raw/--review/--out/--guard (its report title used a shadowed variable, fixed).

2026-09-25 19:30 PDT: weights in the repo, pjev finetune / pjev rlcd, a Qwen3.5 tree server, docs refresh (fork session; $0)¶

• Weights: weights/qwen35_4b_tree (eval2 95.6) and weights/tree_4b_combo (94.5) copied from runs/, tracked with Git LFS (weights/**/*.safetensors), each with a model.json (base model and revision, recipe, prompt, scores, sha256, serve commands); weights/README.md. Not committed.
• pjev finetune / pjev rlcd (src/personal_jev/finetune.py, cli.py): the Qwen3.5 tree recipe on any JSONL of (state, question, target), from scratch or --init weights/qwen35_4b_tree; validation reports accuracy, cross-entropy, Brier, ECE and confidently-wrong counts. RLCD = policy gradient on strictly proper scoring-rule rewards (log, Brier, spherical; accuracy optional): per question, --samples reports sampled from a Gaussian around the logits, reward minus the question's mean reward as the advantage, --beta KL to the starting model. Tests (tests/test_finetune.py, CPU): trained only on RLCD, a shared question with 70% "yes" learns p = 0.70 and a 60/30/10 choice p = 0.60 (calibration); finetune then rlcd end to end on a tiny random Qwen3.5. Not run on the real model yet (finetune.md).
• qwen35_tree.TreeServer: serves Qwen3.5 with the training tree, forward only (text once, each question once, each candidate's own tokens), LoRA merged; python -m personal_jev.qwen35_tree serve, run_qwen35.py --stage eval --tree, latency_sweep.py throughput --qwen35-tree. Exact vs standalone sequences in a CPU test (mixed requests, several packed batches); not timed on a GPU.
• Docs: home page, how it works, speed (Qwen3.5 on vLLM, L40S costs), reproduce, open questions, dead ends, spend ($517 logged; three runs added to the spend table), data mixes, challengers, findings box 22, README; new page "Fine-tune and RLCD" in the nav.

2026-09-25 18:05 PDT: Qwen3.5 on vLLM keeps its 95.6 and is fast for one question, slow for sixteen (fork session)¶

• What: src/personal_jev/vllm_qwen35.py.
• merge: the LoRA merged into a copy of the original (multimodal) checkpoint, replacing exactly the 152 LoRA-targeted tensors (a first version also replaced tensors that differed only by dtype; the count check caught it).
• VllmQwen35Scorer: one prompt per candidate with ChallengerScorer.entry's token ids, vLLM's prefix cache, readout logprob(yes) − logprob(no) over the two allowed tokens (= z_yes − z_no); serve CLI.
• Also: scripts/eval_vllm.py --qwen35 (now under a main guard: vLLM spawns its engine), latency_sweep.py throughput --qwen35 --options-in-question --gpu-price --name, vllm_tree serve --gpu-memory-utilization.
• Accuracy through vLLM 0.30 on eval2: qwen35_4b_tree 95.58, identical to transformers (4 / 4 flips, max score difference 0.40; reports/qwen35_4b_tree/eval2_vllm); tree_4b_combo 94.53 vs 94.48 (3 / 2; reports/tree_4b_combo/eval2_vllm). All of eval2 in 112 s and 51 s.
• Latency, server side p50 ms (L40S, both models served on one GPU, Jev in the same sweep from the Mac; reports/latency/requests_qwen35.jsonl, reports/latency/summary.md; the transformers column is bench.json):

End to end from California add ≈ 60–120 ms to ours (us-east-2 box through an SSH tunnel). - Why 16 questions are slow: for this hybrid model vLLM sets the attention block to 528 tokens (one DeltaNet state per page) and caches the recurrent state only at block boundaries (mamba_cache_mode "align"). Each of the 48 prompts recomputes the text after the last boundary plus its question; texts under ≈ 500 tokens share nothing (993 ms at 256 tokens). mamba_block_size 64 or 16 changes nothing (hits stay on the 528 lcm); a bf16 state cache halves the block to 272 (256 tokens 407 ms, 4K 689 ms, but 1K 744 ms). Probe: reports/qwen35_4b_tree/vllm/probe.log. - Cost per 1,000 requests, GPU fully busy (L40S $2.242/h; reports/latency/throughput_*_l40s.json): the Qwen3 tree on vLLM is below Jev in every cell (1 q × 8 tokens $0.005 vs $0.016; 16 q × 4K $0.243 vs $0.265). Qwen3.5 is at Jev's price for one question from 1K tokens up ($0.127 vs $0.194 at 4K) and 2–6× Jev with 16 questions. - Cost: ≈ $1.88 (spend table). - Verdict: Qwen3.5 serves on vLLM at full accuracy and 1.7–1.9× faster than transformers for one question; for many questions vLLM's coarse hybrid prefix cache makes it 1.3–1.8× slower than our own fork path and 2–4× slower than the Qwen3 tree. Options: route by question count; patch vLLM to checkpoint the state at the shared-prefix junction (0.30 does it only for EAGLE, enable_mamba_fine_grained_prefix_cache); or a faster custom fork server.

2026-09-25 16:05 PDT: Qwen3.5-4B trained with a real tree scores 95.6 on eval2, our best, 1.6 behind Jev (fork session)¶

• What: a shared-prefix tree for Qwen3.5 training (src/personal_jev/qwen35_tree.py, run_qwen35.py --tree).
• One packed row per state: root (system + document), question segments (the longest common prefix of the question's branches), one leaf per candidate. Same token ids as ChallengerScorer.entry, so the existing forked-cache inference serves the adapter unchanged.
• Full-attention layers read the row through tree.tree_mask. Gated DeltaNet layers run level by level (roots, then question segments from their root's final recurrent and conv state, then leaves), with gradients through those states; right padding inside a level is a no-op for the state (q = k = v = 0, beta = 0, g = 0). Own per-layer activation checkpointing (HF's drops the cache path).
• Exactness: tests/test_qwen35_tree.py (tiny random hybrid model, fp32, CPU): scores within 5e-6 and every gradient within 1e-4 of full sequences. On the real model (--stage check-tree, reports in reports/qwen35_4b_tree/checks/): fp32 scores within 0.004, LoRA-gradient cosine 0.99997; bf16 cosine 0.981, the same as the full-sequence reference against itself re-batched (0.980), so the bf16 gap is rounding. The fla kernel's initial-state gradient was checked separately against the torch reference (rel 1.8e-3 in bf16).
• Run: qwen35_4b_tree = qwen35_4b_combo's data and LoRA (option lists, round-2b + round-3, r64) with the tree, texts ≤ 8K (was 2K), batch 8192 × GA 4 as tree_4b_combo.
• 51,774 training questions (2,013 = 3.7% dropped, was 18.5%), 24.4M tree tokens (the combo run: 54.9M pair tokens), 941 steps, 4.1 h on one L40S (the combo run: 6.6 h).
• Validation (1,170 q): 78.1 at step 0, 92.7 at the end; the evals use the final checkpoint (lowest loss, 0.128).
• eval2 (reports/qwen35_4b_tree/eval2, paired):
• 95.6: binary 96.9, multiclass 96.8, multilabel EM 90.1 (Jev 97.8 / 98.1 / 94.2).
• vs qwen35_4b_combo 94.5: 52 / 31, p = 0.028: the tree (more and longer training texts) adds +1.1 on the same base.
• vs tree_4b_combo 94.5: 63 / 41, p = 0.039.
• vs Jev 97.2: 31 / 64, p = 0.0009 (the gap was 2.7). Simple tier 98.6 vs Jev 98.5.
• Errors: 88, of which 23 with margin ≥ 0.4 (tree_4b_combo: 110 and 26).
• Diagnostic only (fixed 50/50 averages, nothing fitted): with tree_4b_combo 95.9; with it + the 27B teacher 96.5.
• Dev benchmark (reports/qwen35_4b_tree/test): 84.4 (binary AUROC 0.972, ECE 0.038; multiclass macro-F1 92.6; multilabel EM 59.3). vs qwen35_4b_combo 84.3 (p = 0.84); vs tree_4b_combo 82.7 (154 / 94, p = 0.0002); vs Jev 82.7 (238 / 178, p = 0.004).
• Speed (reports/qwen35_4b_tree/bench.json): unchanged, same inference code (transformers + forked cache, L40S): 156 ms at 512 tokens, 252 ms at 2K, 950 ms at 8K for 1 question. No vLLM path for Qwen3.5 yet.
• Cost: ≈ $10.80 (spend table). Three check attempts failed first on memory in the check itself (the full-sequence reference, and an unbucketed batch), not in training.
• Verdict: new best model. The tree was the missing piece for Qwen3.5: same data and base as qwen35_4b_combo, +1.1 on eval2, 38% less training time. Next: serve it on vLLM (hybrid prefix cache) for latency.

2026-09-25 06:55 PDT: Qwen3.5-4B + option lists + round 3 ties the best on eval2 (94.5) and beats Jev on the dev benchmark (84.3) (fork session)¶

• Run: qwen35_4b_combo (scripts/run_qwen35.py qwen35_4b --stage train --tag _combo --data-dir data/ova --r3; the --data-dir and --r3 flags are new and additive).
• Recipe: qwen35_4b_r2x64 (Qwen3.5-4B rev 851bf6e, LoRA r64 / alpha 128, every candidate trained as its own full sequence, training texts ≤ 2K, batch 8192 × GA 4, 1 epoch) + every option listed in the question (data/ova/) + round-3 data with r3_* uncapped.
• 43,827 training questions, 2,474 steps, 6.6 h on one L40S. 9,960 questions (18.5%) dropped for > 2K tokens (7.8% in qwen35_4b_r2x64): round-3 texts are longer and the option lists lengthen every question.
• Validation (992 q, round 3 included): 77.5 at step 0, 90.7 at the end. The evals use the lowest-loss checkpoint, step 2,250 (loss 0.165, 90.5).
• eval2 (reports/qwen35_4b_combo/eval2, paired):
• 94.5: binary 96.2, multiclass 95.8, multilabel EM 88.2 (our best multilabel; Jev 94.2).
• vs tree_4b_combo 94.5: 61 / 60, p = 1. A tie with different strengths: multilabel +2.3, multiclass −1.7, temporal reasoning 80.3 vs 85.2, multi-turn 90.6 vs 94.0, sarcasm 94.0 vs 91.3.
• vs qwen35_4b_r2x64 93.7: 62 / 46, p = 0.15. The two levers add +0.8 here (not significant) vs +1.8 on Qwen3 (p = 0.001); the 2K training cap is the likely reason.
• vs Jev 97.2: 23 / 77, p = 6e-8.
• Diagnostic only (fixed 50/50 probability average, nothing fitted, 2× the compute): with tree_4b_combo 95.5 (either right 97.5); with tree_4b_combo + the 27B teacher 95.9.
• Dev benchmark (reports/qwen35_4b_combo/test): 84.3, our best (binary 90.0, multiclass 85.3, multilabel EM 62.2). vs tree_4b_combo 82.7: 172 / 116, p = 0.001; vs Jev 82.7: 250 / 194, p = 0.009; vs qwen35_4b_r2x64 83.4: p = 0.053.
• Speed (reports/qwen35_4b_combo/bench.json; transformers + forked cache, in-process p50 on the L40S; bench requests have 3 short options and no option list): 512 tokens 157 ms (1 question) / 343 ms (16); 2K 252 / 510 ms; 8K 943 / 1,504 ms. The same code on the A10G (qwen35_4b_r2x64) was 1.5–2.6× slower. Not comparable with the Qwen3 tree's vLLM numbers; vLLM serving of Qwen3.5 is untested.
• Cost: ≈ $15.80 (spend table).
• Verdict: ties the best on eval2 and is the first model significantly above Jev on the dev benchmark. tree_4b_combo stays the served model (vLLM path). Next for Qwen3.5: a real tree in training (fork the recurrent state with gradients) to lift the 2K cap, then vLLM serving.

2026-09-25 00:45 PDT: how Jev probably works, and the next architecture (fork 2 session; $0, nothing run)¶

• The user asked what all the findings say about how Jev was built and whether a new architecture is needed.
• Memo: reports/jev_hypothesis_2026-09-25.md. Measured facts first, then inference: shared state + isolated per-question branches (the "state plus the longest question" limit), options inline with one readout per option (113 billed tokens per 3-option question), ≤ a few B parameters on fast GPUs (400K tokens/s marginal; the 113–136 ms floor at 8 tokens is hop + batching), trained on frontier judgments (dev benchmark multilabel EM 40.1 vs our 57.6) with a calibration objective. Nothing measured requires a from-scratch backbone; our three failed new-architecture attempts say the pretrained LM's joint reading and readout are the quality.
• New $0 analysis, scripts/ensemble_eval2.py → reports/eval2/ensembles.md:
• Jev's probabilities are rounded to 0.01, 19.3% graded (ours 10.8%); 7 of its 55 eval2 errors have margin ≥ 0.4, 26 of our 110. On 44 of the 82 questions Jev wins over tree_4b_combo, our margin is below 0.2.
• Fixed 50/50 averages, nothing fitted: combo + Eikos-4B 95.3, combo + Qwen3.5-4B 95.0, combo + 27B teacher 95.8 (multilabel 89.5), all four 95.8. Cascade combo → average with the 27B when margin < 0.3: 9.1% deferred, 95.5; with Jev as the fallback (upper bound) 96.7 at 12% deferred. Deferring to the 27B alone gains nothing (94.9).
• Thresholds must come from validation, never eval2; the curve is flat between tau 0.2 and 0.45.
• Proposals added to the open ideas (awaiting the user's OK): A parallel-readout branch (one branch per question, verdict block with a readout per option + none, listwise loss; ≈ $12), B a calibration term in the same run, C the serving cascade ($0 to decide), D the H100/H200 latency sweep first (RunPod H100 SXM $3.49/h, H200 $4.59/h, low stock; AWS had none), E a prefix-LM (bidirectional state) pilot (≈ $5).
• Eikos report: 3 of 1,991 predictions have a correct flag that a 0.5-threshold recomputation does not reproduce; its own flags are used.
• 01:05 update: the fork's qwen35_4b_tree (95.6) landed meanwhile; ensembles.md regenerated with it as the primary:
• tree_4b_combo 95.9, + the 27B 96.3, + both 96.5; cascade to the 27B average at 7% deferred 96.1. The memo's gates now refer to 95.6.

2026-09-24 23:40 PDT: everything combined scores 94.5 on eval2, the best of our runs, 2.7 behind Jev (fork session)¶

• Run: tree_4b_combo (scripts/run_tree_combined.sh, R3=1).
• Recipe: tree + Qwen3-4B-Instruct-2507 + LoRA r64, every option listed in the question (data/ova/), hf + synthetic
  • hardcases_nb + hardcases_r3 with r2_/r3_ uncapped.
• Batching: MBT 8192 × GA 4, 51,826 questions, 967 steps, g5.xlarge.
• It differs from tree_4b_instruct_r3 only in the option lists.
• eval2 (paired):
• 94.5: binary 96.0, multiclass 97.5, multilabel EM 85.9.
• vs tree_4b_instruct_r3 93.3: 60 / 37, p = 0.025. The option lists stack with round 3.
• vs tree_4b_combo_r2 92.9: 63 / 31, p = 0.001. Round 3 stacks with the option lists.
• vs tree_4b_instruct_r2x64 92.7: 75 / 40, p = 0.001.
• vs Jev 97.2: 27 / 82, p = 1e-7. The gap narrows from 4.5 to 2.7 points.
• It matches the 50/50 ensemble of the 27B teacher with tree_4b_ova (94.5), with one 4B model.
• Dev benchmark: 82.7, the same as Jev (230 / 230) and the control. This benchmark cannot rank these models.
• Validation 92.5, loss 0.126.
• Latency: same architecture and option format as tree_4b_combo_r2, so its vLLM numbers apply (JOURNAL 15:05). The vLLM path was not re-scored for this run; B's and C's matched transformers within 0.1.
• Cost: g5.xlarge 11:59–23:38 ≈ $11.70 (terminated; SG and key pair deleted).
• Serving: merge with vllm_tree merge --model-id Qwen/Qwen3-4B-Instruct-2507 ..., then serve with --options-in-question.

2026-09-24 18:25 PDT: option pointers lose about a point and are only ~1.1x faster, so a dead end on vLLM (fork session)¶

• Idea (the hybrid "read deeply once, score the options cheaply"): number the options once in the question and let each leaf say only "option k" (personal_jev/options.py pointers=True, data/ptr/, serve --option-pointers), instead of repeating the option's description at its leaf.
• Run: tree_4b_combo_ptr, the recipe of tree_4b_combo_r2 (B) with only the leaf text changed. 18.7K questions, 232 steps, g5.xlarge.
• Accuracy vs B (paired):
• eval2: 92.0 vs 92.9 (41 / 58, p = 0.11); multiclass 94.3 vs 95.8, multilabel 80.9 vs 84.3, binary 94.9 vs 94.4.
• Dev benchmark: 82.7 vs 83.5 (88 / 116, p = 0.06).
• vLLM path: 91.9 / 82.7.
• The loss is where the pointer applies: the readout position no longer holds the option's text.
• Latency, scripts/bench_options.py: vLLM, same A10G, in-process, cold prefix, p50, reports/tree_4b_combo_ptr/bench_options_*.json.
• Pointers are 1.00–1.38x faster: 1 × 3 options at 512 tokens 136 → 135 ms; 16 × 8 at 512 1,146 → 833 ms; 16 × 64 at 2,048 11.8 → 10.7 s.
• Why so little: vLLM still receives one full prompt per option (text + question + option list + leaf). Prefix caching saves the compute, but 16 × 64 options means 1,024 prompts and 2–4M submitted tokens to hash and schedule per request, whatever the leaf length.
• A real many-options speed-up needs one sequence per question with a readout at every option (e.g. vLLM per-token pooling), which means a new readout and training.
• Cost: g5.xlarge 15:17–18:23 ≈ $3.10 (terminated; SG and key pair deleted). The Jev sweep was skipped: it would not change the verdict.

2026-09-24 16:18 PDT: Qwen3.5-4B scores 93.7 on eval2, our best; Eikos-4B zero-shot 92.8 (session "SelfJev state of play")¶

• Run qwen35_4b_r2x64 (scripts/run_qwen35.py, pipeline scripts/run_qwen35_gpu.sh, launcher scripts/aws_qwen35.sh):
• Qwen3.5-4B (rev 851bf6e8), LoRA r64 / alpha 128 on q/k/v/o + in_proj_qkv/z/b/a + out_proj (57.5M trainable).
• Data: the tree_4b_instruct_r2x64 mix (hf + synthetic + hardcases_nb, r2_* uncapped), training texts ≤ 2,048 tokens (17,435 questions kept, 1,484 dropped). One epoch, 769 steps, grad accumulation 4, lr 2e-4, 3.8 h on one A10G.
• Training scores each candidate as its own full sequence (14.3M pair tokens): the Qwen3.5 cache writes its recurrent state in place, so the tree's shared-root backward pass is not available. Inference shares the text by forking the native cache; check vs full sequences: max |Δscore| 0.10 (bf16).
• Validation (≤ 2K-token items, not comparable with the tree runs' mix): 74.3 → 81.4 (150) → 86.0 (300) → 88.5 (769, best).
• eval2 (reports/eval2/summary.md; reports reports/qwen35_4b_r2x64/eval2, reports/eikos_4b/eval2, reports/qwen35_r1/eval2):

• Paired (only-row / only-other): vs r2x64 73 / 53, p = 0.09; vs r3 65 / 57, p = 0.53; vs Eikos 75 / 56, p = 0.12; vs Jev 21 / 91, p = 1e-11 (Jev still +3.5). Eikos vs Jev 20 / 109.
• Long texts, although it never trained on > 2K tokens: > 2,048 tokens 97.6 (r2x64 95.7, Jev 98.9); > 4,096 97.0 (r2x64 94.6, Jev 98.8). Sarcasm 92.6 (r2x64 86.6), multi-positive 88.2 (86.3); numeric 80.4 and temporal 82.8 unchanged.
• Old test 83.4: the first run above Jev's 82.7 there. Binary ECE on eval2 0.024 (Jev 0.061).
• Either Qwen3.5-4B or Eikos right: 96.5% of eval2 (their errors differ).
• Speed (reports/qwen35_4b_r2x64/bench.json, same A10G, transformers, forked cache, branch batches of 16; p50 ms) vs the Qwen3 tree R1 transformers path (reports/latency_optimization_2026-09-24/results.md): 512 × 1 q 239 vs 160; 2K × 16 q 1,300 vs 967; 8K × 1 q 1,988 vs 1,777; 8K × 16 q 3,332 vs 2,506. Slower, 1.1–1.5×: at ≤ 8K tokens linear attention does not pay for the per-batch cache copies. vLLM ≥ 0.30 with its hybrid prefix cache (as Eikos ships) is untested here.
• Verdict: best eval2 and old-test scores, but not a significant gain over the Qwen3 Instruct tree (p = 0.09 on the same data, 0.53 vs round 3), with 3× less training data than round 3 and texts ≤ 2K in training. Worth: round-3 data on this base, a vLLM serving path, and (cheap) a Qwen3.5-4B + Eikos ensemble check. Cost ≈ $4.75.

2026-09-24 15:35 PDT: round 3 on the Instruct base: 93.3 on eval2, only +0.6 over round 2 (not significant)¶

• Owner: Jev classifier with Qwen reranker.
• Run: tree_4b_instruct_r3.
• Recipe: configs/tree_4b_instruct_r3.json = Qwen3-4B-Instruct-2507, tree, LoRA r64, max_length 8192.
• Data: hf + synthetic + hardcases_nb + hardcases_r3 (sha d20ecb98…), r2_/r3_ uncapped. 51,853 questions after 1,934 overlength drops; 910 steps.
• Hardware: g6e.2xlarge (L40S) us-east-2, MBT 16384 × GA 2 (same 32K-token effective batch as the control), ≈ 14 s/step, 3.9 h. Cost ≈ $9.25, terminated.
• Best checkpoint: step 900 (val loss 0.1447, val acc 91.7%, still improving at the end); reload exact.
• Reports: reports/tree_4b_instruct_r3/{validation,test,eval2}; adapter runs/tree_4b_instruct_r3/adapter.
• Scores:

• Paired tests on eval2 (only-row / only-other):

• Old test vs control: 92 / 87 (p = 0.77). vs Jev: 216 / 212, a tie.
• Verdict:
• This is the highest eval2 score so far, but 35K more verified hard cases buy only a non-significant +0.6 on top of round 2's ~7K. More of this kind of data has hit diminishing returns.
• Jev still leads by 3.9 points; multilabel EM (84.3 vs 94.2) is the biggest gap.
• Next levers: the options-in-question format (fork's runs A/B, in flight), teacher ensembles / distillation, and multilabel-specific work.
• Updated docs/findings.md (1, 1b), docs/leaderboard.md, the ledger and the eval2 summary.

2026-09-24 15:05 PDT: combined recipe B, the best accuracy so far on both test sets; latency on vLLM (fork session)¶

• Run: tree_4b_combo_r2 (scripts/run_tree_combined.sh, R3=0).
• Recipe: tree + Qwen3-4B-Instruct-2507 + LoRA r64, every option listed in the question (data/ova/), hf + synthetic
  • hardcases_nb with the r2_* families uncapped.
• Batching: MBT 8192 × GA 4, 18,676 questions, 240 steps.
• Otherwise identical to tree_4b_instruct_r2x64.
• Accuracy vs that control (paired):
• eval2: 92.9 vs 92.7 (45 / 42, p = 0.83).
• Old test: 83.5 vs 82.7 (121 / 94, p = 0.08); Jev scores 82.7 (p = 0.22).
• By type, old test: multilabel 59.0 vs 52.9, multiclass 84.4 vs 83.7, binary 90.0 vs 90.9.
• Validation 88.1 vs 85.8.
• So on the Instruct base the option lists help mainly the old test's multilabel questions and tie on eval2.
• Serving path: merged LoRA (bf16) + vLLM, via scripts/eval_vllm.py → reports/tree_4b_combo_r2/{eval2,test}_vllm.
• Scores: eval2 93.02, old test 83.52; merging and vLLM cost no accuracy.
• Throughput: all 1,991 eval2 questions in 144 s, 3,471 old-test questions in 104 s.
• pjev serve / vllm_tree serve have a new --options-in-question flag (src/personal_jev/options.py, shared with scripts/options_in_question.py; test in tests/test_server.py).
• Latency, end to end from the Mac (p50, 10 rounds, reports/latency/requests_combo.jsonl; box RTT 67 ms, OpenRouter 7 ms):

• Against the round-1 reranker on vLLM: the option lists cost about +120 ms at 16 questions (each question carries its list) and nothing at 1 question. The A10G is still the limit.
• Cost: g5.xlarge 11:58–15:02 ≈ $3.10 (terminated; SG and key pair deleted) and Jev $0.02.

2026-09-24 12:40 PDT: docs site glossary and hover definitions (cloud session; $0)¶

• New docs/glossary.md (test sets, data rounds, trap tags, models, run-name decoder, metrics, services) in the top nav.
• Hover tooltips for shorthand on every page: docs/.includes/abbreviations.md, auto-appended by pymdownx.snippets. Add a line there when a new term appears on the site.

2026-09-24 11:50 PDT: docs site and cleanup (cloud session, branch claude/modest-shannon-epq8g4; $0, nothing run)¶

• What: every finding of the project, formatted as a docs site built with Zensical from docs/ (zensical.toml, GitHub Pages workflow .github/workflows/docs.yml). New pages: index, findings (24 findings with evidence), leaderboard (embeds the generated eval2 summary and the ledger), speed, dead_ends, stock_model, challengers, data, landscape, next, costs, how_it_works, reproduce. Numbers copied from the generated tables and report files; the 27B teacher's eval2 score re-computed from reports/teacher/qwen38_27b_eval2.jsonl.
• Cleanup:
• README rewritten as a short landing page; its detailed 0.6B/4B/8B sections moved to docs/stock_model.md.
• experiments.md restructured: current conclusions first, done ideas moved out of the open-ideas table, and the ledger sorted by eval2 (scripts/ledger.py now sorts that way and links Jev/Astra rows to their real reports).
• Absolute /Users/julien/... links in reports/*.md made repo-relative; tmp/pdfs/ page renders removed; the two review PDFs moved from output/pdf/ next to their sources.
• Merge note: other sessions edit docs/experiments.md and this file on master; merge this branch with care.

2026-09-24 11:50 PDT: combined runs started; the 10:17 stop was not the fork session; LFS pointers fixed (fork session)¶

• The unlogged 10:17 PDT stop-instances was not this session. It never calls stop-instances; its last AWS action that day before 11:58 was terminate-instances on its own box (selfjev-tree-ova-kd) at 06:19 PDT.
• After commit ae5a581 the working tree held Git LFS pointer files for data/hardcases*.jsonl, data/ova/hardcases*.jsonl and data/hardcases_r3.jsonl (134-byte files). git lfs checkout restored them from the local LFS store; the hashes match the LFS oids and git status is clean. Check file sizes before syncing data to a box.
• The user asked for all winning approaches combined and trained, then accuracy and latency measured. The owner of the round-3 rerun agreed; runs A and B are in In flight.

2026-09-24 10:45 PDT: partial round-3 checkpoint scores 90.8 on eval2, inconclusive (the run was stopped at step ≈ 500 / 915)¶

• Owner: Jev classifier with Qwen reranker. Scored runs/tree_4b_instruct_r3_step300, the best-validation-loss checkpoint saved before the 10:17 stop (see below). Reports: reports/tree_4b_instruct_r3_step300/{eval2,test}.
• Scores:

• Not a verdict on round 3.
• Step 300 of 915 had seen about a third of the 52K-question mix, at peak learning rate with no decay.
• The control finished its schedule.
• A full round-3 run (~10 h on an A10G, ~$10) is needed to judge the data.
• Cost ≈ $6.25. Box terminated; SG, key pair and pem deleted. Nothing of this session runs in AWS.

2026-09-24 10:42 PDT: Qwen3.5 / Qwen3.8 as the next base, and Eikos-4B (session "SelfJev state of play"; $0, nothing run)¶

• What exists. Qwen3.8 has no small dense model: only 27B (qwen3_5_text, 48 linear-attention + 16 full layers), a 180B MoE "Flash-Next" and a 2.4T MoE. Qwen3.5 has 0.8/2/4/9/27B plus Base variants; 4B = 32 layers, 24 Gated DeltaNet + 8 full attention, hidden 2560, 262K positions.
• We already have a Qwen3.5 result, in the Codex challenger worktree (~/.codex/worktrees/1d33/SelfJev, reports/challengers/qwen35, trainer scripts/run_challenger.py, scorer src/personal_jev/challengers.py): Qwen3.5-2B + LoRA r16 on all its projections (DeltaNet in_proj_*/out_proj included), shared document with the native cache (recurrent state + KV) forked per branch, round-1 data only. Old test 79.9% (tree 4B 81.6, stock 4B 80.3, 0.6B 73.5); 853 steps in 17 min on an L40S. Never scored on eval2.
• Speed, same L40S, p50 ms (bench.json): Qwen3.5-2B vs tree 4B: 512 tok × 1 q 125 vs 114; 2K × 16 q 319 vs 339; 8K × 1 q 394 vs 805; 8K × 16 q 765 vs 1,050. Linear attention pays off only on long texts; at ≤2K the MLPs dominate.
• Eikos-4B (caiovicentino1/Eikos-4B, MIT, created 2026-09-23): a full fine-tune of Qwen3.5-4B, distilled from GLM-5.3-Flash on finance/rules data; options as letters, softmax over the letter logits; vLLM ≥ 0.30 with --enable-prefix-caching --mamba-cache-mode all shares the state across questions (the card reports vLLM 0.11 lost 3–6 points on batched long shared documents). Own-harness claims: JevBench public hard 72.1 (Jev 73.0 on the official leaderboard), long context to 32K trained, ECE 0.033. Not comparable with our numbers.
• Proposal (awaiting the user's OK on the GPU spend): open idea "Qwen3.5-4B tree-equivalent" in experiments.md.

2026-09-24 10:36 PDT: survey of open "Jev-like" models on Hugging Face (session "SelfJev state of play")¶

• Read the model cards of the trending text-classification models; nothing downloaded or run; $0.
• Two camps. Small encoders, 0.15–0.4B: Laya (ModernBERT / mmBERT), open-jev-deberta-v3-large, rlcd-modernbert-151m (GLiClass), plus MLX / CoreML ports. LoRA on decoders with answer-token readout, 0.6–27B: kev 0.5/0.8/4/9B and decider 0.8/2/4/35B-A3B (Qwen3.5 bases), openjev (Qwen3.5 0.8/2/4B), Bespoke-Nimble-9B (Qwen3.5-9B), AutoJev-27B (Qwen3.8-27B), Lumma-fev-0.6b (own base).
• Their claims, each on its own benchmark and not comparable with ours: AutoJev-27B 84.6 vs Jev 82.8; kev-4b beats Jev in distribution but trails out of domain (0.817 vs 0.857); kev-0.8b trails Jev everywhere; decider-35b-a3b JevBench hard 0.676, decider-4b 0.541. Small models advertise speed (33–46 ms vs a quoted 236–276 ms for Jev; we measured 143–178 ms).
• Same pattern as ours: sub-1B trails (our 0.6B −17.7 on eval2, jina 0.6B 73.3), 4B and up is where quality is. Most decoder entries use our recipe (LoRA, logit readout, no generation). Many use Qwen3.5 bases, whose linear-attention layers our tree cannot branch.
• "JevBench" public items exist and several cards report on them: a shared yardstick. Open idea added.

2026-09-24 10:17 PDT: round-3 training box stopped by an unlogged aws ec2 stop-instances from this Mac¶

• Owner of the box: Jev classifier with Qwen reranker (i-01e7e056786127d8e, "personal-jev-instruct-r3", tagged, listed in In flight).
• What happened, per CloudTrail:
• StopInstances at 17:17:57 UTC by julien@connectly.ai, from this Mac's IP.
• User agent aws-cli … md/command#ec2.stop-instances: a CLI call, not the console.
• No session logged it. Training was at step ≈ 500 of 915.
• Fork 2 (curve / capacity session): not this session. Its scratchpad and task logs contain no stop-instances; its only lifecycle calls were terminate-instances on its own four boxes, the last at 03:10 PDT, and it was idle from 03:17.
• Rule for every session: never stop, terminate or modify another session's box. Check JOURNAL In flight first; if a box looks orphaned, ask its owner session or the user.
• Recovery (user's choice): score the best checkpoint saved before the stop.
• That is step 300: val loss 0.176, val acc 89.2%. Step 500 was 0.178 / 89.2, so it wasn't saved.
• The box restarted as a g5.2xlarge, since its AZ had no g5.xlarge capacity. Results in the next entry.
• Validation trajectory: 78.5 (init) → 85.3 (100) → 87.8 (200) → 89.2 (300) → 89.5 (400) → 89.2 (500).
• This validation set includes round-3 questions, so it does not compare with the control's 85.8.

2026-09-24 10:32 PDT: review of Laya (NandhaKishorM/laya), an open "Jev-style" engine (session "SelfJev state of play")¶

• What: read the repo (README, BENCHMARKS pointers, laya/common.py, laya/agent.py) at commit 970dc8c. Nothing was run or downloaded beyond the git clone; $0.
• Architecture: a bidirectional encoder (ModernBERT-large 421M for English, mmBERT-base 322M multilingual), one sequence per question [CLS] type question [SEP] [MASK] opt0 [MASK] opt1 … [SEP] state [SEP], all options in one listwise row. A new 2-layer transformer head + MLP scores each [MASK]; softmax over options. noul is a 2-option choice (false/true), score a softmax over levels. Trained end to end with a GRPO-style policy gradient on proper-scoring-rule rewards (log + spherical + RPS), which they call RLCD. A script/language router picks one of three checkpoints.
• Limits it documents: the state is truncated to max_len − head_max_len (≈320 tokens on laya, ≈768 on multilingual; up to 8K opt-in), options share a 192–256-token budget (Banking77 0.425 vs Jev 0.870), base checkpoints are near chance zero-shot on its own typed-decisions set (0.36 vs 0.318 random; 0.766 only after fine-tuning on that set's train split), negation and noul label-following failures, over-confident before temperature fitting (ECE 0.466 → 0.081).
• Its Jev numbers are third-party published, on different samples and prompts: not comparable with ours. Its Jev latency (236–276 ms p50) is slower than what we measured (143–178 ms, reports/latency/summary.md).
• Relation to our work: the listwise idea (a candidate sees its alternatives) is what tree_4b_ova added (+1.0 on eval2). The new-head-on-encoder idea resembles our custom model, but Laya keeps state and options in one sequence, so they interact in every layer. Closest measured analogue: our jina v3.5 0.6B listwise run, 73.3 on eval2 (docs/jina_model.md).
• Verdict: Laya is the faster and multilingual option (≈33 ms on a T4, CPU/ONNX), not the more accurate one on long, trap-heavy texts; ours reads up to 32K tokens untruncated. Expected, not measured: well below our 92.7 on eval2. Scoring it on eval2 is free on the Mac and is logged as an open idea.

2026-09-24 06:20 PDT: distilling the 27B into the tree does not help (fork session)¶

• Run: tree_4b_ova_kd, the tree_4b_ova recipe with teacher_weight 0.5 and data/teacher/qwen38_27b_ova_train.json.
• The teacher file holds Qwen3.8-27B-FP8 zero-shot probabilities on the 8,372 target-task training rows and gold on the 8,000 hf rows.
• Hardware: g5.xlarge, 241 steps, ≈ $2.90, box terminated.
• Scores vs tree_4b_ova (the same run without the teacher):
• Old test: 82.5 vs 82.6 (45 / 50, p = 0.68).
• eval2: 91.0 vs 91.6 (33 / 45, p = 0.21). By type: multiclass +0.7, binary −0.9, multilabel −1.9.
• Validation: 86.1 vs 86.2.
• Verdict: the 50/50 ensemble's +2.9 on eval2 (94.5%) does not transfer through soft targets on the training rows.
• Where the teacher disagrees with the verified label (12–20% of rows), the 0.5 weight pulls toward a wrong answer.
• Untested variants: teacher on multiclass only (where it is strongest), a lower weight, the r64 student.
• Logged as a dead end for this setting in experiments.md.

2026-09-24 04:22 PDT: round-3 data built (38.6K verified), all-options wins, the 27B teacher is complementary (fork session)¶

• 04:22 (09-24): round-3 training data built: data/hardcases_r3.jsonl, 38,628 verified questions (fork; sha256 d20ecb98…; review data/hardcases_r3/review/REVIEW.md).
• Writers, via gen_hardcases.py --round3: GPT-6 Luna 23,336, Gemini 3.8 Flash 9,772, Grok 4.7 5,520 (60/25/14%). Never eval2's writers or its NOVEL_* domain lists.
• Blind Astra judge (OpenAI Batch): 97.0% agree with the authors (simple 98.8, hard 96.5, very hard 95.6). Only agreements are kept. 0 states dropped by the 8-gram guard vs eval.jsonl + eval2.jsonl.
• Splits: train 34,868 / validation 3,760. Types: binary 16,770 / multiclass 12,402 / multilabel 9,456.
• "None" correct in 7.0% of the multiclass questions offering a none-like option (round 2: 28%); 54% of multilabel questions have 3+ positives (round 2: 21%).
• Cost ≈ $280 (writing $114.5, judge ≈ $165.6 at list batch prices, $4.16 per 1K questions) vs ≈ $250 approved.
• Training on it: owned by "Jev classifier with Qwen reranker" (Instruct base, r64). Remember cap_exempt_families for r2_/r3_.
• 03:25 (09-24): the 27B teacher is as good as our best model on eval2, and its errors differ from ours (fork; reports/teacher/). eval2 is used here for reporting only: nothing was tuned on it.
• Qwen3.8-27B-FP8 zero-shot on eval2: 91.4% (binary 93.4, multiclass 97.8, multilabel 75.9); tree_4b_ova 91.6%.
• Paired: 112 questions only ours gets right, 108 only the teacher (p = 0.84). Either one right: 97.0%.
• A fixed 50/50 average of the two models' probabilities (nothing fitted) scores 94.5% (binary 95.7, multiclass 98.5, multilabel 85.1); Jev scores 97.2%. That is the case for distillation.
• Training split: the teacher scored 22,893 of 22,919 train rows (26 overlong), agreeing 80.1% with the labels.
  • Teacher file: scripts/teacher_scores.py teacher-file --gold-families hf_. The hf rows keep gold targets because the teacher is weaker on them on validation (69.7%).
• Cost: g6e.xlarge us-east-2 07:06–10:26 UTC ≈ $6.20. Box terminated; SG and key pair deleted.
• 02:45 (09-24): all options listed in the question wins on both test sets (fork; tree_4b_ova).
• What: scripts/options_in_question.py lists every candidate in the question text, in a fixed random order per question ("one vs all in context"). Each leaf still judges one candidate, but now sees all the alternatives.
  • The r2b recipe is otherwise unchanged (scripts/run_tree_ova.sh, data/ova/, same 1,600-per-family cap).
  • 154 overlong questions were dropped at max_length 8192.
• Old test (3,471 q): 82.6% vs r2b 81.2% (130 vs 82, p = 0.001), vs tree_4b 81.6% (p = 0.02), vs Jev 82.7% (p = 0.93, tied).
  • By type vs r2b: multilabel 57.8 vs 51.7, multiclass 83.6 vs 82.8, binary 89.2 vs 88.1.
• eval2 (1,991 q): 91.6% vs r2b 90.6% (66 vs 46, p = 0.07); Jev 97.2%. Every type is 0.8–1.6 points up.
• Validation: 86.2 vs 86.1, loss 0.234 vs 0.247.
• Caveat: one run each. Binary rows, which the transform leaves unchanged, also moved ~1 point, so part of the gain may be run-to-run variance.
• Inference must apply the same transform to requests.
• Cost: g5.xlarge 23:53–02:46 ≈ $2.90. Box terminated; SG and key pair deleted.
• The round-3 owner decides with the user whether round 3 uses it.
• 01:30 (09-24): Qwen3.8-27B-FP8 zero-shot as a soft-label teacher, validation check (fork; scripts/teacher_scores.py, AWS us-east-2 L40S; scores in reports/teacher/qwen38_27b_validation.jsonl).
• Method: vLLM, thinking off; the answer probability is read from the next-token distribution restricted to the answer tokens.
• Validation, 2,121 questions (hf, synthetic, eval and hardcases validation rows): 82.4% overall.
  • by file: hf 69.7%, synthetic 98.2%, eval (authored) 93.2%, hardcases (Astra-verified) 88.1%;
  • by type: binary 89.4, multiclass 88.4, multilabel 61.1.
• Same 868 hf + eval validation questions: our r2b 79.8% vs the teacher 72.9% (115 vs 55, p = 5e-6).
  • On the 118 authored ones the teacher leads: 93.2% vs 83.1%.
  • It loses on public-set label conventions and wins on target-task questions.
• Its probabilities are graded (23% of values within 0.05–0.95, vs 0.8% for Astra's verbalized ones), so it is usable as a soft-label teacher for the target-task rows, not the public-set rows.
• It agrees with Astra-verified labels on 88.1% of hard cases, so it cannot replace Astra as the checker.
• Its training-split scores are running (for a later, separate distillation ablation, not the round-3 run). A reference eval2 score (test: reporting only) follows.

2026-09-24 03:35 PDT: Instruct base + round-2 data + r64 scores 92.7 on eval2, the best of our runs¶

• Owner: Jev classifier with Qwen reranker.
• Run: tree_4b_instruct_r2x64, the control for round 3.
• Recipe: configs/tree_4b_instruct_r3.json with R3=0: Qwen3-4B-Instruct-2507, tree, LoRA r64 on q/k/v/o, max_length 8192.
• Data: hf + synthetic + hardcases_nb, with the r2_* families uncapped. 18,681 questions, 218 steps.
• Hardware: AWS g5.xlarge, MBT 8192 × GA 4, about 2.3 h. Cost ≈ $2.45, box terminated.
• Best checkpoint: the last one (val loss 0.226, 85.8%); reload exact.
• Scores:

• Old test: 82.7, identical to Jev (213 / 214, p = 1).
• Paired tests on eval2 (only-row / only-other):

• Verdict: the Instruct base stacks with the data. Most of the gain over r2b is the base, since r2b + r64 alone gives 90.3.
• Jev still leads by 4.5 points, with multilabel EM 94.2 vs 83.5 the biggest gap.
• Round 3 uses the same recipe plus data/hardcases_r3.jsonl.

2026-09-24 03:17 PDT — combined levers: round-2b data + LoRA capacity¶

• Owner: fork 2. Two g5.xlarge (no g6e capacity), max_batch_tokens 8192 × grad_accum 4 (same 32K effective batch), 208 steps each; ≈ $5.26; both boxes terminated, SG and key pair deleted. Configs configs/curve/tree_4b_r2b_r64*.json, runner scripts/run_curve.sh (tree_* branch now also scores eval2); tables reports/curve/summary.md, reports/eval2/summary.md, ledger.
• tree_4b_r2b_r64 (r=64, 47.2M params): eval2 90.3% vs 90.6 for r2b (44 / 50, p = 0.61); old test 81.2 (= r2b); validation 86.6. The +2.1 that r=64 gave on round-1 data is gone on round-2b data.
• tree_4b_r2b_r64_mlp (r=64 + gate/up/down, 132.1M params): eval2 91.3% (59 / 44 vs r2b, p = 0.17); old test 82.4 (129 / 88 vs r2b, p = 0.0065; best old-test score of any of our models); validation 87.0 (best). vs Jev 26 / 144, p = 5e-21; Jev 97.2.
• Slices vs r2b: binary 93.7 vs 92.9, multilabel EM 80.4 vs 78.8, multiclass 94.3 vs 94.1; very-hard tier 89.6 vs 86.5; by author Opus +1.1, GLM +1.3, Kimi −0.2; every trap tag with n ≥ 150 improves (+2 to +4: paraphrase, injection, numeric, long_state, temporal). Small everywhere, significant nowhere: consistent with a real but sub-point gain, not with style overfitting.
• Verdict: capacity and verified data overlap; the best recipes today are r2b data + r=64 + MLP (91.3 eval2) and the fork's tree_4b_ova (91.6, options listed in the question), within noise of each other and 6 points below Jev. The remaining gap is not a capacity or volume problem; it needs data or format changes aimed at the very-hard tier and multilabel (80 vs 94). Instruct base + round-2 data + r=64 is training in the "Jev classifier with Qwen reranker" session.

2026-09-24 08:17 UTC: T5 adaptation scope correction; initial pilot was decoder-only¶

• Owner: Codex tree latency/compact. The initial target filter missed native model.encoder.text_model.layers; the first adapter has 208 decoder tensors, zero encoder tensors, 2,981,888 trainable parameters. Earlier entries describing full text adaptation are superseded by this audit.
• Initial 73.84% development / 76.80% eval2 and 266.61 ms primary latency remain valid decoder-only measurements, archived in reports/t5_round2b_2026-09-24/decoder_only_audit/. They cannot decide the intended full-adaptation experiment. Historical R1 reproduction remains valid: all 3,471 old decisions match, 75.37%; eval2 72.98%.
• Fixed exact module coverage and added an actual native full T5 + PEFT unit test, plus independent encoder/decoder first-step gradient audits. Rerun uses configs/t5gemma2_r2b_full.json, the unchanged frozen 16,375/1,388 questions, seed and recipe. Correction was discovered after evaluation; no evaluation labels enter training or checkpoint selection.
• H100 attempts failed in all Ohio zones; no instance launched, no H100 compute charge. Corrected smoke/training use our existing L40S i-087b024b35cff657b, $3.00424/hour, with the original termination cap. Other sessions' resources untouched.

2026-09-24 01:00 PDT: eval2 re-scoring. The Instruct base and rank 64 help on the real task; 0.6B is far behind¶

• Owner: Jev classifier with Qwen reranker.
• Scored on AWS g5.xlarge i-08aed2c38168e5da6, ≈ $1.12, terminated; SG and key pair deleted.
• The Mac run was stopped at the user's order: no evals or training on the laptop, now in AGENTS.md.
• Table: reports/eval2/summary.md; lever table in experiments.md (eval2 section).
• eval2 accuracy, all trained on round-1 data unless noted:

• Paired tests vs tree_4b (only-row / only-ref):

• The old test ranked these the opposite way, or called them ties.
• Round 2 vs r2b: equal overall. r2 is better on multiclass (95.8 vs 94.1) and r2b on multilabel (78.8 vs 75.4). The "none" cap only mattered for CLINC.
• Verdict: three independent levers measured one at a time: data +5.5, Instruct base +3.0, rank 64 +2.1.
• fork 2's combined run (r2b data + r64) uses the reranker base. An Instruct-base + r2b-data (+ r64) run is the missing combination.
• The 0.6B size is out for quality: −17.7 vs stock 4B.

2026-09-24 07:58 UTC — T5 matched-R2b pilot: quality shortfall¶

• Owner: Codex tree latency/compact. Training completed in 3,685 seconds, 228 steps; best validation loss selected the final checkpoint (77.59% validation accuracy, loss 0.379396). Reload matches scores exactly on 32 validation questions. Adapter SHA256 4c4d1f9d683052f0f2b32196f409a9b46a250255ae84caf49b3288131bfa9c43, saved locally at runs/t5gemma2_r2b/adapter/.
• Fixed unmerged T5 with bounded 16-branch decoder batches: 73.84% on the 3,471-question development benchmark, 76.80% on eval2 (1,991 questions); primary same-L40S local latency 266.61 ms p50 / 269.38 ms p95, 100 fresh-prefix 2K × 16 × 3 requests. Predictions, timings, protocol.
• Verdict so far: large quality shortfall versus the existing round-2b eval2 report (90.56%); this pilot is not a replacement. Same-GPU tree controls, old-T5 rescore, merged/prefix-cache treatments and conditional hardware follow-up are still running. No test-driven retraining or checkpoint change. Compute spend and resource cleanup remain in progress. Ledger refreshed from report files.

2026-09-24 07:19 UTC — prior T5 experiment found; matched round-2b follow-up underway¶

• Owner: Codex tree latency/compact, original master. The user correctly recalled the earlier T5Gemma 2 experiment in the challenger worktree. It already used the pretrained full encoder/decoder and shared cross-KV: 75.37% old development accuracy versus R1 merged tree 81.50%, and 203 versus 297 ms on its L40S 2K × 16 × 3 benchmark. The preceding architecture memo now carries a correction. Historical reports/hashes are preserved under the new experiment folder.
• Matched-data pilot: replayed original R2b sampling/length eligibility exactly, 16,375 train / 1,388 validation questions; all original file hashes match. No train/validation overlap in IDs, source IDs or exact states. One epoch, attention LoRA rank 16, LR 2e-4, 228 larger optimizer updates; checkpoint selection by validation loss. This does not isolate data from training recipe compared with the old T5 run's 823 updates.
• Prototype: bounded decoder batches, view-based document cross-cache, optional native decoder common-prefix sharing. 26 local checks pass; real pretrained FP32 independent/shared inference error <= 0.000014. Current validation at step 100: 73.20% question accuracy, loss 0.445829; initial validation 37.25%, loss 1.006185. No new development/eval2 result yet; 2× speed / <=1 pp quality-loss gate is pending. Source snapshots, checks, exact requests and progress are saved in reports/t5_round2b_2026-09-24/ and runs/t5gemma2_r2b/.
• Cloud: user explicitly authorized GPU provisioning in the conversation. Dedicated Ohio L40S i-087b024b35cff657b, verified $3.00424/hour, eight-hour terminate-on-shutdown cap at 14:38:54 UTC. Virginia capacity attempts launched no instances; their temporary SG and key are deleted. Final cost and Ohio cleanup pending.

2026-09-24 00:11 PDT — correction: LoRA capacity does help, on eval2¶

• Owner: fork 2. My 22:00 verdict "capacity is not the ceiling" was measured on the old test only and is wrong for the target task. Recomputed from the prediction files: on eval2, tree_4b_r64 scores 87.2% (77 / 36 vs tree_4b, p = 0.00014) and tree_4b_mlp 86.3% (62 / 38, p = 0.021) vs 85.1%, with the same adapters that tie on the old test (81.5 / 81.6 vs 81.6). The trainer's validation split predicted the ranking; the old test did not.
• Updated: docs/experiments.md (conclusion 2, dead-end row struck, idea status), reports/curve/summary.md (eval2 columns in the tree-capacity section, generated), the ledger table (new eval2 column, scripts/ledger.py), README.
• Proposed next run (needs the user's OK): round-2b data + r=64 (and + MLP), one A10G each, ≈ $3 each; the two gains have only been measured separately. Cost of this correction: $0.

2026-09-24 06:27 UTC — next architecture recommendation¶

• Owner: Codex tree latency/compact. Reviewed current eval2, controlled latency, previous cross-attention implementation, and primary model/paper sources. Recommendation: pretrained encoder–decoder with shared full-token memory and pretrained independent readers; T5Gemma 2 1B–1B is the proposed pilot, with the other session's smaller-tree work as control.
• Evidence and bounded protocol: architecture memo. Current eval2: round-2b 90.6% versus Jev 97.2% (reports/eval2/summary.md). Current compact experiment's speed gain is 7.4% (reports/latency_optimization_2026-09-24/acceptance.json). No proposed architecture has been trained or benchmarked here.
• Findings: our tree already shares the document and avoids generation; the prospective gain must come from cheaper pretrained token processing. The old two-block custom reader does not test a deeply pretrained encoder–decoder. More questions still require more arithmetic. Jev's exact internals and training origin remain undisclosed in the official announcement.
• Cost: $0 new cloud/API spend. Verdict: research proposal only; no new branches, commits, GPU instances or model downloads. Eval2 must stay out of tuning, with a fresh final confirmation set after these test-informed choices.

2026-09-24 06:17 UTC — latency/compact work integrated into original master¶

• Owner: Codex tree latency/compact. At the user's request, moved this session's completed experiment into the original master checkout, preserving the newer source and concurrent edits. No new commits. Removed only this session's extra branch/worktree; the other experiment's worktree remains.
• Code and results: conclusions, integration record. Compact weights are at runs/tree_4b_compact_v2/adapter. Archived the former worktree and its Git history in runs/latency_optimization_2026-09-24/integration_archive and verified every one of its 597 regular files. Verified all 24 migrated model/metadata files.
• Verification in the original checkout: 16 focused tree/compact tests passed, 2 real-model tests excluded (previous GPU evidence is in the experiment report). All 190 preexisting files outside the explicitly updated documentation remained byte-identical.
• Completed experiment findings are now available to the other sessions: direct branch-mask allocation preserves scores; vLLM's development quality was checked; compact training reduces branch tokens but fails the predeclared 15% cold-prefix vLLM speed-improvement gate and increases CLINC over-rejection. See measured tables and raw predictions in reports/latency_optimization_2026-09-24/. These runs use R1 and have not been evaluated on eval2.
• This session owned i-0172e64b9d0dc20ac (selfjev-tree-compact-20260924); it is terminated and its security group/key pair are deleted. Root EBS deletion remains asynchronous (cleanup.json). The two selfjev-challengers instances belong to another session. Faster GPU launches failed due to capacity/quota; no faster-GPU measurements exist from this experiment.
• Cost of this integration: $0 new cloud/API spend. Prior experiment billing has not been reconciled. Verdict: all further work stays in the original branch; compact remains experimental.

2026-09-24¶

• 01:30–02:00: jina-reranker-v3.5 (0.6B, listwise) trained the tree-r2b way (Synthetic data generation strategy; write-up docs/jina_model.md, box selfjev-jina-20260924 g5.xlarge $1.006/h, ≈ 1.5 h ≈ $1.60).
• What: new backend jina.py (state = query, every question/candidate = a passage in one causal context, logit = scale·cos(proj(query), proj(passage)) + bias; LoRA r=16 on q/k/v/o + the projector + 2 head scalars), trainer train_jina.py, pjev --jina / train-jina, config configs/jina_r2b.json = the r2b recipe (hf + synthetic + hardcases_nb, max_length 8192, 1 epoch). 16,301 train questions / 11,774 contexts, 346 steps, 72 min on the A10G.
• Numbers: validation 77.4 (tree 4B r2b 86.1); old test 76.6 (calibrated 75.9; 0.6B Qwen LoRA 73.5, tree 4B r2b 81.2); eval2 73.3 (tree 4B r2b 90.6, Jev 97.2): binary 79.8 / multiclass 83.5 / multilabel EM 40.1; simple 80.6 / hard 70.7 / very hard 68.5. Zero-shot 58.6 validation, 46.5 eval2. Reload check 0.0.
• Verdict: behaves like the 0.6B Qwen reranker, not like the 4B: ≈ 17 points behind the tree on eval2, multilabel and long/distractor/injection cases worst. Its late-epoch gains equal the 4B's, so it is a slower learner from a lower start, not a steeper one. Speed on the same A10G, eval2: jina 58 ms per question vs tree 4B r2b 118 ms (reports/tree_4b_r2b_gpu/eval2_timing, 90.4 there): only 2× faster for 17 points less. License CC BY-NC 4.0: experiments only.
• Follow-ups prepared, not run: configs/jina_r2b_e2.json (2 epochs), configs/jina_r2b_r64.json (r=64 + MLP), scripts/run_jina_followups.sh.

• 00:40: judge_hardcases.py made crash-safe for large runs (Synthetic data generation strategy): submit() saves review/batches.json after every chunk and stops cleanly on an OpenAI error (e.g. enqueued-token limit), so a rerun submits only the remaining sources instead of paying twice. Pitfall list for the round-3 run sent to the fork session (judge ≈ $3.40 per 1K questions ≈ $135 for 40K; Grok 4× Gemini per state; one judge process per review dir; one generator per prefix; check the none-correct rate after the build).

• 23:05: first eval2 scores: the round-2 data is worth +5.5 points on the real task, and Jev leads by 6.6 (session "Jev classifier with Qwen reranker"; table reports/eval2/summary.md via scripts/eval2_summary.py).

• Scores: Jev 97.2%; tree_4b_r2b 90.6%; tree_4b 85.1%.
• Paired tests: r2b vs r1 142 / 34 (p = 7e-17). r2b vs Jev 24 / 157 (p = 4e-25).
• Multilabel exact match: 63.9 → 78.8, Jev 94.2.
• Very-hard tier: 79.8 → 86.5, Jev 96.5.
• Traps, r1 → r2b (Jev):
  • improved: multi-positive 67 → 80 (96), numeric 67 → 79 (92), temporal 70 → 78 (89), injection 72 → 82 (97), sarcasm 72 → 83 (97), distractor 80 → 88 (97);
  • unchanged: paraphrase 83.5 → 83.9 (95).
• Text length is not a weakness: 1K–4K tokens scores 93.9, >4K 91.6.
• By author: Opus-written questions are hardest for us (87.7), Kimi's easiest (93.4).
• Takeaways:
  • The old test's near-tie was a public-label-noise ceiling.
  • Verified target-task data is the lever the old test could not see. The round-1 "saturation" and dead ends need re-checking on eval2.
• How it was scored:
  • scripts/run_eval2.sh on this Mac (MPS bf16, --max-length 16384), about 22 min per tree model.
  • Jev via compare_external.py --data data/eval2.jsonl --ours tree_4b/eval2 --only jev --tag eval2, $0.05 (a re-run on the corrected file came from cache for $0).
  • tree_4b and r2b were scored on the first export and their ids remapped (meta.remapped_to), after checking every target, type and candidate list.
• ≈ 21:20–23:40: eval2, the frozen target-task test set, built (session "Synthetic data generation strategy"; data/eval2.jsonl, sha256 2fa954592d0a3124…, review data/eval2/REVIEW.md, human spot-check data/eval2/review/SPOTCHECK.md).
• What: 1,991 questions / 647 states, every row split=test. Authors never used for training data: Claude Opus 5.5 (560 q, $13.05), Kimi K3 (649 q, $5.55), GLM 5.3 (782 q, $2.81). Tiers 673 / 664 / 654 (hard / simple / very hard); lengths 177–222 questions per bucket of the 8 → 8K ladder; types 1,016 binary / 593 multiclass / 382 multilabel; per-trap n 57–474 (evidence_start 17); multilabel with 3+ positives 245 of 382; none offered in 391 multiclass questions and correct in 14.8% of them. A third of the calls used domains, genres and instruction styles absent from the training brief.
• Judges: GPT-6 Astra (OpenAI batch, $7.95) and Gemini 3.1 Pro (sync, $3.32), blind; kept only where both equal the author: 1,991 of 2,070 (96.2%; Astra alone 97.5%, Gemini alone 97.0%). 0 states dropped by the 8-gram guard.
• Jev on eval2 (never used for keep/drop): 97.2% (Jev $0.05; the other session's compare_external run agrees, 1,981 of 1,991 decisions identical, the rest is Jev's own run-to-run noise). Weakest: numeric, temporal, multilabel. On the old test Jev scores 82.7, so most of that gap was public-set label noise.
• Correction 00:10: the first export (sha f18549eb…) used source-format lines, which renumber question ids after a drop, so my review answers no longer joined and I reported Jev 95.2%. The file now holds expanded rows with the raw ids (sha 2fa95459…, same 1,991 questions and targets); data/eval2/review/id_map_sourceformat_to_real.json remaps reports scored on the first export.
• Cost: $32.72 total, all API. Scoring of tree r1 / r2 / r2b, stock 4B and Jev on eval2 is the other session's job.
• Verdict: usable now; still LLM-verified only, the 50-question SPOTCHECK is for the user.

2026-09-23¶

• 23:35: what TypeSafe discloses about Jev (fork; public web, no spend).
• Architecture: per MarkTechPost, TypeSafe says Jev is "transformer-based, but it is not a large language model", with a "new model architecture, parallel sampler": all outputs come from one pass, with no token-by-token generation.
• Questions run "in parallel and in isolation against the same state" (docs).
• Limits: "64k tokens per request; 32k tokens for state plus the longest question" (docs/models); rate limit 250K tokens/s. This is the shape of our tree: each question sees the state plus itself.
• Training: "Reinforcement Learning for Calibrated Decisions (RLCD)", with no details. Their benchmark's reference answer is the average of GPT-6 Astra and Fable 5.1.
• Not disclosed: size, base model, data, context details. They "cannot prove the price is unsubsidized".
• Sources: typesafe.ai/blog/introducing-system-one-models-and-jev, docs.typesafe.ai/models.md, marktechpost.com/2026/09/19/typesafe-ai-releases-jev/.
• Reading, with our own evidence:
  • the gap is mostly training, not architecture;
  • model 4B → 8B, adapter r64/MLP and stock → tree each moved ≤ 1.3 points, while 10K verified target-task questions moved eval2 +5.5;
  • candidates next: soft Astra targets (already paid for), more verified data, a listwise choice branch.
• 23:15: how Jev stays flat (fork; free analyses of data already paid for).
• Fit on the 400 Jev requests of the latency sweep: time = 132–137 ms fixed + 2.2–2.6 ms per 1,000 input tokens. That is ~400K tokens/s of marginal speed for one request; our tree 4B with vLLM on the A10G does 165 ms per 1,000 tokens (~6K tokens/s).
• Jev reads the whole text:
  • on the verified round-2 hard cases (data/hardcases.jsonl × data/hardcases/review/jev_answers.jsonl), it is right on 97.2% of the 107 questions tagged evidence_end whose text is over 4K tokens (up to 17.6K);
  • start and middle score 96.8% and 98.8%;
  • its accuracy by text length is 94.7% up to 256 tokens and 97–98% from 2K to 8K+.
  • So it doesn't truncate.
• Reading: every architecture must touch each token, so the flat curve means a very small cost per token, not "no attention". Consistent with a ~0.5–1B-parameter model (or an MoE with ~1B active parameters) on H100/B200-class GPUs, or a larger model split over several GPUs.
• Jev's billed token count for our text is 1.036× Qwen's, so it probably doesn't use the Qwen tokenizer.
• 21:17: tree 4B round 2b result: 81.2% (runs/tree_4b_r2b, reports/tree_4b_r2b; box terminated, ≈ $6.70 for all of round 2).
• Overall: vs round 1 81.6 p = 0.39, vs round 2 80.6 p = 0.08, vs Jev 82.7 p = 0.02. Calibrated: 80.9.
• CLINC is back to 92.0% (round 1: 94.7, round 2: 83.3). In-scope requests sent to none: 24 (16, 50).
• Authored slice: 83.0% (round 1: 78.4, round 2: 81.9, Jev: 94.7).
• Traps vs round 1: temporal 52 → 69, numeric 44 → 69, multi-positive 44 → 51, long state 76 → 84, exception 29 → 57 (small n).
• TREC dropped 88.7 → 84.7: numeric/description questions sent to entities. Cause unknown; single seed.
• Outside CLINC, rounds 1, 2 and 2b score 80.35 / 80.35 / 80.23. The round-2 data changes only what this test barely measures.
• Verdict: keep tree_4b as the aggregate reference. tree_4b_r2b is the candidate for the target task, and eval2 decides. The CLINC fix came from a test diagnosis.
• Scripts: scripts/rebalance_nota.py, scripts/run_tree_r2.sh.
• Ready for eval2: scripts/run_eval2.sh (our 4B models, --max-length 16384), and scripts/compare_external.py --data data/eval2.jsonl --ours <run>/eval2 --only jev for Jev on the same questions.
• 21:10: latency and cost vs Jev, text 8 → 4,096 tokens (fork; scripts/latency_sweep.py, src/personal_jev/vllm_tree.py; tables in reports/latency/summary.md).
• Setup:
  • the same decisions-API requests from the Mac (California), one at a time, endpoints in random order;
  • Jev through OpenRouter (edge 12 ms away) vs our tree 4B + LoRA (runs/tree_4b) on one AWS A10G in us-east-1 (71 ms away);
  • 1 or 16 choice questions × 3 options, 10 timed rounds, fresh random text each time.
• Jev: flat 143–178 ms p50 at every size, including 4,096 tokens × 16 questions. OpenRouter's server timing (OpenRouter + Jev) is 109–154 ms.

• Ours, p50 from the Mac (8 → 4,096 tokens):

  server	1 question	16 questions
  transformers	196 → 1,062 ms	688 → 1,700 ms
  transformers, merged LoRA (--merge)	153 → 938 ms	602 → 1,517 ms
  vLLM	120 → 755 ms	336 → 1,195 ms

• vLLM beats Jev end to end up to 128 tokens with 1 question (120–133 vs 143–165 ms), despite the 71 ms network handicap. Jev is 5× faster at 4,096 tokens and 2–7× faster with 16 questions.

• The A10G is the limit: it processes 6.5–10K tokens/s with vLLM (4.7–6K with transformers). Jev's flat curve means its compute for about 6K tokens takes tens of ms: much faster GPUs, a smaller model, or both.
• Cost with the GPU fully busy ($1.006/h):
  • vLLM $0.005–0.27 per 1,000 requests vs Jev $0.016–0.27: 1.3–3× cheaper, and equal at 4,096 × 16;
  • Jev charges $0.042 per million input tokens and bills about 372 tokens of overhead per request plus about 113 per extra 3-option question;
  • a g5.xlarge left on all month (≈ $734) beats Jev only above about 7 requests/s sustained, for 512-token requests.
• vLLM path:
  • merged LoRA (bf16), with vLLM's prefix cache sharing the text between leaves;
  • readout = vLLM's Qwen3-reranker 1-logit head;
  • on the example request it makes the same decisions as the transformers path (max score difference 0.06 logit);
  • engine start takes about 6 min the first time (compilation) and about 1 min cached.
• Cost: AWS ≈ $0.87, Jev $0.04. Box terminated; SG and key pair deleted.
• Not measured: the vLLM path's accuracy on the test set, and any GPU faster than the A10G. The L40S was out of capacity in every zone.
• 22:00: tree LoRA capacity is not the ceiling (fork 2; two g5.xlarge, ≈ $1.60, both terminated, SG and key pair deleted). Same data and recipe as runs/tree_4b (10,080 questions, 84 steps), configs configs/curve/tree_4b_r64.json, tree_4b_mlp.json, runner scripts/run_curve.sh; tables in reports/curve/summary.md.
• r=64 on q/k/v/o (47.2M trainable, 4×): 81.5% test (binary 88.5, AUROC 0.956, multiclass 83.0, multilabel EM 52.3); 82 run-only vs 84 reference-only, p = 0.94. Validation 81.8% vs 80.2%.
• r=16 on q/k/v/o + gate/up/down (33.0M, 2.8×): 81.6% (binary 87.9, AUROC 0.955, multiclass 83.4, EM 51.7); 72 vs 73, p = 1.0. Validation 81.4%.
• Both vs Jev: p = 0.07–0.08, unchanged. Family swings (±5–12 points on eval_agent_output, TREC, eval_policy) go both ways and are within noise for 26–300-question slices.
• Verdict: with this data, more adapter capacity fits validation better and moves the test by nothing. Together with the data curves and 4B = 8B, the recipe (data + objective), not the model or the adapter, sets the 80–82% level. Full fine-tuning is now low priority. Remaining levers: rejection/threshold policy (in progress in the other sessions) and new verified data for the binary families.
• 20:40: SST-2's binary gap is a bias, not a reading problem (reports/tree_4b/test, a diagnosis on test only; nothing was tuned).
• SST-2 is held out: no SST-2 data was trained on.
• The tree ranks SST-2 reviews well (AUROC 0.973, Jev 0.994), but predicts "positive" on 34.7% of reviews when 50% are positive.
• Accuracy is 84.0%; the best possible single threshold would give 91.7% (Jev 96.7%).
• BoolQ is different: AUROC 0.927 vs Jev 0.965, best threshold only 84.3% vs 83.3% at 0.5. That's a real ranking gap.
• Across all families, SST-2 and BoolQ account for about 60 of the 63 binary answers the tree is behind Jev.
• Next step: a label-free fix, see experiments.md "Calibration policy". One option is contextual calibration: subtract each question's score on an empty text. Pick it on validation, then confirm on a fresh holdout. It is test-motivated now, so an SST-2 gain alone is not proof.
• 20:24: learning curves done: the stock 4B recipe saturates at 80–81% (fork 2; 10 runs, 8 g5.xlarge 16:05–19:20, ≈ $23, all boxes terminated, SG and key pair deleted). Tables and paired tests: reports/curve/summary.md; README subsection "Data and base-model curves (4B)"; configs configs/curve/, runner scripts/run_curve.sh.
• Nested data subsets 25/50/100% of the 10,112-question mix: 77.0 / 78.9 / 80.3 (1 epoch), 79.1 / 79.1 / 79.8 (2 epochs). About 1.5 points per doubling; the second epoch never helps (best checkpoint inside epoch 1).
• New task type: +100 / +300 / +1,000 / +3,000 BoolQ training questions → BoolQ test 84.3 / 83.3 / 84.3 / 86.7 (from 83.3; Jev 90.7); overall 80.0 / 80.9 / 80.9 / 80.9, none significant (p ≥ 0.09).
• Base model: Qwen3-4B-Instruct-2507 zero-shot 71.3 (binary 84.2) vs 62.8 for the reranker; after LoRA 80.6 vs 80.3 (p = 0.68). Prompt picked on validation (reports/curve/instruct_prompt_selection).
• Verdict: not a data-volume or model-size problem. Next lever claimed: tree LoRA capacity (r=64, MLP targets; configs/curve/tree_4b_r64.json, tree_4b_mlp.json, scripts/run_curve.sh tree_4b_r64 tree_4b_mlp), waiting for the user's OK on the spend.
• 20:19: tree 4B round 2b started.
• Why: round 2 lost CLINC (94.7 → 83.3%). In the hard-case training data, "none" is the correct answer in 23% of the multiclass questions that offer it, vs 8% in round 1.
• Fix: scripts/rebalance_nota.py drops 160 random none-answer train questions to reach 10%, giving data/hardcases_nb.jsonl with 9,982 questions. Validation is unchanged.
• Run: TAG=tree_4b_r2b HARD_TRAIN=data/hardcases_nb.jsonl scripts/run_tree_r2.sh.
• Motivated by a test diagnosis: report CLINC with that caveat.
• Validation during training: step 100 84.9% (round 2: 83.9%), step 150 85.4% (round 2: 85.6%).
• 20:15: stock 4B round 2 (scripts/run_4b_r2.sh) stopped before it finished to free the GPU for r2b. No report exists; the stock-vs-tree round-2 control is still open.
• ≈ 19:20–20:10: tree 4B round 2 (runs/tree_4b_r2, recipe from the tree session, + data/hardcases.jsonl, max_length 8192): 80.6% vs 81.6% for round 1 (p = 0.01).
• Traps improved: numeric +25, paraphrase +14, role reversal +13, injection +10, distractor +9 points.
• CLINC in-scope requests routed to "none" rose from 16 to 50 of 255.
• On the other 3,171 questions, rounds 1 and 2 score the same (80.35%).
• Full analysis: docs/hardcases_round2.md and reports/tree_review_2026-09-23/review.md.
• ≈ 16:30–19:15: round-2 hard cases, OpenRouter part + blind judge (session "Synthetic data generation strategy"; write-up docs/hardcases_round2.md).
• Why: more trap-heavy training data without Claude Code usage limits; the user asked for ⅓ simple / hard / very hard, a clean state-length ladder (8 → 8K tokens) and heavy prompt variance so the model learns the task, not one phrasing.
• How: scripts/gen_hardcases.py (system prompt data/hardcases/BRIEF.md, random per-call ASSIGNMENT). Gemini 3.8 Flash $12.87 → 950 states / 2,462 q; Grok 4.7 $8.19 → 319 / 1,022 (reasoning cannot be disabled, 4× Gemini per state); GPT-6 Luna $1.62 → 1,676 / 3,988; DeepSeek V4 Flash $0.17 → 458 / 1,264 (stopped early, slow provider). Total 3,403 states / 8,736 questions, $22.86.
• Judge: scripts/judge_hardcases.py, GPT-6 Astra reasoning low, blind, OpenAI Batch API (OpenRouter's batch endpoint rejects this key). 5 batches, 0 failures, $36.24 for 10,627 questions incl. the 1,891 Sonnet ones; Jev second opinion $0.27. Author = judge 95.4% (Grok 98.9, Sonnet 98.3, Gemini 97.6, Luna 96.0, DeepSeek 82.1); tables in data/hardcases/review/JUDGE.md. 432 questions are author = Astra but Jev wrong.
• Verdict: pipeline works and resumes; data quality good except DeepSeek; the retrain diagnosis above (CLINC none) is the data's one known flaw. Nothing in flight from this session; no AWS boxes launched by it.
• ≈ 14:20: custom shared-state model (v1 spec) finished and written up (fork 2 + the "Synthetic data" fork; docs/custom_model.md, reports/custom_diagnostics/, tests in tests/test_custom.py, 0.6B backbone, A10G).
• As specified (frozen backbone + cross-attention + heads): 39.0%. Joint LoRA alone: 39.7% (p = 0.32). With the similarity term + joint LoRA (runs/custom_sim_lora): 58.2%; distillation from the stock 0.6B teacher: no gain.
• Why: label memorization, binary head at chance (text and question never meet inside the backbone), and the frozen features already match unseen labels better than the trained heads (MaxSim 75–77% vs 23–28%).
• Speed: 38–43× faster than stock pairs for 16 × 3 questions on 8K–16K-token texts. Its lesson (share the text, keep deep joint reading) led to the tree scorer.
• Engineering notes: MPS caches one graph per tensor shape, hence shape buckets in custom.py; an explicit attention mask was 1.7–2.2× slower for identical outputs and was removed.
• Hard-case data, Sonnet part:
• 8 Claude Sonnet sub-agents wrote 700 sources / 1,891 questions (data/hardcases/raw/h*, brief data/hardcases/BRIEF_sonnet_agents.md).
• The OpenRouter part and the Astra judge were run by the "Synthetic data generation strategy" session.
• scripts/build_hardcases.py keeps only questions where the blind Astra judge agrees with the author: 10,142 kept.
• Stock 4B and 8B on AWS (scripts/run_model.sh, one g6e.xlarge, 3.16 h):
• 4B: 62.8 → 80.3% with LoRA. 8B: 66.2 → 80.7%. 4B + LoRA ≈ 8B + LoRA (p = 0.52).
• Tables: reports/scale_comparison.md.
• Jev and GPT-6 Astra on the full test (scripts/compare_external.py): Jev 82.7%, Astra 85.8%.
• The questions we fail most: reports/external/failures.md. That's a test-set list: use it to design data, never to train on.
• Decisions-API-shaped endpoint added to server.py (/api/alpha/decisions).
• LinkedIn draft and chart: docs/linkedin_post.md, docs/linkedin_accuracy.png (scripts/make_post_chart.py).

2026-09-22¶

• Project built:
• Qwen3-Reranker yes/no scorer, validation, typed decisions, grouped-loss LoRA training, held-out calibration, evaluation harness, benchmark.
• Data: data/eval.jsonl (398 authored questions, blind-reviewed by LLM), data/hf.jsonl (16,800 questions from 11 pinned datasets, 6 of them held-out families), data/synthetic.jsonl (2,448 questions).
• 0.6B on the Mac: 61.0% unmodified → 73.5% with LoRA (reports/summary.md).