// HACKER NEWS — CYBERSECURITY
Jeff – Jev-compatible 0.8B decision models, trained at home, ~30 ms
Fine-tunes of Qwen3.5 and Gemma 4 for zero-shot classification: small, fast decision models you slot into your
code, with the same request format as Jev. You describe a situation and list the options in plain words; Jeff returns a
calibrated probability for each option from a single forward pass. No generated text, no parsing: about 22 ms per
decision on an RTX PRO 6000 and 28 ms on an Apple M4 Max (MLX).
Zero-shot means the options can be anything: support queues, user intents, moderation labels, voice commands, game
moves. Your categories don't need to appear in the training data; you describe them, and Jeff picks.
What it is, and what it isn't. These are very small models. They make extremely fast, well-calibrated judgement
calls between options, and they slot easily into your local code. On benchmarks they approach, and sometimes beat, Jev;
but at this size their reasoning won't match Jev's, which runs on a much larger model. If zero-shot accuracy isn't
good enough for your purposes, a short fine-tune on your own examples takes you much further: our
voice-navigation fine-tune moved held-out accuracy from 31.7% to 95.8% in under half an hour on one GPU.
Built entirely on local hardware. Training on one RTX PRO 6000 workstation GPU (the 0.8B trains in about 2 hours,
the 2B in about 3.5), all synthetic training data written by an open model (Qwen3.8-Flash-Next) on two DGX Sparks,
testing on a MacBook. No cloud GPUs, and no closed-model output in the training data; a closed model was used only to
spot-check the quality of a sample of the synthetic data.
Independent project. Jeff uses the same request format as Jev, but it is not affiliated with or endorsed by TypeSafe, the
makers of Jev. Our training code starts from the open-source AutoJev recipe.
Models on Hugging Face: Jeff-Qwen3.5-0.8B · Jeff-Qwen3.5-2B · Jeff-Gemma4-E2B
Each answer has a probability per option, the chosen option and a confidence. Three question types: choice (pick one
of up to 255 options), noul (yes/no, returned as a probability) and score (a point on a scale you describe).
Several independent questions in one request are answered together.
4,599 questions from five public benchmarks, plus JevBench's public hard tier (105 items, scored separately):
Bold: the winner of Jeff against Jev in each row. Bold italic: AutoJev-27B where it is the best of all models
in the row (on RAGTruth, tied with Jeff-Qwen3.5-2B); it is shown for reference, since the head-to-head comparison is
with Jev. The published Jev and AutoJev figures were measured on a different sample of the same benchmarks. Jeff's
overall score comes from classification and grounding, where it matches or beats the large models; on the
reasoning-heavy benchmarks (BBH, JudgeBench, JevBench) it stays well below them, as you would expect at this size.
To test zero-shot performance on tasks unlike anything in the benchmarks, we had Jeff play three games. Games aren't
the ideal zero-shot test, since a game's state isn't typical unstructured data; but they are a common, and fun, way to
test a System 1 model. Each turn, the code describes the situation and the legal moves in words, and the model picks
one. The options state what each move leads to (Frogger: "you would be hit by a car and lose a life"; Doom: "the
nearest monster is a little to your left"), but never which move is right. Each result is 20 episodes, seed 1234; ▶
opens a video of the run's first episode.