One $3k Box Matched Frontier AI on a Public Decision Benchmark

Jev-style "System-One" models answer typed questions (yes/no, pick one, rate on a scale) and return probabilities instead of prose. Open alternatives appeared within weeks of Jev's launch. We ran them, and a large local model as a baseline, on one HP Z2 Mini G1a (Ryzen AI MAX+ PRO 395, 128 GB) to answer a practical question: can a small team replace a hosted decision API with a private one, and where does that break?

JevBench publishes 231 of its items (easy, standard and hard tiers); the rest stay held out. We ran the public items through the unchanged harness and recomputed the leaderboard rows from JevBench's own per-task file on the same 231 items.

The hard tier is where the gap opens: 0.766 without reasoning, 0.964 with a small reasoning budget (DeepSeek and GPT-5.6 also score 0.964 there). Model served by Halogen 0.13.4; strict JSON-schema probabilities through the harness's openai_compat adapter, temperature 0.

On a typed-decisions test set covering four business workflows (agent-trace review, customer service, invoices, security incidents: 400 cases, 2,000 decisions), a 421M model tuned on those workflows beats the hosted general model:

This is a task-tuned model against a general one, not "open beats closed". The same checkpoint scores 0.541 on JevBench's general mix. Tuning buys accuracy on your task, not generality: tune the fast model per workflow, and keep a big model for everything else.

• "~95% phishing accuracy by decomposing the question." On 2,000 PhishNChips emails, every small model asked "is this phishing?" collapses to about 0.50. Five narrow checks plus a logistic combiner lifted Laya to 0.701 (AUROC 0.765): better than Jev's published 0.626, well short of Claude Haiku 4.5's 0.813, and far from 95%.
• Routing by the small model's confidence. Sending Laya's unsure items to the big model reached 0.874 at 80% escalation, below the big model alone (0.887). A general small model's confidence is a poor router.
• Calibration. Without reasoning, the local model states 100% confidence on 80% of items, including most of its errors.

We now serve this as jev-local: TypeSafe's /v1/systemone wire format, answered by the local model in one schema-constrained call per request (1–2 s warm), with bulk yes/no filtering handed to the small encoder. Any tool built for Jev (browser agents, judges, routers) can point its base URL at it and run privately with no per-call cost.

• ROCm PyTorch runs these encoders on the iGPU fast (0.25 s per five-question request vs ~1.4 s on 16 CPU threads), but the HIP runtime kept one CPU thread spinning while idle: about 23 W extra, around the clock. For always-on services we run the encoder on CPU.
• Laya 0.3.11's batch API was about 25× slower than single calls on gfx1151; stay on single calls.
• A 125B-A6B model reserving ~87 GB leaves ~15 GB for PyTorch: run 4B-class deciders with the big model unloaded.
• Reproduce a model's own published numbers first; it caught two of our harness bugs before they reached a table.

Public benchmark data only. JevBench (MIT): public items only; held-out and judge items are scored by the maintainer. Numbers measured on our hardware unless marked leaderboard or published.