Jeeves needs an H100.
OpenJevX needs a laptop.

We ran OpenJevX (598 MB, CPU only) on the open JevBench public tiers, with calibration, next to the Laya base model it came from. Then we put the published numbers for the big GPU models beside ours. We lose on accuracy, and on the hard tier we are no better than chance. This page says so first.

Sub-1 GB, CPU: OpenJevX vs Laya base

The open JevBench harness (MIT) by fstandhartinger, commit bb05a335, its own adapter, the 231 public items PostHog's Jeeves report uses: easy 48 + original 72 + hard 111. Three full runs per system, strictly one item at a time, results identical across runs (latency varies). Both models are 8-bit ONNX on CPU.

Chance rates: easy 28.4%, original 31.1%, hard 33.6%. Lower ECE and Brier are better. Laya base is the unmodified base model that OpenJevX was fine-tuned from, 8-bit quantized by the same script; we did not check it against the upstream weights, and Laya is described by its makers as a base to specialise, not a zero-shot engine.

Accuracy by tier, with published GPU numbers on the same items

* Published by PostHog in the Jeeves model card on the same 231 items (overall 0.866 and 0.935, hard 0.730 and 0.865); we did not run them. Not published for easy or original.

Is the confidence number real?

Partly. A confidence of 0.9 or more was right every time, but that band is 22 answers, all easy ones. Below it the number is a rough ranking, and on hard items it carries almost no signal.

• Easy items: underconfident (mean confidence 0.87, accuracy 100%).
• Original items: informative: the 0.8 to 0.9 band is right about 88% of the time.
• Hard items: bands at 0.6 to 0.8 and below 0.6 are right 38% and 30% of the time; the 12 answers at 0.8 to 0.9 are right 25%. Treat confidence there as noise.
• The fine-tune helped calibration more than accuracy: ECE 0.207 for the base, 0.121 for OpenJevX.

This is the measurable form of a fair critique: one independent test (Kanta Hayashi, "Jev Does Not Play Dice") found a hosted decision model giving 83% probability to an outcome that came up 19% of the time on a hidden fair die. Our advice is the same as theirs: treat the number as a score, and measure it on your own labelled data before you trust it. That is what our pilot does.

Is it judging, or is it a prior?

On the easy and original tiers the model beats every control we threw at it, and none of the 231 items appear in OpenJevX's training data (checked, see below). On the hard tier it does not beat the controls: its headroom there is about zero, so its answers are mostly a prior.

A confident answer can be meaningless. An independent study of a hosted decision model found questions that scored well only because of a plain feature in the data (digit count, a short word list) or because the model returned the same answer whatever the input. We ran the same controls on OpenJevX over the same 231 public items (2 Oct 2026). Headroom is |AUC of the real model - 0.5| minus the best control's |AUC - 0.5|; zero or below means the task is not demonstrated, whatever the absolute score. Intervals are bootstrap 95% over items. Aggregate only; no item text is published.

AUC mixes yes/no, pick-one and rating items (pick-one is pooled one-vs-rest). Easy and original headroom comes from the pick-one items.

• Per item, on the 139 pick-one items: the model's accuracy is 0.59 against 0.23 by chance (+0.36 over chance), and its MRR is 0.75 against 0.49 (+0.26). By tier, over chance: easy +0.79, original +0.54, hard +0.03.
• Same answer when the options are listed in reverse order: easy 35/36 (95%: 0.86 to 0.99), original 33/36 (0.78 to 0.97), hard 50/67 (0.63 to 0.84).
• Why two headroom columns: the first estimate took the best of 16 controls on the same items, which biases headroom down. The corrected column picks the best control on one half of the items and scores it on the other half. That moves hard from negative to about zero, and leaves easy and original about where they were.

The model and every control, side by side (|AUC - 0.5|)

The plain-feature baselines in full (no model, held-out half, mean of 20 splits)

The word lists are fit on the training half only and scored on the held-out half. With 48 easy items they see two or three examples per class, so on easy this control was too starved to count: it could not have reproduced the easy result, and a leave-one-out version is being run. On hard, a single number parsed from the item (their sum) beats the model.

Where we lose

• Hard answers are mostly a prior. 53% of hard answers are unchanged when the item's state is deleted entirely, and accuracy with the state deleted (0.35) equals accuracy with it (0.32). Hard confidence (0.583) is the same as confidence with the state deleted (0.569 and 0.535).
• Hard answers depend on option order. Reordering the options alone changes the top answer on a quarter of hard pick-one items. Agreement falls as the tiers get harder: 0.972 on easy (36 items), 0.917 on original (36), 0.746 on hard (67).
• Yes/no items show about zero headroom, and that is a finding about the benchmark. Replace the rubric text with ids and the model still scores AUC 1.00 on easy yes/no and about 0.72 on original: the question stem and the state carry the answer. Easy yes/no items are answerable without their options, so they measure nothing about option judgement; a yes/no score here is not evidence of reading the rubric.
• On original items the filler arm is more confident (0.741) than the real arm (0.721). Confidence tracks how constrained a question is, not whether it is right.

Where the easy 100% and the 22-of-22 band come from

Not from a constant-state prior, length or digit count. With the state deleted or replaced, none of the 22 items in the 0.9-and-above band gives the same answer at similar confidence, and that band's accuracy with the state removed is 0.27 and 0.23. Plain length, digit and number features on easy sit at about chance (best 0.27 accuracy). Memorisation of the items is ruled out as far as we can check: none of the 231 items is in the training files we hold (see the note above, with its limits). One thing we have not ruled out: a word-list explanation. The word-list control on easy has about 2 items per answer, too few to count; leave-one-out and 5-fold versions do not beat chance on any tier, which says nothing either way. Only outside labelled data could settle it.

Sanity checks

• No collapse onto the first option. The highest rate of picking the first listed option was 0.47 (bare A/B ids, original); with opt1/opt2 ids the model rarely picks it (0.03 on easy and hard). In contrast, an independent study of a hosted model saw bare ids collapse onto the first option. Caveat: OpenJevX renders each option as "key: description", so some of this may be a rendering choice and not a model property; we are testing that.
• No degenerate arm reached 0.9 confidence. The independent study saw 0.898 to 0.983 even on refusals; OpenJevX does not do that.
• The shuffle control can fail. A perfect oracle falls from 1.000 to about 0.51 to 0.53 under the per-item shuffle, and a garbage arm sits at chance, so "control at chance" here means something.

Limits: one run per arm; pick-one AUC pooled one-vs-rest; token count is a regex, not the model's tokenizer; the baseline floors are biased upward by half-split noise and by taking the maximum over several baselines, so headroom is conservative; the word-list and numeric baselines see few examples on small tiers; the standard and judge tiers and the official composite were not run. You can run the first controls on your own question in the live check after a run: "Is your question actually judging?".

The GPU reference and other small models

Different benchmarks, different hardware: do not read across rows. Each row says what its number is on. Sources and dates are in the last column; all read on 1 Oct 2026.

Not an exhaustive list. Several browser-only models published this week (for example nodd, 18 to 24 MB) quote their own latency but no JevBench accuracy that we could verify, so we leave them out rather than guess.

Per millisecond and per dollar

Limits and how to check us

• We ran only the public tiers that have open files: easy, original, hard (231 items). The standard and judge tiers, and the official composite JevBench score (which needs held-out items and speed and cost axes), were not run.
• Training overlap was checked against every OpenJevX training file we hold (0 of 231 items matched), but the exact v0.4.0 training sample is no longer available and the base model's pre-training data is not public.
• OpenJevX was served by the v0.5.0 server binary running the v0.4.0 model. The machine was not idle. Latency is a floor on noise, not a clean benchmark.
• No GPU numbers of our own; GPU rows are published figures.
• Reproduce: run the open JevBench harness against any /v1/systemone endpoint with python -m jevbench.cli run --tasks datasets/public/<tier>.jsonl --adapter typesafe --endpoint http://127.0.0.1:<port>. Our aggregate results are in results.json (no item text). Questions about our method: [email protected].