Build your own Jev from your agent logs

1. What you get, and what you don't

A Jev-style model is a cross-encoder: it reads a state (here: your request plus what the agent said and did) and a hypothesis (a question with one answer filled in), and returns how strongly the state supports it. Asked the right way, that is a judge. Trained on your reactions, it is your judge.

2. The Jev contract in one minute

Every question becomes one entailment check per answer, with the state as the premise. The hypothesis is the instructions plus one criterion: trim(instructions) + " " + trim(criterion). Three answer types come out of the same forward pass:

Confidence for n options is clip((n·p_max − 1)/(n − 1), 0, 1): 0 at a uniform guess, 1 when one option takes everything. The HTTP shape is the System One one (POST /v1/systemone with a state and a map of typed questions), so any client written for it works against your local server by changing a base URL.

3. The pipeline

4. Turn logs into decisions

Claude Code keeps every session as JSONL under ~/.claude/projects/; Codex keeps its own under ~/.codex/sessions/. One decision event is one agent turn:

• request: the last genuine human message (skip tool results, system reminders, slash-command wrappers);
• proposal: the agent's last text plus every tool call as a short line (Bash: go test ./..., Edit src/app.py: <excerpt>);
• reaction: what happened next: your next message, an interrupt, a denied tool call.

The model's input (the state) is only request + proposal, rendered exactly the same way at training and at serving time, trimmed to a 440-token budget (actions first, then agent text, then request):

Request: fix the typo in the README heading
Agent: Fixed and verified. README.md line 1 now reads "Installation". grep returns nothing.
Actions:
- Edit README.md: # Installation
- Bash: grep -n Instalation README.md

5. Hindsight labels

The trick is to label with hindsight, not taste guessing. A labeller model reads the turn and your actual reaction, and reports what you decided. The model then learns to predict that decision from the turn alone.

• An ensemble of small models labels everything (we used six: one subscription model plus cheap OpenRouter models), in batches of 20, each with a dollar cap.
• A bigger model only on their conflicts (6,511 of 11,598 events for us). Total labelling cost: about $3.
• The target is the vote distribution, a soft label, not the majority vote.
• Some labels need no model at all: a denied tool call, an interrupt, a tool error, the skill that was loaded and the tool that was called are exact facts in the log.
• Pilot agreement between two labellers: accepted 85%, overreach 90%, stopped-short 82%, satisfaction exact 52% / within one level 93%. That is the ceiling your model is chasing.

6. Rows, soft targets, balance

Each labelled question becomes training pairs (state, hypothesis) → [P(entailment), 0, P(contradiction)]: the true criterion with p, the false criterion with 1 − p; a Choice or Score becomes one row per option with that option's vote share. Training minimises log loss against these soft targets, so the model learns your uncertainty too, not just the majority.

Two things we added after the first runs:

• Balance the rare outcomes. Real logs are lopsided: 99% of logged actions were allowed, and "the agent did more than asked" was only 4% yes. A head can score 0.985 accuracy by always saying "allowed". Repeat the minority outcome per question (we cap it at 8×) and, more importantly, judge heads by AUC, not accuracy.
• Split by time, not at random. The validation set is the latest sessions, never trained on. Random splits leak the same session's near-duplicate turns into validation and flatter every number.

7. Open data that is safe to add

Public Jev-style datasets help the general heads (routing, reading documents, code review), not your taste. Add them with generic wording ("a reviewer accepts this code"), never with your personal question texts: only your logs may speak for you. Vet the provenance of every set; several popular ones are distilled from a commercial Jev and must not be used.

Excluded after reading the cards: a distillation corpus whose card states its teacher is a commercial Jev, two benchmarks that ship a commercial Jev's predictions, sets labelled by an unnamed teacher model or by a closed LLM, and the distilled rows of an otherwise human-labelled mixture. In the sets we used, mind duplicated option texts (identical hypotheses with opposite labels) and cap each source so none dominates.

8. Training on a rented GPU

The recipe is deliberately boring: microsoft/mdeberta-v3-base (multilingual, licence-clean), max length 512, 2 epochs, AdamW 2e-5, bf16, plus ~51k general NLI rows replayed so it does not forget how to read. About 330k training rows; one run is ~650k–840k examples.

1. Smoke on CPU firstEvery training stage on a tiny sample (teacher, export, eval, benchmark), on any CPU box. The launcher refuses to rent a GPU without a passing smoke.
2. Dry runCounts the rows, confirms your personal rows actually made it into the merge, prints the estimate and the cheapest suitable offer. Rents nothing.
3. Rent, upload, preflightCheapest verified GPU under a price ceiling; attach your SSH key; upload scripts and data with sha256 checks; preflight checks CUDA, disk, RAM and GPU memory at the run's real shapes, then runs the smoke again on the GPU.
4. Train, export, evaluateTeacher training, fp32 ONNX export with an ONNX-vs-PyTorch agreement check (0 flips on 300 pairs), evaluation on your held-out logs and a public benchmark.
5. Pull, then destroyA finisher pulls the results (weights in 8 parallel byte ranges, sha256-checked) and destroys the box.

Length bucketing is the cheapest speed-up we found: shuffle, then sort each window of 100 batches by length so a batch pads to similar lengths, then shuffle the batch order again. Same recipe, a third faster. Moving the export's agreement check from CPU to the GPU saved another hour of paid idle time per run.

9. Don't lose money on idle GPUs

Every dollar we wasted came from a finished or broken box that nobody destroyed. Four layers, each covering the one above:

1. An EXIT trap in the launcherIf anything fails before the job starts, the box is destroyed. Use EXIT, not ERR: bash skips ERR traps for failures inside functions, and that once left a box idle for 4.7 hours.
2. A finisher, detached from your terminalWaits for == DONE (or a failure marker), pulls, destroys. Start it in its own session (setsid); a finisher started from an agent's shell dies when the session restarts, and the finished box bills on.
3. A watchdogmin($ cap, 2 × estimate + 30 min), then destroy no matter what. A dollar cap alone is a 10-hour leash on a cheap GPU.
4. A reaper serviceA launchd job every 5 minutes, independent of any session: restarts a dead finisher on a finished job, destroys a box whose GPU sat idle 45 minutes without one, and anything older than 10 hours. Plus caffeinate -w <finisher pid>: a sleeping laptop pauses finishers and sleep timers alike.

10. Evaluate on your own future

The only number that matters is how well the model predicts your later sessions. For each question, take the true-criterion rows from the held-out (latest) sessions and compute the AUC of P(entailment) against what actually happened. Then keep or drop each new version with one rule:

KEEP  if the mean AUC over your personal questions rises
      and no strong question (AUC ≥ 0.85) drops by more than 0.01
DROP  otherwise; change one thing and run again

Accuracy lies on lopsided questions; AUC doesn't. And look at the base rates: a head at 0.985 accuracy on a question that is 99% "yes" has learned nothing.

Calibrate after the gate (per-question temperature, fitted on half the held-out logs and kept only where the other half's calibration error falls). It is free and needs no retraining; the server applies p' = sigmoid(logit(p)/T) per question.

11. Serve it on CPU

Export fp32 ONNX (model, tokenizer.json, config.json, calibration.json) and serve it with any System One compatible server on ONNX Runtime. On a recent laptop CPU:

• Check parity once: replay a few thousand held-out pairs through the live server and compare the per-question AUC with the training eval. It caught a server limit that silently refused 8% of our pairs (a 64-token hypothesis cap; our routing questions are longer).
• Skip int8 for this model family: dynamic int8 flipped 69–71% of answers on an x86 CPU without VNNI (7.7% with reduced range), and on Apple silicon it was no faster, because mDeBERTa's 250k-token embedding table stays fp32. For CPU speed, distil a smaller student instead.
• Run the server as a service (launchd with KeepAlive, interactive priority): a busy laptop compiling in other terminals starved it to 2–9 s per request until it got priority.

12. Put it to work: hook, CLI, skill

A Stop hook that judges every agent turn

A global Claude Code Stop hook rebuilds the last turn with the same parser and state format as training, asks P(accept) and P(wanted more), and, in block mode, sends the agent back with a specific reason:

myjev (the owner's decision model): the owner would likely NOT accept this as it is (P(accept) 0.07):
verify the result and show the evidence; the owner would likely want MORE (P 0.70): finish the missing
part yourself instead of handing steps back. If this is already right, say so briefly with the evidence and stop.

• Start in shadow mode (score and log, never block) and score in the background, so an agent's stop waits ~0.1 s.
• Never block twice in a row (stop_hook_active), and fail open: no server means a normal stop.
• After a few days, line every would-block verdict up with what you actually said next. Switch to block mode only if those were real pushbacks.

A CLI

myjev hook install --mode shadow     # add the Stop hook (backs up settings, touches only its own entry)
myjev hook uninstall                 # remove it, other hooks untouched
myjev hook mode block                # shadow | block | off
myjev hook review                    # verdicts next to what you said next
myjev judge --request "..." --proposal "..." --action "Bash: go test ./..."
myjev ask "Which should the agent do next?" --context "..." \
  --option "stop=Stop and report, wait for my call" --option "run=Start the paid run now"

An agent skill

A skill tells every agent when to consult the model: judge your own turn before handing back; before asking the human "A or B?", ask the model and act only on a confident answer (choice confidence ≥ 0.6, yes/no P ≤ 0.2 or ≥ 0.8), otherwise ask the human and mention the model's lean. Never as a safety gate.

13. Results so far

First version, held-out on the latest sessions (AUC of P(entailment) against what really happened):

The clearest thing it learned is a taste for evidence. The same one-line fix scored P(accept) 0.007 as a bare "Fixed X" and 0.995 with the grep that proves it. Its clearest shortcut: a long list of actions reads as "did the work", so scope creep with many tool calls still scores high. Balancing alone (second version) helped five questions and hurt three, and failed the gate; the next versions add the vetted open data aimed at exactly the weak questions.

14. Rules we would not break

• Never train on a commercial Jev's outputs. Its terms forbid training an imitating model on them, and several "open" datasets are distilled from one. Read every card.
• Keep the personal model personal. It is trained on your logs, which contain client work and personal details, and small models can repeat training text. Serve it locally; publish the method, not the weights.
• Mind your labellers' terms. Some model providers restrict using their outputs to build competing models. Fine for a private model you run for yourself; check before any release.
• Secrets never reach a prompt or a file. Redact, scan every file before it leaves the machine, and pass keys to commands through a broker, never on the command line.

15. Lessons that cost money

Start this weekend

1. Count your decision events (a few thousand turns is enough to start).
2. Extract turns and redact secrets; check every file.
3. Label with two or three small models and a dollar cap; add a bigger model only on their conflicts.
4. Build rows with soft targets and a time split; smoke every stage on a CPU.
5. One GPU run behind a watchdog and a detached finisher (about $1).
6. Gate it on your latest sessions by AUC; serve it locally; put the hook in shadow mode and watch.

Then the loop that makes it better: every day of your normal work becomes new labelled data about you.