AISOLATE THE TARGET
Target only
Only the target hypothesis is visible. Request only its label.
1 visible1 output
COMPARATIVE EVALUATION · CONTRACT INFERENCE
Same score.
Different decisions.
Cost, accuracy, and consistent correctness
in contract inference
What does it cost to get a contract judgment right—and keep it right? We evaluate accuracy, response time, and decision consistency across ten models and controlled changes to the request.
Preprint · arXiv:2609.27678 · Code and records are publicly available.
10evaluated models
123test contracts
2,091baseline judgments / model
12responses / fixed target
01 / THE QUESTION
An average can hide
the decisions that changed.
A contract and its target hypothesis stay fixed. Yet showing other hypotheses, requesting more labels, or changing output order can change the answer.
Corrections and regressions can cancel in aggregate accuracy. And a model that always agrees with itself can still be consistently wrong.
Follow a fixed judgmentILLUSTRATIVE EXAMPLE4 fixed judgments
Request AFirst configuration
✓✓✓×
75%Request BChanged configuration
✓✓×✓
75%Same accuracyTwo decisions changed
02 / THE PROTOCOL
One contract. One target.
Four ways to ask.
We preserve the contract, target hypothesis, and gold label. Each condition is repeated three times, yielding twelve responses for each of 30 fixed targets.
BEXPAND THE CONTEXT
Full catalog, one answer
All 17 hypotheses are visible. Request only the target label.
17 visible1 output
CEXPAND THE OUTPUT
All answers, target first
Keep the full catalog. Request all labels, with the target first.
17 visible17 outputs
DCHANGE OUTPUT ORDER
All answers, target last
Keep the full catalog. Request all labels, with the target last.
17 visible17 outputs
THE STRICTER QUESTION
Is the target correct in all twelve responses?
Agreement alone is insufficient. We separately count consistently correct targets, stable errors, changing valid labels, and targets with an invalid response.
Official ContractNLI test split; classification only. The baseline jointly requests all 17 labels once per contract. The 30-target panel is a subset of the baseline contracts. Extra outputs in C/D do not add primary targets. For Jev, C/D changes question-key order, which differs from autoregressive output positioning.
03 / THE EVIDENCE
A trade-off,
not a single winner.
Jev has the lowest recorded cost and median response time in these configurations. Hosted language models achieve higher baseline accuracy. Persistent correctness tells a different story.
Accuracy and the cost of a decision
123 contracts · 2,091 judgments per model
JevHosted LMsLocal LMs
Jev 1.13.077.38% accuracy$0.000228 / contract1.24 s median23 / 30 all twelve correct
Jev 1.13.0Each point is one evaluated configuration. Descriptive 95% whole-contract bootstrap intervals are available in the full metrics. The horizontal axis uses a logarithmic scale. Select a point or use Tab and Enter to inspect it. Scroll the chart horizontally on small screens.
Higher accuracy has a price.
Flash-Lite gains 5.16 percentage points over Jev at about 6× its baseline cost. Gemini Pro has the highest observed accuracy, at 83.21%.
The ranking can change.
Sonnet has 24 of 30 targets correct in all twelve responses; Jev has 23. Small gaps on this panel do not establish a general stability advantage.
Consistency can be wrong.
Jev produces the same wrong label in all twelve responses for 5 of 30 targets. Stable errors are reported separately from persistent correctness.
The measured configurations
Full metrics| Model | Accuracy ↑ | All 12 correct ↑ | Cost / contract ↓ | Median time ↓ |
|---|---|---|---|---|
| Jev 1.13.0 | 77.38% | 23 / 30 | $0.000228 | 1.24 s |
| Gemini 3.5 Flash-Lite | 82.54% | 21 / 30 | $0.001353 | 1.60 s |
| Gemini 3.1 Pro Preview | 83.21% | 22 / 30 | $0.008474 | 3.73 s |
| GPT-5.6 Luna | 78.96% | 22 / 30 | $0.000931 | 1.74 s |
| GPT-5.6 Terra | 82.59% | 20 / 30 | $0.011839 | 11.61 s |
| GPT-6 Astra | 81.83% | 22 / 30 | $0.091890 | 20.94 s |
| Claude Haiku 4.5 | 79.44% | 19 / 30 | $0.005396 | 5.65 s |
| Claude Sonnet 5 | 82.40% | 24 / 30 | $0.011352 | 3.27 s |
| Qwen3.5-4B | 74.22% | 17 / 30 | $0.053653 | 86.87 s |
| Qwen3.5-9B | 79.15% | 18 / 30 | $0.077361 | 134.64 s |
Gemini 3.5 Flash-Lite
GPT-5.6 Luna
Claude Haiku 4.5
Qwen3.5-4BHow to read these results
Cost is configuration-specific. API costs use reported charges or recorded token rates. Local estimates value observed GPU time at $2.00/GPU-hour, excluding loading, idle allocation, storage, transfer, and taxes.
Time includes the deployment. Median client elapsed time includes preparation, network, service processing, and validation. It does not isolate model computation or time to first token.
Settings are unequal. Reasoning effort, generation limits, interfaces, hardware, and providers differ. Sonnet, Haiku, and Terra were added after inspecting earlier results.
The sample bounds the claim. The repeated panel has 30 independent contracts, not 360. Intervals use 5,000 whole-contract bootstrap draws and are descriptive, without multiple-comparison adjustment.
4,830 recorded attempts · 13 invalid responses. Failures count as incorrect and remain in cost and response-time summaries. No failed attempt was repaired, replaced, or retried.
04 / REPRODUCE & EXTEND
From claims
back to records.
The benchmark release includes sanitized prediction records, frozen reference metrics, model configurations, and offline analysis code.
Recompute the recorded results after installing dependencies. New evaluation requests are a separate, explicit step and may incur charges.
Open the benchmarkOFFLINE REPRODUCTION
git clone https://github.com/ZF-Utokyo/Jev-Benchmark.git
cd Jev-Benchmark
python -m venv .venv
source .venv/bin/activate
python -m pip install -e .
python -m analysis.reproduce --check
python -m analysis.figures
python -m unittest discover -s testsPython 3.10+ · Analysis and plotting make no model calls.