Untuned (off-the-shelf) models, measured before and during the fine-tune program through Ollama. 'routing' = the production command-routing prompt (~5.6k tokens); 'content' = 11 generation workloads (summaries, tagging, narration and similar) scored by a judge model. 'floor' tier = unambiguous cases every acceptable model must pass; 'hard' tier = deliberately ambiguous cases. Tournament numbers below were recomputed from the raw per-cycle score records by time window and checked against the program's written notes where they quote them.
Untuned model tournament
Counts are passed/total. Blended is the pass fraction across floor and hard tiers. Median seconds per case is for floor and hard tiers.
Model
Role and notes
Routing floor
Routing hard
Routing blended
Median s/case (floor, hard)
Content floor
Content hard
Content blended
Floor block
Hard block
Resident VRAM (GB)
Measured on
gemma4:12b-it-qat
production router before the fine-tune; baseline for every version
qwen3.5:9b vs its q8_0: same 98% pass rate, q8 2.2x slower (6.3 vs 2.9 s/case) and 3.3 GB more VRAM
Caveat
Earlier, easier case set, all 12 workloads pooled; not comparable to the tiered blocks. LFM2 has no context-shift support and surfaced the num_ctx overflow as HTTP 400 errors (see gotchas), so its routing scores from this window are suspect.
Measured on
Tournament blocks
Each block is a time window of the soak run that produced the scores above.
Block
Window
Note
pre-0908 (12 workloads, no tier split)
2026-09-07T22:48Z to 2026-09-08T19:32Z
pre-0908 hard
2026-09-08T21:06Z to 2026-09-09T01:41Z
floor1
2026-09-09T03:24Z to 04:21Z
hard1
2026-09-09T04:23Z to 07:20Z
route1
2026-09-09T07:23Z to 15:38Z
superseded by route1b; heavy contention (median s/case 10-38)
route1b
2026-09-09T15:38Z to 16:29Z
the routing result the program notes and ledger quote
content1
2026-09-09T17:07Z to 2026-09-10T04:14Z
e2bgranite_content1
2026-09-13T03:18Z to 10:38Z
filled the content gap for e2b and granite; ran under contention from the live router
The untuned 12B on each test-set era
Test set
Passed
Accuracy
Wall clock s
S per case
Note
Held out passed
Held out total
Held out accuracy
flags
Measured on
550 cases
469
0.8527
1012.9
865 cases
691
0.7988
2.124
865, train-flagged slice
0.76
Ledger-only. Recomputing with today's train flags gives 79.5% (667/839) because flags changed after 09-18.
unverified
865, held-out slice
0.941
Ledger-only. With today's train flags the held-out slice is 26 cases and the 12B scores 24/26 (92.3%).
unverified
1,125 runnable, harness fixed
924
0.8213
2.685
96
108
0.8889
Comparison to the fine-tunes
Statement
On the 1,125-case set the rank-112 fine-tunes route about 2x faster than the untuned 12B (1.22-1.34 vs 2.69 s/case) and are 11-14 pp more accurate overall; v9 was the first to beat it on held-out (91.7% vs 88.9%).
Measured on
Warm head-to-head
Method
evict, cold-load, then repeated warm calls through the production routing code path, under low system load
Results
Model
Cold load s
Warm route s
Warm 200 token generation s
qwen3.5:9b
8.8
Median: 6.03; Min: 6.03; Max: 6.79
6.73
gemma4:e2b-it-qat
5.3
Median: 6.03; Min: 6.03; Max: 6.03
3.04
granite4.2:3b-q8_0
2.9
Median: 6.03; Min: 6.02; Max: 6.03
4.08
Note
warm-route times converge on ~6.03 s for all three with no speed-up across same-prefix calls, which the program notes reads as the static prompt prefix not being cached call to call. The 200-token generation column is the clean comparison.
Measured on
Sliding-window cache reuse test
Method
6 trials per model, each with fresh ~300-char context through the real routing path; static ~5.6k tokens + dynamic tail = 6.2-6.5k tokens
Results
Model
Sliding window
Trial0 s
Trials1 5 s
Median s
Fast plateau
gemma4:12b-it-qat
1024
269.96
49-56
53.8
no
gemma4:e2b-it-qat
512
60.74
11-12
11.76
yes
granite4.2:3b-q8_0
142.14
23-33
29.2
no
qwen3.5:9b
142.79
24-36
26.0
no
Confound
system load rose from ~3.0 to ~6-8 between the gemma runs and the granite/qwen runs; a clean test would run all four back to back