# Untuned baselines

> The untuned model tournament and the untuned 12B scored on each test-set era.

Canonical: https://marvinlabs.org/baselines
Data: [baselines.json](https://marvinlabs.org/data/baselines.json)
HTML: https://marvinlabs.org/baselines

Untuned (off-the-shelf) models, measured before and during the fine-tune program through Ollama. 'routing' = the production command-routing prompt (~5.6k tokens); 'content' = 11 generation workloads (summaries, tagging, narration and similar) scored by a judge model. 'floor' tier = unambiguous cases every acceptable model must pass; 'hard' tier = deliberately ambiguous cases. Tournament numbers below were recomputed from the raw per-cycle score records by time window and checked against the program's written notes where they quote them.

## Untuned model tournament {#tournament}

Counts are passed/total. Blended is the pass fraction across floor and hard tiers. Median seconds per case is for floor and hard tiers.

| Model | Role and notes | Routing floor | Routing hard | Routing blended | Median s/case (floor, hard) | Content floor | Content hard | Content blended | Floor block | Hard block | Resident VRAM (GB) | Measured on |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| `gemma4:12b-it-qat` | production router before the fine-tune; baseline for every version | 150/150 | 130/150 | 0.933 | 3.0, 4.5 | 490/490 | 323/330 | 0.991 | Routing: 30/30; Content: 123/123 | Routing: 34/45; Content: 80/82 | Ledger GB: 8.8; Soak scores max size VRAM GB: 8.5; Soak scores typical GB: 4.76 | 2026-09-09 |
| `gemma4:e2b-it-qat` | first-choice fine-tune base; abandoned for tooling reasons (see gotchas) | 150/150 | 120/150 | 0.9 | 1.0, 1.2 | 476/490 | 315/330 | 0.965 | unavailable (load errors in that block) | Routing: 35/45; Content: 77/82 | Soak scores max size VRAM GB: 1.79 | 2026-09-13 |
| `granite4.2:3b-q8_0` | hard-tier mean 116.2 s vs median 27.6 s per workload record under contention; one record took 1357.88 s | 140/150 | 129/150 | 0.897 | 0.7, 0.9 | 488/490 | 306/330 | 0.968 | Routing: 28/30; Content: 120/123 | Routing: 35/45; Content: 78/82 | Ledger GB: 4.7; Soak scores max size VRAM GB: 4.44 | 2026-09-13 |
| `qwen3.5:9b` | the original fine-tune plan, dropped; hybrid SSM/attention (qwen35.ssm.* fields in ollama show), not a plain dense model | 140/150 | 109/150 | 0.83 | 1.4, 1.5 | 476/480 | 304/330 | 0.963 |  | Routing: 34/45; Content: 76/82 | Ledger GB: 5.8; Soak scores max size VRAM GB: 6.32 | 2026-09-09 |
| `qwen3.5:4b-q8_0` | not in the program notes's or ledger's comparison table; numbers are from the raw records only | 150/150 | 120/150 | 0.9 | 1.2, 1.6 | 489/490 | 299/330 | 0.961 |  | Routing: 36/45; Content: 77/82 | Soak scores max size VRAM GB: 5.31 | 2026-09-09 |
| `nemotron-3-nano:4b-q8_0` | floor tier only; dropped after floor1 |  |  |  |  |  |  |  | Routing: 26/30; Content: 122/123 |  | Soak scores max size VRAM GB: 4.46 | 2026-09-09 |
| `qwen3.5:4b-q4_K_M, qwen3.5:9b-q8_0, LFM2.5-2.6B (Q8_0, Q4_K_M)` |  |  |  |  |  |  |  |  |  |  |  | 2026-09-08 |

### Details: qwen3.5:4b-q4_K_M, qwen3.5:9b-q8_0, LFM2.5-2.6B (Q8_0, Q4_K_M) {#details-qwen3-5-4b-q4-k-m-qwen3-5-9b-q8-0-lfm2-5-2-6b-q8-0-q4-k-m}

#### Pre0908 summary {#details-qwen3-5-4b-q4-k-m-qwen3-5-9b-q8-0-lfm2-5-2-6b-q8-0-q4-k-m-pre0908-summary}

| Model | Pass | S per case | VRAM GB | Routing workload |
| --- | --- | --- | --- | --- |
| qwen3.5:4b-q8_0 | 778/781 | 8.7 | 5.1 |  |
| qwen3.5:4b-q4_K_M | 774/781 | 10.9 | 2.2 |  |
| qwen3.5:9b | 768/781 | 2.9 | 5.8 |  |
| qwen3.5:9b-q8_0 | 760/778 | 6.3 | 9.1 |  |
| LFM2.5-2.6B Q8_0 | 621/703 | 12.4 | 3.0 | 0.68 |
| LFM2.5-2.6B Q4_K_M | 607/703 | 6.5 | 1.8 | 0.58 |

#### Pre0908 hard summary {#details-qwen3-5-4b-q4-k-m-qwen3-5-9b-q8-0-lfm2-5-2-6b-q8-0-q4-k-m-pre0908-hard-summary}

| Model | Pass | S per case |
| --- | --- | --- |
| qwen3.5:4b-q8_0 | 530/607 | 9.9 |
| qwen3.5:9b | 520/607 | 10.0 |

- **Q8 vs q4 note:** qwen3.5:9b vs its q8_0: same 98% pass rate, q8 2.2x slower (6.3 vs 2.9 s/case) and 3.3 GB more VRAM
- **Caveat:** Earlier, easier case set, all 12 workloads pooled; not comparable to the tiered blocks. LFM2 has no context-shift support and surfaced the num_ctx overflow as HTTP 400 errors (see gotchas), so its routing scores from this window are suspect.
- **Measured on:** 2026-09-08

### Tournament blocks {#tournament-blocks}

Each block is a time window of the soak run that produced the scores above.

| Block | Window | Note |
| --- | --- | --- |
| pre-0908 (12 workloads, no tier split) | 2026-09-07T22:48Z to 2026-09-08T19:32Z |  |
| pre-0908 hard | 2026-09-08T21:06Z to 2026-09-09T01:41Z |  |
| floor1 | 2026-09-09T03:24Z to 04:21Z |  |
| hard1 | 2026-09-09T04:23Z to 07:20Z |  |
| route1 | 2026-09-09T07:23Z to 15:38Z | superseded by route1b; heavy contention (median s/case 10-38) |
| route1b | 2026-09-09T15:38Z to 16:29Z | the routing result the program notes and ledger quote |
| content1 | 2026-09-09T17:07Z to 2026-09-10T04:14Z |  |
| e2bgranite_content1 | 2026-09-13T03:18Z to 10:38Z | filled the content gap for e2b and granite; ran under contention from the live router |

## The untuned 12B on each test-set era {#untuned-12b-on-gold-sets}

| Test set | Passed | Accuracy | Wall clock s | S per case | Note | Held out passed | Held out total | Held out accuracy | flags | Measured on |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 550 cases | 469 | 0.8527 | 1012.9 |  |  |  |  |  |  | 2026-09-14 |
| 865 cases | 691 | 0.7988 |  | 2.124 |  |  |  |  |  | 2026-09-17 |
| 865, train-flagged slice |  | 0.76 |  |  | Ledger-only. Recomputing with today's train flags gives 79.5% (667/839) because flags changed after 09-18. |  |  |  | [unverified] | 2026-09-18 |
| 865, held-out slice |  | 0.941 |  |  | Ledger-only. With today's train flags the held-out slice is 26 cases and the 12B scores 24/26 (92.3%). |  |  |  | [unverified] | 2026-09-18 |
| 1,125 runnable, harness fixed | 924 | 0.8213 |  | 2.685 |  | 96 | 108 | 0.8889 |  | 2026-09-19 |

## Comparison to the fine-tunes {#comparison-to-finetunes}

- **Statement:** On the 1,125-case set the rank-112 fine-tunes route about 2x faster than the untuned 12B (1.22-1.34 vs 2.69 s/case) and are 11-14 pp more accurate overall; v9 was the first to beat it on held-out (91.7% vs 88.9%).
- **Measured on:** 2026-09-21

## Warm head-to-head {#warm-head-to-head}

- **Method:** evict, cold-load, then repeated warm calls through the production routing code path, under low system load

### Results {#warm-head-to-head-results}

| Model | Cold load s | Warm route s | Warm 200 token generation s |
| --- | --- | --- | --- |
| qwen3.5:9b | 8.8 | Median: 6.03; Min: 6.03; Max: 6.79 | 6.73 |
| gemma4:e2b-it-qat | 5.3 | Median: 6.03; Min: 6.03; Max: 6.03 | 3.04 |
| granite4.2:3b-q8_0 | 2.9 | Median: 6.03; Min: 6.02; Max: 6.03 | 4.08 |

- **Note:** warm-route times converge on ~6.03 s for all three with no speed-up across same-prefix calls, which the program notes reads as the static prompt prefix not being cached call to call. The 200-token generation column is the clean comparison.
- **Measured on:** 2026-09-13

## Sliding-window cache reuse test {#swa-cache-reuse-test}

- **Method:** 6 trials per model, each with fresh ~300-char context through the real routing path; static ~5.6k tokens + dynamic tail = 6.2-6.5k tokens

### Results {#swa-cache-reuse-test-results}

| Model | Sliding window | Trial0 s | Trials1 5 s | Median s | Fast plateau |
| --- | --- | --- | --- | --- | --- |
| gemma4:12b-it-qat | 1024 | 269.96 | 49-56 | 53.8 | no |
| gemma4:e2b-it-qat | 512 | 60.74 | 11-12 | 11.76 | yes |
| granite4.2:3b-q8_0 |  | 142.14 | 23-33 | 29.2 | no |
| qwen3.5:9b |  | 142.79 | 24-36 | 26.0 | no |

- **Confound:** system load rose from ~3.0 to ~6-8 between the gemma runs and the granite/qwen runs; a clean test would run all four back to back
- **Measured on:** 2026-09-13