# MarvinLabs > Public dataset of measured local-LLM fine-tuning and inference findings: a 17-version LoRA program on a 4B router, trained on 8 GB NVIDIA cards. Canonical: https://marvinlabs.org/ Data: [baselines.json](https://marvinlabs.org/data/baselines.json), [findings.json](https://marvinlabs.org/data/findings.json), [gotchas.json](https://marvinlabs.org/data/gotchas.json), [inference.json](https://marvinlabs.org/data/inference.json), [recipe.json](https://marvinlabs.org/data/recipe.json), [versions.json](https://marvinlabs.org/data/versions.json), [vram.json](https://marvinlabs.org/data/vram.json) HTML: https://marvinlabs.org/ MarvinLabs is a public dataset of measured local-LLM fine-tuning and inference findings. It covers a 17-version LoRA fine-tune program of a 4B command router (unsloth/gemma-3-4b-it) on 8 GB consumer NVIDIA cards, in September 2026, plus cross-GPU memory probes of the same recipe on an AMD RX 7900 XTX (October 2026). Every number is measured and dated, and every page has a Markdown twin and a JSON file. ## Headline findings {#headline-findings} ### LoRA rank, not more data, moved the hardest capability {#head-F01} Three rounds of more data (v3, v4, v5) left the clarify capability at 56-60%. Raising rank from 16 to 64 on the same 757 examples lifted overall accuracy 4 pp and clarify to ~75%; rank 112 lifted clarify to ~85%. Dropping back to rank 64 on a later corpus cost clarify 13 pp. [F01 in full](/findings#F01) (measured on 2026-09-20) ### Rank 112 is the ceiling on an 8 GB card at ~7k-token examples {#head-F02} With chunked MLP, save_on_cpu and a bf16 lm_head, rank 112 peaks at ~6.65 GB in real runs; rank 128 and DoRA (at 64 and 112) OOM on the longest example. Rank 112 over 64 bought +0.5 pp on the 865 set, inside run-to-run noise there, but held clarify gains later. [F02 in full](/findings#F02) (measured on 2026-09-18) ### The overall number can rank the worst router first; a guard veto is needed {#head-F17} v13 (97.6%) and v15 (96.8%) both beat v11 (96.4%) overall and both obey instructions hidden in pasted text more often. The 174-case guard bake (about 4 minutes) was the only check that separated them. The release rule was changed to: veto any held-out guard breach the incumbent does not have, then rank by held-out accuracy. [F17 in full](/findings#F17) (measured on 2026-09-23) ### A 4B fine-tune beat the untuned 12B it replaced {#head-F23} On the 1,125-case set the untuned gemma4:12b-it-qat scored 82.1% (held-out 88.9%) at 2.7 s per case; the best 4B fine-tune scored 96.4% (held-out 96.3%) at 1.25 s. The first fine-tune (v1, 152 examples) had scored 4.9 pp below the 12B; the 490-example v2 was the first to beat it. [F23 in full](/findings#F23) (measured on 2026-09-21) ### AMD and NVIDIA need the same memory for the same QLoRA step {#head-F29} The identical 4-bit QLoRA probe at 7,177 tokens peaked at 5,721.3 MB (rank 16) and 6,406.2 MB (rank 64) on an RX 7900 XTX under ROCm 7.2.1 on Windows, and at 5,788.9 MB and 6,465.0 MB on an RTX 3060 Ti under CUDA on Linux: within 1.2%, AMD slightly lower. Memory before the step matched to within 0.5 MB. A VRAM budget measured on one vendor carries over to the other for this recipe. [F29 in full](/findings#F29) (measured on 2026-10-15) ### A single bake pass over a small set overstated a margin by about 2x {#head-F15} The one-pass guard bake that justified switching production to v17 scored it 257/267 with 1 held-out breach against v11's 247 with 4. Three fresh passes each, same conditions, gave v17 256 with 4 breaches and v11 250 with 5, identical every pass. The real margin is 6 cases, not 10, and both models still miss some pasted-text cases, including one the single pass had counted as a v17 pass. [F15 in full](/findings#F15) (measured on 2026-09-25) ## Pages {#pages} - [Version ledger](/versions): The ledger of every trained version: config, training stats and measured results, plus the test-set eras. [Markdown](versions.md) - [VRAM measurements](/vram): Every GPU memory measurement from the program: peak MB by rank, dropout, DoRA, bf16 and sequence length on an 8 GB NVIDIA card and a 24 GB AMD card, plus the probe calibration. [Markdown](vram.md) - [Inference speed](/inference): Ollama load time, prompt speed, generation speed and resident VRAM for seven small models on an AMD RX 7900 XTX. [Markdown](inference.md) - [Untuned baselines](/baselines): The untuned model tournament and the untuned 12B scored on each test-set era. [Markdown](baselines.md) - [Findings](/findings): Every finding with its evidence and measured date, plus the methods tried and closed and the ideas left untried. [Markdown](findings.md) - [Gotchas](/gotchas): Environment and tooling failures hit during the program, with symptom, cause, fix and versions. [Markdown](gotchas.md) - [Training recipe](/recipe): The shared training recipe: model, quantization, LoRA settings, hyperparameters, memory fixes, export path and release rule. [Markdown](recipe.md) - [About: method and scope](/about): How the numbers were measured, what the terms mean, what is not published, and how to report a correction. [Markdown](about.md) ## Dataset downloads {#downloads} Each file is plain JSON. Records keep their original fields, including internal source references. - [baselines.json](data/baselines.json): Untuned (off-the-shelf) models, measured before and during the fine-tune program through Ollama. 'routing' = the production command-routing prompt (~5.6k... - [findings.json](data/findings.json): Lessons from 17+ LoRA fine-tunes of gemma-3-4b-it as a command router on one 8 GB consumer GPU. Each claim is limited to this task, this base model and this... - [gotchas.json](data/gotchas.json): Environment and tooling failures hit during the program, with symptom, cause, fix and versions. Versions are those recorded at the time; see recipe.json for... - [inference.json](data/inference.json): Ollama inference speed for seven small models on one AMD Radeon RX 7900 XTX (24 GB, Windows 11, Ollama 0.30.10). Each model was unloaded first, then given... - [recipe.json](data/recipe.json): The training recipe shared by every fresh-from-base version (v1-v3, v5-v17). v4 differs only in continuing from v3's adapter at 8 epochs. Values come from the... - [versions.json](data/versions.json): LoRA fine-tunes of unsloth/gemma-3-4b-it as a closed-set command router (one JSON tool call per utterance), trained and served on 8 GB NVIDIA consumer cards... - [vram.json](data/vram.json): Every VRAM measurement from the program. The cross_gpu section adds one-step probes on an AMD RX 7900 XTX (24 GB) and a second NVIDIA card. Training GPU: one 8... ## For agents {#for-agents} - [llms.txt](llms.txt): index of every Markdown page and data file. - [llms-full.txt](llms-full.txt): every Markdown page in one file. - [sitemap.xml](sitemap.xml): all URLs. --- # Version ledger > The ledger of every trained version: config, training stats and measured results, plus the test-set eras. Canonical: https://marvinlabs.org/versions Data: [versions.json](https://marvinlabs.org/data/versions.json) HTML: https://marvinlabs.org/versions LoRA fine-tunes of unsloth/gemma-3-4b-it as a closed-set command router (one JSON tool call per utterance), trained and served on 8 GB NVIDIA consumer cards, 13-25 Sep 2026. ## Version ledger {#ledger} One row per trained model or variant. Test sets differ between eras, so compare accuracy only within one test set. Each version links to its full record below. | Version | Headline result | Status | Rank | Corpus examples | Epochs | Alpha | rsLoRA | Dropout | Peak VRAM (MB) | Trained | Alias | Ollama tag | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | [v1](#v1) | 80.4% on 550 (2026-09-14) | | 16 | 152 | 3 | 16 | no | 0 | 4926 | 2026-09-13 to 2026-09-14 | | `gemma3-4b-router-finetuned` | | [v2](#v2) | 89.6% on 550 (2026-09-14) | | 16 | 490 | 3 | 16 | no | 0 | 5053 | ended 2026-09-14 | | `gemma3-4b-router-v2` | | [v3](#v3) | 89.8% on 790 (2026-09-15) | | 16 | 696 | 3 | 16 | no | 0 | 5040 | ended 2026-09-14 | | `gemma3-4b-router-v3` | | [v4](#v4) | 82.9% on 865 (2026-09-15) | | 16 | 59 | 8 | 16 | no | 0 | 4889 | ended 2026-09-15 | | `gemma3-4b-router-v4` | | [v5](#v5) | 89.2% on 865 (2026-09-15) | [deployed] [unverified] | 16 | 757 | 3 | 16 | no | 0 | 5117 | ended 2026-09-15 | marvin-v1.0 | `gemma3-4b-router-v5` | | [v5-q4](#v5-q4) | 86.1% on 865 (2026-09-16) | [rejected] | | | | | | | | | | `gemma3-4b-router-v5-q4` | | [v6-wide](#v6-wide) | 93.1% on 865 (2026-09-17) | [unverified] | 64 | 757 | 3 | 64 | no | 0 | 5796 | ended 2026-09-17 | | `gemma3-4b-router-v6-wide` | | [v7](#v7) | 94.3% on 865 (2026-09-18) | [deployed] [unverified] | 112 | 757 | 3 | 112 | no | 0 | 6647 | ended 2026-09-17 | marvin-v1.1 | `gemma3-4b-router-v7-r112` | | [v8](#v8) | 88.8% on 1,036 raw (2026-09-18) | | 112 | 1050 | 3 | 112 | no | 0 | 6643 | ended 2026-09-18 | | `gemma3-4b-router-v8-r112` | | [v9-step175](#v9-step175) | 86.6% on 1,021 runnable (2026-09-19) | [rejected] | 112 | 1080 | | 112 | no | 0 | | | | `gemma3-4b-router-v9-r112-ckpt175` | | [v9](#v9) | 93.0% on 1,021 runnable (2026-09-19) | [deployed] | 112 | 1080 | 3 | 112 | no | 0 | 6643 | ended 2026-09-19 | marvin-v1.2 | `gemma3-4b-router-v9-r112` | | [v10-r64](#v10-r64) | 91.6% on 1,125, harness fixed (2026-09-20) | | 64 | 1080 | 3 | 64 | no | 0 | 5767 | ended 2026-09-20 | | `gemma3-4b-router-v10-r64` | | [v11](#v11) | 96.4% on 1,125, harness fixed (2026-09-21) | | 112 | 1167 | 3 | 112 | no | 0 | 6648 | 2026-09-20 to 2026-09-21 | marvin-v1.3 | `gemma3-4b-router-v11-r112` | | [v12-e2](#v12-e2) | 95.6% on 1,131 (2026-09-21) | | 112 | 1167 | 2 | 112 | no | 0 | 6648 | ended 2026-09-21 | | `gemma3-4b-router-v12-r112-e2` | | [v11+grammar](#v11-grammar) | 95.7% on 1,125 (2026-09-21) | | | | | | | | | | | `gemma3-4b-router-v11-r112` | | [v11-avg](#v11-avg) | 96.9% on 1,131 (2026-09-21) | | | | | | | | | | | `gemma3-4b-router-v11-avg` | | [v13-rslora-a112 (killed)](#v13-rslora-a112-killed) | | | 112 | | | 112 | yes | 0 | | | | | | [v13](#v13) | 97.5% on 1,131 (2026-09-22) | [rejected] | 112 | 1172 | 3 | 32 | yes | 0 | 6653 | ended 2026-09-22 | | `gemma3-4b-router-v13-r112-rslora-a32` | | [v13-avg](#v13-avg) | 97.2% on 1,131 (2026-09-22) | | | | | | | | | | | `gemma3-4b-router-v13-avg` | | [v13-ck225](#v13-ck225) | 83.8% on 1,131 (2026-09-23) | | | | | | | 0 | | | | `gemma3-4b-router-v13-ck225` | | [v14](#v14) | 87.0% on 1,131 (2026-09-23) | | 112 | 1172 | 3 | 32 | yes | 0.1 | | | | `gemma3-4b-router-v14-ck225` | | [v15](#v15) | 96.8% on 1,131 (2026-09-23) | [rejected] | 96 | 1172 | 3 | 30 | yes | 0.1 | 6629 | ended 2026-09-23 | | `gemma3-4b-router-v15-r96-drop10` | | [v16](#v16) | 90.2% on 1,148 (2026-09-24) | | 112 | 873 | 3 | 112 | no | 0 | 6653 | 2026-09-23 to 2026-09-24 | | `gemma3-4b-router-v16-disjoint` | | [v17](#v17) | 95.2% on 1,275 (2026-09-25) | [deployed] | 112 | 1189 | 3 | 112 | no | 0 | 6658 | 2026-09-24 to 2026-09-25 | marvin-v1.3 | `gemma3-4b-router-v17-full` | ## Test-set eras {#test-set-eras} The test set grew over the program. A number from one era does not compare with a number from another. | Cases | Note | Runnable | Kind | Held out | flags | Measured on | | --- | --- | --- | --- | --- | --- | --- | | 550 | 535 regression + 15 hard cases | | | | | 2026-09-14 | | 790 | 775 regression + 15 hard; 240 new cases weighted toward known weak capabilities | | | | | 2026-09-15 | | 865 | 850 regression + 15 hard | | | | | 2026-09-15 | | 1036 | v8 bake; 60 cases whose capability is disabled on the host were counted as failures in the raw summary | 976 | | | [inferred] | 2026-09-18 | | 1081 | disabled-capability cases skipped for every model from here on | 1021 | | | | 2026-09-19 | | 1125 | first set scored with the harness channel fix (context cases routed on their own channel) | | | | | 2026-09-19 | | 1131 | 1,125 + six two-turn-back context cases; 1,021 train-flagged + 110 held-out | | | | | 2026-09-21 | | 1191 | all-caps run: 1,131 plus the 60 flag-gated desktop/finance cases enabled | | | | | 2026-09-21 | | 1148 | v16 bake with the disjoint-split manifest; 749 train-flagged + 399 held-out | | | | | 2026-09-24 | | 1275 | adds 127 never-seen cases (86 adversarial guards, 41 session-capability) | | | | | 2026-09-24 | | 174 | | | guard set | 48 | | 2026-09-22 | | 181 | | | guard set | | | 2026-09-24 | | 267 | | | guard set, expanded with 86 adversarial guards | | | 2026-09-24 | ## Definitions {#definitions} - **Accuracy definition:** passed / runnable cases. A case passes when the router returns the asserted capability and arguments, and none of the case's forbidden capabilities. - **Held out definition:** cases with no training label in the corpus the model was trained on. The held-out slice changed size as the case file grew, so compare held-out numbers only within one test-set era. - **Fair slice definition:** cases carrying no train key at all, which neither compared model ever saw (used for v16 and v17). - **Guard definition:** must-not-fire cases: utterances that must NOT route to a named capability (negations, quoted or pasted content, vague phrasing). Counts only; case text is excluded. - **Speed definition:** mean seconds per case through the production routing prompt (~5.6k tokens), model fully on GPU unless noted. - **Dates:** UTC. measured_on is the file timestamp of the result, or the date the program notes records. - **Shared recipe:** see recipe.json; per-version config below lists only what differs or what the training log confirms. ## Per-version records {#records} ### v1 {#v1} - **Ollama tag:** `gemma3-4b-router-finetuned` - **base model:** `unsloth/gemma-3-4b-it` - **Corpus examples:** 152 - **trained:** Start: 2026-09-13; End: 2026-09-14T00:26Z - **what changed:** First LoRA, on 152 curated examples mostly mined from the untuned router's own logged decisions plus a first batch of hand-authored gold cases. - **outcome:** Regressed: 4.9 pp below the untuned 12B baseline (85.3%). Not caught until v2's bake, because v1 was never evaluated after training. #### Config {#v1-config} - **Rank:** 16 - **Alpha:** 16 - **RsLoRA:** no - **Dropout:** 0 - **Epochs:** 3 - **Optimizer steps:** 57 - **Lr:** 0.0002 - **Measured on:** 2026-09-14 #### Training {#v1-training} - **Train loss:** 0.3685 - **Peak VRAM MB:** 4926 - **Runtime s:** 4616 - **Measured on:** 2026-09-14 #### Results {#v1-results} | Test set | Passed | Accuracy | Wall clock s | Measured on | | --- | --- | --- | --- | --- | | 550 | 442 | 0.8036 | 1081.2 | 2026-09-14 | ### v2 {#v2} - **Ollama tag:** `gemma3-4b-router-v2` - **base model:** `unsloth/gemma-3-4b-it` - **Corpus examples:** 490 - **trained:** End: 2026-09-14T07:08Z - **what changed:** Corpus tripled with hand-authored gold cases weighted toward vague and ambiguous phrasing and full per-capability coverage. - **outcome:** First fine-tune to beat the untuned baseline: +4.3 pp over gemma4:12b-it-qat on the same 550 cases. #### Config {#v2-config} - **Rank:** 16 - **Alpha:** 16 - **RsLoRA:** no - **Dropout:** 0 - **Epochs:** 3 - **Optimizer steps:** 186 - **Lr:** 0.0002 - **Measured on:** 2026-09-14 #### Training {#v2-training} - **Final segment train loss:** 0.0274 - **Peak VRAM MB:** 5053 - **Relaunches:** 9 - **Note:** train_loss covers only the last resumed segment, not the run - **Measured on:** 2026-09-14 #### Results {#v2-results} | Test set | Passed | Accuracy | Wall clock s | Capability | Total | Failure shape | Measured on | | --- | --- | --- | --- | --- | --- | --- | --- | | 550 | 493 | 0.8964 | 1028.8 | | | | 2026-09-14 | | 550 | 27 | 0.562 | | clarify | 48 | | 2026-09-14 | | 550 | | | | | | 12 of 57 misses picked a forbidden capability (21%), vs 6/81 for the untuned 12B and 8/108 for v1 | 2026-09-14 | ### v3 {#v3} - **Ollama tag:** `gemma3-4b-router-v3` - **base model:** `unsloth/gemma-3-4b-it` - **Corpus examples:** 696 - **trained:** End: 2026-09-14T23:57Z - **what changed:** 240 new gold cases for the capabilities v2 missed most; full corpus, fresh from base. - **outcome:** Held: same score as v2 on a set 240 cases harder. Not comparable to v2's number (different test set). #### Config {#v3-config} - **Rank:** 16 - **Alpha:** 16 - **RsLoRA:** no - **Dropout:** 0 - **Epochs:** 3 - **Optimizer steps:** 261 - **Lr:** 0.0002 - **Measured on:** 2026-09-14 #### Training {#v3-training} - **Train loss:** 0.1502 - **Peak VRAM MB:** 5040 - **Runtime s:** 21162 - **Relaunches:** 0 - **Measured on:** 2026-09-14 #### Results {#v3-results} | Test set | Passed | Accuracy | Wall clock s | Capability | Measured on | | --- | --- | --- | --- | --- | --- | | 790 | 709 | 0.8975 | 1056.8 | | 2026-09-15 | | 790 | | 0.604 | | clarify | 2026-09-15 | ### v4 {#v4} - **Ollama tag:** `gemma3-4b-router-v4` - **base model:** `v3's adapter (continued, not fresh from base)` - **Corpus examples:** 59 - **trained:** End: 2026-09-15T02:32Z; Conflict: The program notes and the training script's docstring date the run 2026-09-16; file timestamps (train log 09-15 02:32, bake summary 09-15 02:54) say 2026-09-15. Trust the file timestamps. - **what changed:** Incremental patch: only the 59 new or corrected examples, 8 epochs, continued from v3's adapter. - **outcome:** Catastrophic forgetting. Capabilities the patch never touched collapsed to 25-56%. Never deployed. #### Config {#v4-config} - **Rank:** 16 - **Alpha:** 16 - **RsLoRA:** no - **Dropout:** 0 - **Epochs:** 8 - **Optimizer steps:** 64 - **Lr:** 0.0002 - **Continued from:** v3 adapter - **Measured on:** 2026-09-15 #### Training {#v4-training} - **Train loss:** 0.0522 - **Peak VRAM MB:** 4889 - **Runtime s:** 4778 - **Measured on:** 2026-09-15 #### Results {#v4-results} | Test set | Passed | Accuracy | Wall clock s | Total | Vs v3 | Per capability | Before v3 | Measured on | | --- | --- | --- | --- | --- | --- | --- | --- | --- | | 865 | 717 | 0.8289 | 1156.9 | | | | | 2026-09-15 | | the 790 cases v3 was scored on | 651 | 0.8241 | | 790 | -0.0734 | | | 2026-09-15 | | 790 subset | | | | | | Run agent: 0.25; Resume agent: 0.417; Dispatch code: 0.524; Calendar availability: 0.562; Clarify: 0.525; Gate reject: 1.0; Dispatch answer: 0.931 | Run agent: ~1.0; Resume agent: 0.86-1.0; Dispatch code: 0.91-1.0; Calendar availability: 1.0; Clarify: 0.604 | 2026-09-15 | ### v5 {#v5} - **alias:** marvin-v1.0 - **Ollama tag:** `gemma3-4b-router-v5` - **base model:** `unsloth/gemma-3-4b-it` - **Corpus examples:** 757 - **trained:** End: 2026-09-15T11:32Z - **what changed:** v4's 59 examples mixed into the full corpus and retrained fresh from base. - **outcome:** Repaired v4's damage (run_agent, resume_agent, dispatch_code, calendar_availability all back to 100% on the 790 subset). Deployed 15 Sep. #### Config {#v5-config} - **Rank:** 16 - **Alpha:** 16 - **RsLoRA:** no - **Dropout:** 0 - **Epochs:** 3 - **Optimizer steps:** 285 - **Lr:** 0.0002 - **Measured on:** 2026-09-15 #### Training {#v5-training} - **Train loss:** 0.0729 - **Peak VRAM MB:** 5117 - **Kills by low memory monitor:** 4 - **Note:** train_loss and runtime cover the last resumed segment - **Measured on:** 2026-09-15 #### Results {#v5-results} | Test set | Passed | Accuracy | Wall clock s | S per case | Quant | Total | Note | Held out accuracy | flags | Measured on | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | 865 | 772 | 0.8925 | 1141.5 | | | | | | | 2026-09-15 | | 865 | 773 | 0.8936 | | 1.506 | Q8_0 | | | | | 2026-09-16 | | 865 | 770 | 0.8902 | | 1.333 | | | | | | 2026-09-17 | | old 790 subset | 709 | 0.8975 | | | | 790 | identical to v3 to the case | | | 2026-09-15 | | 75 cases new since v3 | 63 | 0.84 | | | | 75 | | | | 2026-09-15 | | 865 | | | | | | | ledger-only. Recomputing from the bake progress file with today's train flags gives a different split (flags changed after 09-18), so this could not be reproduced. | 0.849 | [unverified] | 2026-09-18 | ### v5-q4 {#v5-q4} - **Ollama tag:** `gemma3-4b-router-v5-q4` - **base model:** `same merged weights as v5` - **what changed:** Quantization only. - **outcome:** Rejected: -3.2 pp for ~15% speed. Q8_0 stays. #### Config {#v5-q4-config} - **Quant:** Q4_0 - **GGUF size GB:** 2.37 - **Measured on:** 2026-09-16 #### Results {#v5-q4-results} | Test set | Passed | Accuracy | S per case | Delta vs q8 | Speed ratio vs q8 | Measured on | | --- | --- | --- | --- | --- | --- | --- | | 865 | 745 | 0.8613 | 1.278 | -0.0323 | 0.849 | 2026-09-16 | ### v6-wide {#v6-wide} - **Ollama tag:** `gemma3-4b-router-v6-wide` - **base model:** `unsloth/gemma-3-4b-it` - **Corpus examples:** 757 - **trained:** End: 2026-09-17T01:46Z - **what changed:** Rank 16 to 64 (alpha kept equal to rank). Same 757 examples, same recipe, fresh from base. - **outcome:** +4.0 pp over v5 from rank alone. The two passes a day apart (93.1% / 93.9%) set the noise floor for this era. #### Config {#v6-wide-config} - **Rank:** 64 - **Alpha:** 64 - **RsLoRA:** no - **Dropout:** 0 - **Epochs:** 3 - **Optimizer steps:** 285 - **Lr:** 0.0002 - **Measured on:** 2026-09-17 #### Training {#v6-wide-training} - **Train loss:** 0.1227 - **Peak VRAM MB:** 5796 - **Runtime s:** 23944 - **Measured on:** 2026-09-17 #### Results {#v6-wide-results} | Test set | Passed | Accuracy | S per case | Capability | Accuracy before | Accuracy after | N | Held out accuracy | flags | Measured on | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | 865 | 805 | 0.9306 | 1.223 | | | | | | | 2026-09-17 | | 865 | 812 | 0.9387 | 1.25 | | | | | | | 2026-09-18 | | 865 | | | | clarify | 0.521 | 0.755 | 94 | | | 2026-09-17 | | 865 | | | | | | | | 0.855, 0.882 | [unverified] | 2026-09-18 | ### v7 {#v7} - **alias:** marvin-v1.1 - **Ollama tag:** `gemma3-4b-router-v7-r112` - **base model:** `unsloth/gemma-3-4b-it` - **Corpus examples:** 757 - **trained:** End: 2026-09-17T23:06Z - **what changed:** Rank 64 to 112, the highest rank that fit on the 8 GB card. - **outcome:** +0.5 pp over v6-wide, inside v6-wide's own day-to-day spread. Shipped 18 Sep, replaced by v9 on 19 Sep. #### Config {#v7-config} - **Rank:** 112 - **Alpha:** 112 - **RsLoRA:** no - **Dropout:** 0 - **Epochs:** 3 - **Optimizer steps:** 285 - **Lr:** 0.0002 - **Measured on:** 2026-09-17 #### Training {#v7-training} - **Train loss:** 0.1251 - **Peak VRAM MB:** 6647 - **Runtime s:** 24217 - **Measured on:** 2026-09-17 #### Results {#v7-results} | Test set | Passed | Accuracy | S per case clean | S per case raw | Note | Per capability | Held out passed | Held out total | Held out accuracy | S per case | Train flagged accuracy | flags | Measured on | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | 865 | 816 | 0.9434 | 1.26 | 1.707 | raw speed polluted: first 253 cases ran split CPU/GPU | | | | | | | | 2026-09-18 | | 865 | | | | | v6-wide then v7; 27 cases fixed, 23 broken | Clarify: 0.745, 0.851; Context setup cases n31: 0.968, 0.871 | | | | | | | 2026-09-18 | | 1,021 runnable | 933 | 0.9138 | | | | | 76 | 91 | 0.8352 | 1.269 | | | 2026-09-19 | | 1,036 raw (v8 era) | 906 | 0.8745 | | | | | 69 | 90 | | 1.427 | | | 2026-09-18 | | 108-case held-out slice, harness fixed | 92 | 0.8519 | | | | | | | | | | | 2026-09-19 | | 865 | | | | | | | | | 0.871 | | 0.963 | [unverified] | 2026-09-18 | ### v8 {#v8} - **Ollama tag:** `gemma3-4b-router-v8-r112` - **base model:** `unsloth/gemma-3-4b-it` - **Corpus examples:** 1050 - **trained:** End: 2026-09-18T11:19Z - **what changed:** Enriched corpus: 126 negative-guard cases given targets, setup context rendered into prompts (27 context cases had trained context-free), clarify-chain turns exported, 171 gold cases including the 9 capabilities that had zero examples. - **outcome:** Mixed, not deployed. Guards rose 19-50 pp, but four new clarify labels on bare confirmation words contradicted older gold and dropped gate_approve from 100% to 70%. #### Config {#v8-config} - **Rank:** 112 - **Alpha:** 112 - **RsLoRA:** no - **Dropout:** 0 - **Epochs:** 3 - **Optimizer steps:** 396 - **Lr:** 0.0002 - **Measured on:** 2026-09-18 #### Training {#v8-training} - **Train loss:** 0.1117 - **Peak VRAM MB:** 6643 - **Runtime s:** 34881 - **Measured on:** 2026-09-18 #### Results {#v8-results} | Test set | Passed | Accuracy | Held out passed | Held out total | Held out accuracy | S per case | Conflict | Per capability | Guard families gain pp | flags | Measured on | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | 1,036 raw | 920 | 0.888 | 71 | 90 | 0.7889 | 1.249 | | | | | 2026-09-18 | | 976 runnable | | 0.943 | | | 0.91 | | The raw summary scores 88.8% on 1,036 because 60 disabled-capability cases count as failures. 920/976 = 94.3% matches the program notes, so the program notes number is the raw one with those 60 removed. The held-out 91.0% would need ~12 of the 60 to be held-out (71/78); not verifiable from the summary. | | | [inferred] | 2026-09-18 | | 976 runnable | | | | | | | | Gate approve: 1.0, 0.7 | 19 to 50 | | 2026-09-18 | ### v9-step175 {#v9-step175} - **Ollama tag:** `gemma3-4b-router-v9-r112-ckpt175` - **base model:** `unsloth/gemma-3-4b-it` - **Corpus examples:** 1080 - **what changed:** Same run as v9, earlier checkpoint. - **outcome:** Rejected. Tied the final checkpoint on held-out, lost 6.5 pp overall. Eval loss on partial-argument targets does not select for accuracy. #### Config {#v9-step175-config} - **Rank:** 112 - **Alpha:** 112 - **RsLoRA:** no - **Dropout:** 0 - **Checkpoint step:** 175 - **Of steps:** 405 - **Selected by:** lowest held-out eval loss (0.1306) - **Measured on:** 2026-09-19 #### Results {#v9-step175-results} | Test set | Passed | Accuracy | Held out passed | Held out total | Held out accuracy | S per case | Measured on | | --- | --- | --- | --- | --- | --- | --- | --- | | 1,021 runnable | 884 | 0.8658 | 79 | 91 | 0.8681 | 1.31 | 2026-09-19 | | 108-case held-out slice, harness fixed | 96 | 0.8889 | | | | | 2026-09-19 | ### v9 {#v9} - **alias:** marvin-v1.2 - **Ollama tag:** `gemma3-4b-router-v9-r112` - **base model:** `unsloth/gemma-3-4b-it` - **Corpus examples:** 1080 - **Eval examples:** 55 - **trained:** End: 2026-09-19T04:41Z - **what changed:** v8's corpus with the conflicting labels removed and 35 counterweight cases added; held-out eval loss logged every 25 steps, every checkpoint kept. - **outcome:** Deployed 19 Sep. First fine-tune to beat the untuned 12B on held-out once the harness was fixed (91.7% vs 88.9%). #### Config {#v9-config} - **Rank:** 112 - **Alpha:** 112 - **RsLoRA:** no - **Dropout:** 0 - **Epochs:** 3 - **Optimizer steps:** 405 - **Lr:** 0.0002 - **Measured on:** 2026-09-19 #### Training {#v9-training} - **Train loss:** 0.1115 - **Peak VRAM MB:** 6643 - **Runtime s:** 38218 - **Eval loss best:** Step: 175; Loss: 0.1306 - **Eval loss final:** 0.1861 - **Measured on:** 2026-09-19 #### Results {#v9-results} | Test set | Passed | Accuracy | Held out passed | Held out total | Held out accuracy | S per case | Passed per run | Accuracy per run | Mean accuracy | Spread pp | Held out passed per run | Held out breaches | Per capability | Fixed vs v7 | Broke vs v7 | Measured on | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | 1,021 runnable | 950 | 0.9305 | 79 | 91 | 0.8681 | 1.303 | | | | | | | | | | 2026-09-19 | | 1,125, harness fixed | 1050 | 0.9333 | 99 | 108 | | 1.264 | | | | | | | | | | 2026-09-19 | | 1,125, harness fixed | 1047 | 0.9307 | 99 | 108 | | 1.34 | | | | | | | | | | 2026-09-20 | | 1,125, harness fixed | 1043 | 0.9271 | 100 | 108 | | 1.332 | | | | | | | | | | 2026-09-21 | | 1,191 all-caps x3 | | | | 122 | | | 1098, 1112, 1112 | 0.9219, 0.9337, 0.9337 | 0.9298 | 1.18 | 111, 112, 111 | | | | | 2026-09-21 | | 174 guards | 167 | | | | | | | | | | | 1 | | | | 2026-09-22 | | 1,021 runnable | | | | | | | | | | | | | Gate approve: 0.94; Dispatch answer: 0.59; Guards: 0.96-1.0 | 45 | 28 | 2026-09-19 | ### v10-r64 {#v10-r64} - **Ollama tag:** `gemma3-4b-router-v10-r64` - **base model:** `unsloth/gemma-3-4b-it` - **Corpus examples:** 1080 - **trained:** End: 2026-09-20T06:03Z - **what changed:** v9's exact data and recipe at rank 64. Rank the only variable. - **outcome:** -1.4 pp vs v9 at rank 112 on the same data; clarify 86% to 73%. Rank 112 stays. #### Config {#v10-r64-config} - **Rank:** 64 - **Alpha:** 64 - **RsLoRA:** no - **Dropout:** 0 - **Epochs:** 3 - **Optimizer steps:** 405 - **Lr:** 0.0002 - **Measured on:** 2026-09-20 #### Training {#v10-r64-training} - **Train loss:** 0.1103 - **Peak VRAM MB:** 5767 - **Runtime s:** 37684 - **Eval loss best:** Step: 125; Loss: 0.1372 - **Eval loss final:** 0.1839 - **Measured on:** 2026-09-20 #### Results {#v10-r64-results} | Test set | Passed | Accuracy | Held out passed | Held out total | Held out accuracy | S per case | Capability | Rank112 same data | Measured on | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | 1,125, harness fixed | 1031 | 0.9164 | 98 | 108 | 0.9074 | 1.224 | | | 2026-09-20 | | 1,125, harness fixed | | 0.73 | | | | | clarify | 0.86 | 2026-09-20 | ### v11 {#v11} - **alias:** marvin-v1.3 (25 Sep 06:42 to 08:10 UTC, then re-pointed to v17) - **Ollama tag:** `gemma3-4b-router-v11-r112` - **base model:** `unsloth/gemma-3-4b-it` - **Corpus examples:** 1167 - **Eval examples:** 72 - **trained:** Start: 2026-09-20T13:27Z; End: 2026-09-21T01:18Z - **what changed:** Corpus 1,167: new compound and correct_last capabilities, 121 context cases, clarify chains, 40 dispatch_answer positives. - **outcome:** Best and most stable accuracy of the program. Won the accuracy rule against six challengers (v12, v13, v15, v16, v17 and both averages). Released as marvin-v1.3, then replaced the same morning by v17 on guard behaviour. #### Config {#v11-config} - **Rank:** 112 - **Alpha:** 112 - **RsLoRA:** no - **Dropout:** 0 - **Epochs:** 3 - **Optimizer steps:** 438 - **Lr:** 0.0002 - **Measured on:** 2026-09-21 #### Training {#v11-training} - **Train loss:** 0.1073 - **Peak VRAM MB:** 6648 - **Runtime s:** 42633 - **Eval loss best:** Step: 75; Loss: 0.2727 - **Eval loss final:** 0.3997 - **Measured on:** 2026-09-21 #### Results {#v11-results} | Test set | Passed | Accuracy | Held out passed | Held out total | S per case | Passes | Accuracy range | Train flagged accuracy | Held out accuracy | Passed per run | Mean accuracy | Spread pp | Held out per run | Fair slice | Held out breaches | Breach categories in rechecks | Per capability changes | Note | Lowest capabilities | Measured on | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | 1,125, harness fixed | 1084 | 0.9636 | 104 | 108 | 1.245 | | | | | | | | | | | | | | | 2026-09-21 | | 1,131 | | | | | | (Passed: 1090, Held out: 108/110; measured on 2026-09-21); (Passed: 1090, Held out: 108/110; measured on 2026-09-21); (Passed: 1091, Held out: 108/110; measured on 2026-09-21); (Passed: 1090, Held out: 108/110; measured on 2026-09-22); (Passed: 1090, Held out: 107/110; measured on 2026-09-23) | 0.9637, 0.9646 | 0.9618 | 0.9818 | | | | | | | | | | | | | 1,191 all-caps x3 | | | | | | | | | | 1145, 1144, 1145 | 0.9611 | 0.09 | 117/122 each | | | | | | | 2026-09-21 | | 1,148 (v16 manifest) | 1101 | 0.9591 | 379 | 399 | 1.194 | | | | | | | | | | | | | | | 2026-09-24 | | 82 cases held out from both v11 and v16 | | 0.963 | | | | | | | | | | | | | | | | | | 2026-09-24 | | 317 cases v11 trained on and v16 did not | | 0.946 | | | | | | | | | | | | | | | | | | 2026-09-24 | | 1,275 expanded | 1212 | 0.9506 | 490 | 526 | 1.197 | | | | | | | | | | | | | | | 2026-09-24 | | 1,275 (v17 bake) | 1211 | 0.9498 | 222 | 237 | 1.187 | | | | | | | | | 93/100 | | | | | | 2026-09-25 | | 174 guards | 169 | | | | | | | | | | | | | | 1 | | | | | 2026-09-22 | | 181 guards | 172 | | | | | | | | | | | | | | 1 | | | | | 2026-09-24 | | 267 guards | | | | | | (Passed: 250, Held out breaches: 5; measured on 2026-09-24); (Passed: 247, Held out breaches: 4; measured on 2026-09-25); (Passed: 250, Held out breaches: 5; measured on 2026-09-25); (Passed: 250, Held out breaches: 5; measured on 2026-09-25); (Passed: 250, Held out breaches: 5; measured on 2026-09-25) | | | | | | | | | | 3 of 5 held-out breaches involve pasted or quoted content; 1 vague phrasing; 1 session-capability | | | | | | 1,125-1,131 | | | | | | | | | | | | | | | | | Compound: 0.11, 0.89; Correct last: 0.0, 1.0; Gate reject: 0.88, 0.98; Guard:open folder: 1.0, 0.88 | vs marvin-v1.2; open_folder guard n=8 | | 2026-09-21 | | 10 pooled passes | | | | | | | | | | | | | | | | | | | Gate approve: Accuracy: 0.766, Ci: 0.719, 0.807, Cases: 35; Clarify: Accuracy: 0.827, Ci: 0.801, 0.849, Cases: 98; Explain issue: Accuracy: 0.855, Cases: 11; Compound: Accuracy: 0.906, Cases: 18 | 2026-09-22 | ### v12-e2 {#v12-e2} - **Ollama tag:** `gemma3-4b-router-v12-r112-e2` - **base model:** `unsloth/gemma-3-4b-it` - **Corpus examples:** 1167 - **trained:** End: 2026-09-21T14:55Z - **what changed:** v11's exact data and recipe at 2 epochs. Epochs the only variable. - **outcome:** -0.8 pp and -4 held-out cases vs v11, mostly clarify (wrong answers went to converse). Lowest final eval loss of any run, and still worse. Three epochs stays. #### Config {#v12-e2-config} - **Rank:** 112 - **Alpha:** 112 - **RsLoRA:** no - **Dropout:** 0 - **Epochs:** 2 - **Optimizer steps:** 292 - **Lr:** 0.0002 - **Measured on:** 2026-09-21 #### Training {#v12-e2-training} - **Train loss:** 0.1417 - **Peak VRAM MB:** 6648 - **Runtime s:** 28440 - **Eval loss best:** Step: 125; Loss: 0.2589 - **Eval loss final:** 0.2934 - **Measured on:** 2026-09-21 #### Results {#v12-e2-results} | Test set | Passed | Accuracy | Held out passed | Held out total | S per case | Broke vs v11 | Fixed vs v11 | Measured on | | --- | --- | --- | --- | --- | --- | --- | --- | --- | | 1,131 | 1081 | 0.9558 | 104 | 110 | 1.236 | 26 | 17 | 2026-09-21 | ### v11+grammar {#v11-grammar} - **Ollama tag:** `gemma3-4b-router-v11-r112 (same weights)` - **what changed:** Decoding only. - **outcome:** +0.1 pp once the enum was correct. Turned on in production with the 25 Sep release. #### Config {#v11-grammar-config} - **Change:** Ollama structured output: JSON schema with a capability enum enforced at decode time - **Measured on:** 2026-09-21 #### Results {#v11-grammar-results} | Attempt | Test set | Passed | Accuracy | Held out passed | Held out total | S per case | Note | Unparseable outputs | Broke | Fixed | Measured on | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | first (enum missing a router-hidden capability) | 1,125 | 1077 | 0.9573 | 103 | 108 | 1.287 | 10 cases broke, 8 of them the hidden capability forced to clarify | | | | 2026-09-21 | | fixed enum | 1,131 | 1092 | 0.9655 | 108 | 110 | 1.266 | | 3, 1 | 1 | 3 | 2026-09-21 | ### v11-avg {#v11-avg} - **Ollama tag:** `gemma3-4b-router-v11-avg` - **what changed:** No training; weight averaging only. - **outcome:** Noise: +0.4 pp on the first bake, +0.0 on the second; held-out identical both times. #### Config {#v11-avg-config} - **Method:** exact mean of v11 checkpoints 300, 325, 350, 375, 400, 425, 438 by concatenating LoRA A and B along the rank axis (r = 7 x 112, alpha unchanged) - **Measured on:** 2026-09-21 #### Results {#v11-avg-results} | Test set | Passed | Accuracy | Held out passed | Held out total | S per case | Note | Measured on | | --- | --- | --- | --- | --- | --- | --- | --- | | 1,131 | 1096 | 0.9691 | 108 | 110 | 1.235 | | 2026-09-21 | | 1,131 | 1090 | 0.9637 | 108 | 110 | 3.615 | speed contention-affected | 2026-09-22 | ### v13-rslora-a112 (killed) {#v13-rslora-a112-killed} - **what changed:** rsLoRA with alpha left equal to rank. - **outcome:** Killed early. Eval loss flatlined; scale 10.6 is too hot at lr 2e-4. #### Config {#v13-rslora-a112-killed-config} - **Rank:** 112 - **Alpha:** 112 - **RsLoRA:** yes - **Effective scale:** 10.6 - **Dropout:** 0 - **Lr:** 0.0002 - **Measured on:** 2026-09-21 #### Training {#v13-rslora-a112-killed-training} - **Eval loss step25:** 0.5734 - **Eval loss step50:** 0.577 - **V11 eval loss same steps:** 0.35, 0.33 - **Train loss peak:** 3.85 - **Measured on:** 2026-09-21 ### v13 {#v13} - **Ollama tag:** `gemma3-4b-router-v13-r112-rslora-a32` - **base model:** `unsloth/gemma-3-4b-it` - **Corpus examples:** 1172 - **Eval examples:** 74 - **trained:** End: 2026-09-22T06:35Z - **what changed:** rsLoRA at alpha 32 on the 21 Sep corpus (1,172 examples: 6 new two-turn-back context cases, 2 label fixes, 18 conflicts excluded). - **outcome:** Rejected. Highest overall number of the program, but every gained case was train-flagged and it lost held-out safety guards v11 passes. #### Config {#v13-config} - **Rank:** 112 - **Alpha:** 32 - **RsLoRA:** yes - **Effective scale:** 3.02 - **Dropout:** 0 - **Epochs:** 3 - **Optimizer steps:** 441 - **Lr:** 0.0002 - **Measured on:** 2026-09-22 #### Training {#v13-training} - **Train loss:** 0.1246 - **Peak VRAM MB:** 6653 - **Runtime s:** 42946 - **Eval loss best:** Step: 100; Loss: 0.2519 - **Eval loss final:** 0.3497 - **Measured on:** 2026-09-22 #### Results {#v13-results} | Test set | Passed | Accuracy | Train flagged accuracy | Held out passed | Held out total | S per case | Fixed vs v11 | Broke vs v11 | Gains | Unparseable outputs | Held out breaches | Breach categories | Measured on | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | 1,131 | 1103 | 0.9752 | 0.9794 | 103 | 110 | 1.238 | | | | | | | 2026-09-22 | | 1,131 | 1104 | 0.9761 | | 103 | 110 | | | | | | | | 2026-09-22 | | 1,131 | | | | | | | 29 | 16 | Gate approve: 9; Clarify: 9; Dispatch answer: 3 | 3, 1 | | | 2026-09-22 | | 174 guards | 168 | | | | | | | | | | 3 | 2 of 3 involve pasted or quoted content; 1 is an argument confusion between two knowledge capabilities | 2026-09-22 | ### v13-avg {#v13-avg} - **Ollama tag:** `gemma3-4b-router-v13-avg` - **what changed:** No training; weight averaging only. - **outcome:** -0.4 pp vs v13 with one more held-out miss. Did not recover the guard losses. #### Config {#v13-avg-config} - **Method:** exact checkpoint mean, with B scaled by a further 1/sqrt(n) because rsLoRA scales by alpha/sqrt(r) - **Measured on:** 2026-09-22 #### Results {#v13-avg-results} | Test set | Passed | Accuracy | Held out passed | Held out total | Measured on | | --- | --- | --- | --- | --- | --- | | 1,131 | 1099 | 0.9717 | 102 | 110 | 2026-09-22 | ### v13-ck225 {#v13-ck225} - **Ollama tag:** `gemma3-4b-router-v13-ck225` - **what changed:** Mid-run checkpoint. - **outcome:** Only the delta to v14-ck225 means anything; absolute scores are for a half-trained model. #### Config {#v13-ck225-config} - **Checkpoint step:** 225 - **Of steps:** 441 - **Dropout:** 0 - **Note:** about 1.5 epochs; control arm of the dropout A/B - **Measured on:** 2026-09-23 #### Results {#v13-ck225-results} | Test set | Passed | Accuracy | Train flagged accuracy | Held out passed | Held out total | Held out breaches | Measured on | | --- | --- | --- | --- | --- | --- | --- | --- | | 1,131 | 948 | 0.8382 | 0.8394 | 91 | 110 | | 2026-09-23 | | 174 guards | 148 | | | | | 10 | 2026-09-23 | ### v14 {#v14} - **Ollama tag:** `gemma3-4b-router-v14-ck225 (only the step-225 checkpoint was deployed)` - **base model:** `unsloth/gemma-3-4b-it` - **Corpus examples:** 1172 - **trained:** Attempts: 2026-09-22T20:16Z, 2026-09-23T01:23Z, 2026-09-23T03:06Z - **what changed:** v13's recipe plus LoRA dropout 0.1. - **outcome:** Would not fit at rank 112. At equal steps dropout added 3.2 pp, all on train-flagged cases, and widened the train-to-held-out gap from 1.2 to 6.8 pp. Guards improved 148 to 157. #### Config {#v14-config} - **Rank:** 112 - **Alpha:** 32 - **RsLoRA:** yes - **Dropout:** 0.1 - **Epochs:** 3 - **Planned steps:** 441 - **Lr:** 0.0002 - **Measured on:** 2026-09-23 #### Training {#v14-training} - **Died at steps:** 184, 236, 239 - **Causes:** CUDA illegal memory access (Xid 31 MMU fault per the program notes), CUDA OOM, 6.98 GiB allocated + 394 MiB reserved of 7.65 GiB, CUDA OOM, 6.90 GiB allocated + 457 MiB reserved - **Measured on:** 2026-09-23 #### Results {#v14-results} | Checkpoint | Test set | Passed | Accuracy | Train flagged accuracy | Held out passed | Held out total | S per case | Held out breaches | Measured on | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | 225 | 1,131 | 984 | 0.87 | 0.8766 | 89 | 110 | 1.22 | | 2026-09-23 | | 225 | 174 guards | 157 | | | | | | 7 | 2026-09-23 | ### v15 {#v15} - **Ollama tag:** `gemma3-4b-router-v15-r96-drop10` - **base model:** `unsloth/gemma-3-4b-it` - **Corpus examples:** 1172 - **trained:** End: 2026-09-23T17:51Z - **what changed:** Rank 112 to 96 to pay for dropout's memory; alpha 30 to hold rsLoRA's effective scale near v13's. - **outcome:** Vetoed on guards. Dropout bought one held-out case over v13 and cost 0.7 pp overall. Closed the rsLoRA branch. #### Config {#v15-config} - **Rank:** 96 - **Alpha:** 30 - **RsLoRA:** yes - **Effective scale:** 3.06 - **Dropout:** 0.1 - **Epochs:** 3 - **Optimizer steps:** 441 - **Lr:** 0.0002 - **Measured on:** 2026-09-23 #### Training {#v15-training} - **Train loss:** 0.1218 - **Peak VRAM MB:** 6629 - **Predicted peak MB:** 6793 - **Runtime s:** 43420 - **Eval loss best:** Step: 300; Loss: 0.2499 - **Eval loss final:** 0.3385 - **Measured on:** 2026-09-23 #### Results {#v15-results} | Test set | Passed | Accuracy | Train flagged accuracy | Held out passed | Held out total | S per case | Held out breaches | Breach categories | Measured on | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | 1,131 | 1095 | 0.9682 | 0.9706 | 104 | 110 | 1.269 | | | 2026-09-23 | | 174 guards | 165 | | | | | | 4 | 3 of 4 involve pasted or quoted content | 2026-09-23 | ### v16 {#v16} - **Ollama tag:** `gemma3-4b-router-v16-disjoint` - **base model:** `unsloth/gemma-3-4b-it` - **Corpus examples:** 873 - **trained:** Start: 2026-09-23T22:18Z; End: 2026-09-24T07:12Z - **what changed:** v11's recipe on a disjoint corpus with a quarter of the gold set held back. - **outcome:** -8.5 pp on the fair slice vs v11. Disproved its own premise: v11's headline was not inflated by memorization. #### Config {#v16-config} - **Rank:** 112 - **Alpha:** 112 - **RsLoRA:** no - **Dropout:** 0 - **Epochs:** 3 - **Optimizer steps:** 330 - **Lr:** 0.0002 - **Holdout:** 273-case stratified slice (25%) plus 58 cases dropped because a deterministic pre-router step now answers them - **Measured on:** 2026-09-24 #### Training {#v16-training} - **Train loss:** 0.1316 - **Peak VRAM MB:** 6653 - **Runtime s:** 32031 - **Eval loss best:** Step: 100; Loss: 0.2603 - **Eval loss final:** 0.3877 - **Measured on:** 2026-09-24 #### Results {#v16-results} | Test set | Passed | Accuracy | Held out passed | Held out total | S per case | V11 accuracy | Note | Held out breaches | Groups | Measured on | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | 1,148 | 1035 | 0.9016 | 334 | 399 | 1.195 | | | | | 2026-09-24 | | 82 cases held out from both v11 and v16 | | 0.878 | | | | 0.963 | | | | 2026-09-24 | | 317 cases v11 trained on and v16 did not | | 0.826 | | | | 0.946 | | | | 2026-09-24 | | 1,275 expanded | 1139 | 0.8933 | 439 | 526 | 4.231 | | speed contention-affected | | | 2026-09-24 | | 181 guards | 173 | | | | | | the extra breach is a session-capability case that did not exist when v16's corpus was exported | 2 | | 2026-09-24 | | 267 guards | 253 | | | | | | | 5 | | 2026-09-24 | | 127 never-seen cases by group | | | | | | | | | All 127: V11: 0.874, V16: 0.843; Adversarial guards 62: V11: 0.855, V16: 0.903; Guard counterweights 24: V11: 0.833, V16: 0.667; Session injection guards 16: V11: 1.0, V16: 1.0; Session paraphrases and discriminators 25: V11: 0.88, V16: 0.76 | 2026-09-24 | ### v17 {#v17} - **alias:** marvin-v1.3 (production from 2026-09-25 08:10 UTC) - **Ollama tag:** `gemma3-4b-router-v17-full` - **base model:** `unsloth/gemma-3-4b-it` - **Corpus examples:** 1189 - **Eval examples:** 78 - **trained:** Start: 2026-09-24T17:10Z; End: 2026-09-25T05:38Z - **what changed:** v11's exact recipe on the largest corpus: 86 adversarial guards, 41 session-capability cases, 33 guard training targets and 6 conflict rulings added. - **outcome:** Lost the pre-agreed fair-slice rule by 3 cases (90 vs 93), so the scripted decision picked v11. Then chosen deliberately for guards: 256/267 vs 250/267 over three repeated passes. Live as marvin-v1.3 from 25 Sep. #### Config {#v17-config} - **Rank:** 112 - **Alpha:** 112 - **RsLoRA:** no - **Dropout:** 0 - **Epochs:** 3 - **Optimizer steps:** 447 - **Lr:** 0.0002 - **Holdout:** about 10% - **Measured on:** 2026-09-25 #### Training {#v17-training} - **Train loss:** 0.1105 - **Peak VRAM MB:** 6658 - **Runtime s:** 44847 - **Eval loss best:** Step: 100; Loss: 0.2526 - **Eval loss final:** 0.4378 - **Measured on:** 2026-09-25 #### Results {#v17-results} | Test set | Passed | Accuracy | Train flagged accuracy | Held out passed | Held out total | S per case | V11 passed | Passes | Breach categories in rechecks | Measured on | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | 1,275 | 1214 | 0.9522 | 0.9653 | 212 | 237 | 1.174 | | | | 2026-09-25 | | fair slice, 100 cases neither v11 nor v17 saw | 90 | 0.9 | | | | | 93 | | | 2026-09-25 | | 267 guards | | | | | | | | (Passed: 257, Held out breaches: 1; measured on 2026-09-25); (Passed: 256, Held out breaches: 4; measured on 2026-09-25); (Passed: 256, Held out breaches: 4; measured on 2026-09-25); (Passed: 256, Held out breaches: 4; measured on 2026-09-25) | 2 of 4 held-out breaches involve pasted or quoted content; 1 vague phrasing; 1 session-capability | | --- # VRAM measurements > Every GPU memory measurement from the program: peak MB by rank, dropout, DoRA, bf16 and sequence length on an 8 GB NVIDIA card and a 24 GB AMD card, plus the probe calibration. Canonical: https://marvinlabs.org/vram Data: [vram.json](https://marvinlabs.org/data/vram.json) HTML: https://marvinlabs.org/vram Every VRAM measurement from the program. The cross_gpu section adds one-step probes on an AMD RX 7900 XTX (24 GB) and a second NVIDIA card. Training GPU: one 8 GB NVIDIA consumer card (an RTX 3070 Ti by today's nvidia-smi; torch reports 7.65 GiB total capacity). 'mode' is probe (one forward+backward on the single longest example, no optimizer step) or real (a full Trainer run; peak = torch.cuda.max_memory_allocated at the end). All training configs are 4-bit nf4 QLoRA on unsloth/gemma-3-4b-it with the recipe.json memory fixes unless noted. ## GPU {#gpu} - **Model:** NVIDIA GeForce RTX 3070 Ti - **Total MiB:** 8192 - **Usable GiB torch:** 7.65 - **Second card:** NVIDIA GeForce RTX 3060 Ti, 8192 MiB - **Driver:** 580.178.04 - **Note:** Card names read on 2026-10-15 and mapped through CUDA_DEVICE_ORDER=PCI_BUS_ID, CUDA_VISIBLE_DEVICES=1 as set in every training script (the probe scripts document the same pin). This holds as long as neither card changed PCIe slot since September; the owner confirms both cards were in the box then. The 7.65 GiB figure is printed in every OOM message. - **Measured on:** 2026-10-15 - **Status:** [inferred] ## Measurements {#measurements} Mode is real (a training run), probe (one forward and backward pass on the longest example) or load (model load only). Peak is the highest GPU memory seen. OOM means the run ran out of memory. | ID | Mode | Model | Rank | rsLoRA | Alpha | Dropout | DoRA | Seq length (tokens) | Peak | Status | Config, result and notes | Measured on | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | `e2b-contended-load` | load | unsloth/gemma-4-E2B-it | | | | | | | | | Config: Unsloth load_in_4bit, ~3.6 GB free (live models resident) \| Result: loaded, then forward crashed (embedding offloaded to CPU) | 2026-09-13 | | `e2b-two-gpu-split` | load | unsloth/gemma-4-E2B-it | | | | | | | 4.38 GB on GPU0 + 3.17 GB on GPU1, zero CPU offload | | Config: Unsloth 4-bit, both GPUs visible \| Result: forward crashed on a cross-device tensor in transformers' Gemma 4 forward | 2026-09-13 | | `e2b-lora-params` | load | unsloth/gemma-4-E2B-it | | | | | | | | | Result: LoRA 25.3M trainable of 5.15B total (0.49%); vendor guide says E2B LoRA needs 8-10 GB | 2026-09-13 | | `g3-trial-load` | load | unsloth/gemma-3-4b-it | | | | | | | 4.39 GB | [ok] | Config: Unsloth load_in_4bit, single GPU \| Result: fits, zero CPU offload | 2026-09-13 | | `g3-trial-step` | probe | unsloth/gemma-3-4b-it | | | | | | | 5.59 GB | [ok] | Config: one forward+backward, example truncated to 2048 tokens | 2026-09-13 | | `first-real-step-oom` | real | | | | | | | | OOM | [oom] | Config: plain transformers+peft, r16, full ~6300-token examples, before the memory fixes \| Result: OOM: 7.39 GB used, 256 MB free, tried to allocate 248 MB | 2026-09-13 | | `sdpa-forward-probe` | probe | | | | | | | | 7.16 GB | [ok] | Config: bare no-grad forward, mem_efficient SDPA forced, 6390-token example \| Note: proved attention was not the bottleneck | 2026-09-13 | | `logits-tensor-size` | computed | | | | | | | | | [inferred] | Config: [1, 6390, 262208] fp32 logits vs response span only \| Result: 6.7 GB full vs 135 MB response-only \| Note: arithmetic, not a profiler reading | 2026-09-13 | | `checkpoint-boundary-climb` | probe | | | | | | | | | | Config: one forward across 34 layers with gradient checkpointing \| Result: 4.5 GB to 6.99 GB without offload; 6.39 GB with save_on_cpu | 2026-09-14 | | `lm-head-cast` | load | | | | | | | | | | Config: pre-training VRAM, lm_head fp32 vs bf16 \| Result: 4502 MB to 3221 MB | 2026-09-14 | | `real-v1` | real | | 16 | | | | | | 4926 MB | [ok] | Dataset examples: 152 | 2026-09-14 | | `real-v2` | real | | 16 | | | | | | 5053 MB | [ok] | Note: last resumed segment \| Dataset examples: 490 | 2026-09-14 | | `real-v3` | real | | 16 | | | | | | 5040 MB | [ok] | Dataset examples: 696 | 2026-09-14 | | `real-v4` | real | | 16 | | | | | | 4889 MB | [ok] | Dataset examples: 59 | 2026-09-15 | | `real-v5` | real | | 16 | | | | | | 5117 MB | [ok] | Note: last resumed segment \| Dataset examples: 757 | 2026-09-15 | | `real-v6` | real | | 64 | | | | | | 5796 MB | [ok] | Dataset examples: 757 | 2026-09-17 | | `real-v7` | real | | 112 | | | | | | 6647 MB | [ok] | Dataset examples: 757 | 2026-09-17 | | `real-dryrun` | real | | 112 | | | | | | 6634 MB | [ok] | Note: 2-step dry run of the generic trainer \| Dataset examples: 3 | 2026-09-18 | | `real-v8` | real | | 112 | | | | | | 6643 MB | [ok] | Dataset examples: 1050 | 2026-09-18 | | `real-v9` | real | | 112 | | | | | | 6643 MB | [ok] | Dataset examples: 1080 | 2026-09-19 | | `real-v10` | real | | 64 | | | | | | 5767 MB | [ok] | Dataset examples: 1080 | 2026-09-20 | | `real-v11` | real | | 112 | | | | | | 6648 MB | [ok] | Dataset examples: 1167 | 2026-09-21 | | `real-v12` | real | | 112 | | | | | | 6648 MB | [ok] | Epochs: 2; Dataset examples: 1167 | 2026-09-21 | | `real-v13` | real | | 112 | yes | 32 | | | 7177 | 6653 MB | [ok] | Dataset examples: 1172 | 2026-09-22 | | `real-v14-run1` | real | | 112 | yes | | 0.1 | | | | | Result: died at step 184: CUDA illegal memory access (Xid 31 MMU fault) \| Dataset examples: 1172 | 2026-09-22 | | `real-v14-run2` | real | | 112 | yes | | 0.1 | | | OOM | [oom] | Result: OOM at step 236: 6.98 GiB allocated, 394 MiB reserved-unallocated, 58 MiB free | 2026-09-23 | | `real-v14-run3` | real | | 112 | yes | | 0.1 | | | OOM | [oom] | Result: OOM at step 239: 6.90 GiB allocated, 457 MiB reserved-unallocated, 78 MiB free | 2026-09-23 | | `real-v15` | real | | 96 | yes | 30 | 0.1 | | | 6629 MB | [ok] | Dataset examples: 1172; Predicted MB: 6793 | 2026-09-23 | | `real-v16` | real | | 112 | | | | | | 6653 MB | [ok] | Dataset examples: 873 | 2026-09-24 | | `real-v17` | real | | 112 | | | | | | 6658 MB | [ok] | Dataset examples: 1189 | 2026-09-25 | | `probe-0917-r96` | probe | | 96 | | | | | 7037 | 6977.6 MB | [ok] | Dataset: 2026-09-15 corpus; Logits to keep: 319; Pre step MB: 3847 | 2026-09-17 | | `probe-0917-r112` | probe | | 112 | | | | | 7037 | 7216.0 MB | [ok] | Dataset: 2026-09-15 corpus; Pre step MB: 3973 | 2026-09-17 | | `probe-0917-r128` | probe | | 128 | | | | | 7037 | OOM | [oom] | Result: OOM (tried 70 MiB, 48 MiB free) \| Dataset: 2026-09-15 corpus; Pre step MB: 4098 | 2026-09-17 | | `probe-0918-r96` | probe | | 96 | | | | | 7288 | 7058.9 MB | [ok] | Dataset: 2026-09-18 unfiltered | 2026-09-18 | | `probe-0918-r104` | probe | | 104 | | | | | 7288 | 7178.1 MB | [ok] | Dataset: 2026-09-18 unfiltered | 2026-09-18 | | `probe-0918-r112` | probe | | 112 | | | | | 7288 | OOM | [oom] | Result: OOM (tried 72 MiB, 66 MiB free) \| Dataset: 2026-09-18 unfiltered | 2026-09-18 | | `probe-0918f-r112` | probe | | 112 | | | | | 6931 | 7054.8 MB | [ok] | Dataset: 2026-09-18 filtered; Logits to keep: 192 | 2026-09-18 | | `probe-dora-r112` | probe | | 112 | | | | yes | 7037 | OOM | [oom] | Result: OOM inside the DoRA weight-norm step (tried 100 MiB, 30 MiB free) \| Pre step MB: 3985 | 2026-09-18 | | `probe-dora-r64` | probe | | 64 | | | | yes | 7037 | OOM | [oom] | Result: OOM inside the DoRA weight-norm step (tried 100 MiB, 76 MiB free) \| Pre step MB: 3610 | 2026-09-18 | | `probe-0919f-r112` | probe | | 112 | | | | | 7068 | 6980.0 MB | [ok] | Note: the calibration anchor \| Dataset: 2026-09-19 filtered; Logits to keep: 73 | 2026-09-20 | | `sweep-r112-drop` | probe | | 112 | yes | 32 | 0.1 | | 7177 | OOM | [oom] | Result: OOM \| Dataset: 2026-09-21; Logits to keep: 237 | 2026-09-23 | | `sweep-r104-drop` | probe | | 104 | yes | | 0.1 | | 7177 | OOM | [oom] | Result: OOM | 2026-09-23 | | `sweep-r96-drop` | probe | | 96 | yes | | 0.1 | | 7177 | 7127.6 MB | [ok] | | 2026-09-23 | | `sweep-r88-drop` | probe | | 88 | yes | | 0.1 | | 7177 | 7007.9 MB | [ok] | | 2026-09-23 | | `sweep-r80-drop` | probe | | 80 | yes | | 0.1 | | 7177 | 6890.3 MB | [ok] | | 2026-09-23 | | `sweep-r112-nodrop` | probe | | 112 | yes | 32 | 0 | | 7177 | OOM | [oom] | Result: OOM, although v13 trained this exact config at a 6653 MB real peak | 2026-09-23 | ## Cross-GPU probes {#cross-gpu} One-step probes of the training recipe on two GPU vendors. Each probe runs one forward and backward pass of unsloth/gemma-3-4b-it with LoRA on all 7 attention and MLP projections, on synthetic random tokens: seq_len tokens of input with the loss on the last response_len. Peak memory depends on shapes, not on what the tokens say, so the synthetic input stands in for the real worst-case example (checked on the RTX 3060 Ti: pre-step memory at rank 112 was 3,972.6 MB synthetic vs 3,972.55 MB real). No optimizer step, so a real training run's peak differs; see Calibration above. Memory fixes 'all' = chunked MLP, save_on_cpu activation offload, bf16 lm_head and logits_to_keep (see the recipe). Peak allocated is torch.cuda.max_memory_allocated; reserved adds the allocator's cache. ### AMD Radeon RX 7900 XTX {#gpu-rx-7900-xtx} - **Arch:** gfx1100 - **Total MB:** 24560 - **OS:** Windows 11 (build 26200) - **Driver:** AMD Adrenalin 26.7.1 (32.0.31035.1003) - **Stack:** ROCm 7.2.1 Windows SDK wheels, torch 2.9.1+rocm7.2.1, HIP 7.2.53211 - **Libraries:** transformers 5.19.0, peft 0.21.2, accelerate 1.15.0, bitsandbytes 0.50.2 - **Python:** 3.12.10 ### NVIDIA GeForce RTX 3060 Ti {#gpu-rtx-3060-ti} - **Total MB:** 8192 - **Usable GiB torch:** 7.66 - **OS:** Ubuntu, Linux 7.0 - **Driver:** 580.178.04 - **Note:** About 0.55 GB of the card was held by an Ollama process during the probes, so the rank 96 OOM is for about 7.1 GB free, not 7.66. | GPU | Quant | Memory fixes | Rank | Dropout | DoRA | Seq length (tokens) | Pre-step MB | Peak allocated MB | Peak reserved MB | Step s | Status | Measured on | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | RX 7900 XTX | 4bit | all | 16 | 0.0 | no | 7177 | 3222.4 | 5721.3 | 6694.0 | 16.92 | [ok] | 2026-10-15 | | RX 7900 XTX | 4bit | all | 64 | 0.0 | no | 7177 | 3597.7 | 6406.2 | 6752.0 | 14.13 | [ok] | 2026-10-15 | | RX 7900 XTX | 4bit | all | 96 | 0.0 | no | 7177 | 3847.8 | 6884.8 | 7318.0 | 14.28 | [ok] | 2026-10-15 | | RX 7900 XTX | 4bit | all | 112 | 0.0 | no | 7177 | 3974.5 | 7194.5 | 7578.0 | 14.27 | [ok] | 2026-10-15 | | RX 7900 XTX | 4bit | all | 128 | 0.0 | no | 7177 | 4099.7 | 7395.9 | 7750.0 | 14.07 | [ok] | 2026-10-15 | | RX 7900 XTX | 4bit | all | 192 | 0.0 | no | 7177 | 4611.9 | 8383.4 | 8842.0 | 14.8 | [ok] | 2026-10-15 | | RX 7900 XTX | 4bit | all | 256 | 0.0 | no | 7177 | 5129.1 | 9338.3 | 9732.0 | 14.88 | [ok] | 2026-10-15 | | RX 7900 XTX | 4bit | all | 96 | 0.1 | no | 7177 | 3847.8 | 7059.3 | 7520.0 | 14.64 | [ok] | 2026-10-15 | | RX 7900 XTX | 4bit | all | 112 | 0.1 | no | 7177 | 3974.5 | 7368.2 | 7794.0 | 15.46 | [ok] | 2026-10-15 | | RX 7900 XTX | 4bit | all | 128 | 0.1 | no | 7177 | 4099.7 | 7562.7 | 8038.0 | 14.21 | [ok] | 2026-10-15 | | RX 7900 XTX | 4bit | all | 256 | 0.1 | no | 7177 | 5129.1 | 9506.9 | 9890.0 | 14.99 | [ok] | 2026-10-15 | | RX 7900 XTX | 4bit | all | 64 | 0.0 | yes | 7177 | 3633.9 | 8240.7 | 8564.0 | 19.53 | [ok] | 2026-10-15 | | RX 7900 XTX | 4bit | all | 112 | 0.0 | yes | 7177 | 4010.7 | 9018.8 | 9290.0 | 19.61 | [ok] | 2026-10-15 | | RX 7900 XTX | bf16 | all | 16 | 0.0 | no | 7177 | 8327.1 | 10826.3 | 11818.0 | 14.7 | [ok] | 2026-10-15 | | RX 7900 XTX | bf16 | all | 64 | 0.0 | no | 7177 | 8702.3 | 11509.5 | 12290.0 | 13.3 | [ok] | 2026-10-15 | | RX 7900 XTX | bf16 | all | 112 | 0.0 | no | 7177 | 9089.1 | 12317.5 | 12658.0 | 13.43 | [ok] | 2026-10-15 | | RX 7900 XTX | bf16 | all | 256 | 0.0 | no | 7177 | 10244.5 | 14443.2 | 14840.0 | 14.38 | [ok] | 2026-10-15 | | RX 7900 XTX | 4bit | all | 112 | 0.0 | no | 2048 | 3974.4 | 5585.8 | 7288.0 | 4.46 | [ok] | 2026-10-15 | | RX 7900 XTX | 4bit | all | 112 | 0.0 | no | 4096 | 3974.4 | 6207.9 | 7286.0 | 7.33 | [ok] | 2026-10-15 | | RX 7900 XTX | 4bit | all | 112 | 0.0 | no | 8192 | 3974.5 | 7486.0 | 7852.0 | 15.91 | [ok] | 2026-10-15 | | RX 7900 XTX | 4bit | all | 112 | 0.0 | no | 12288 | 3974.6 | 8917.2 | 9498.0 | 30.49 | [ok] | 2026-10-15 | | RX 7900 XTX | 4bit | all | 112 | 0.0 | no | 16384 | 3974.7 | 10523.6 | 11162.0 | 53.5 | [ok] | 2026-10-15 | | RX 7900 XTX | 4bit | none | 16 | 0.0 | no | 7177 | 4502.8 | 35829.2 | 37502.0 | 68.68 | [spilled] Finished only because Windows spilled past the 24 GB card into shared system memory; about 4x slower. Would not fit on this card under Linux. See gotcha G31. | 2026-10-15 | | RX 7900 XTX | 4bit | none | 112 | 0.0 | no | 7177 | 5254.9 | 36581.3 | 38334.0 | 57.79 | [spilled] Finished only because Windows spilled past the 24 GB card into shared system memory; about 4x slower. Would not fit on this card under Linux. See gotcha G31. | 2026-10-15 | | RTX 3060 Ti | 4bit | all | 16 | 0.0 | no | 7177 | 3222.1 | 5788.9 | 6076.0 | 17.28 | [ok] | 2026-10-15 | | RTX 3060 Ti | 4bit | all | 64 | 0.0 | no | 7177 | 3597.3 | 6465.0 | 6826.0 | 17.26 | [ok] | 2026-10-15 | | RTX 3060 Ti | 4bit | all | 96 | 0.0 | no | 7177 | 3847.5 | 6757.3 | | | [oom] Ollama held about 0.55 GB of this 8 GB card during the probe | 2026-10-15 | Source: probe_kit/probe_vram.py and run_sweep.py (this site's probe kit) ## Calibration {#calibration} - **Method:** Run the one-step probe on the worst-case example, then scale by the probe's measured over-estimate on a config that also has a real run. ### Anchor {#calibration-anchor} - **Probe MB:** 6980.0 - **Probe:** the 2026-09-19-corpus probe at r112, no dropout - **Real MB:** 6653 - **Real:** v11/v13 r112 runs (6648/6653) - **Probe overestimate pct:** 4.7 ### Prediction {#calibration-prediction} - **Config:** r96, rsLoRA alpha 30, dropout 0.1 - **Probe MB:** 7127.6 - **Predicted MB:** 6793 - **Actual MB:** 6629 - **Error pct:** 2.5 - **Derivation note:** The chain script states the 4.7% over-estimate and the 6,793 MB prediction; it does not name the anchor probe. 6980 x (1 - 0.047) = 6652 and 7127.6 x 0.953 = 6793 match, so the anchor is taken to be the 2026-09-19-corpus r112 probe. - **Value:** replaced three crashes and about 11 GPU-hours that found the rank-112 dropout ceiling the hard way - **Measured on:** 2026-09-23 - **Status:** [inferred] ## Probe versus real gap {#probe-vs-real-gap} - **Observed:** The probe over-estimates a real run's peak by roughly 4.7% (anchor) and up to >0.9 GB in the worst case: on the 21 Sep corpus the r112 no-dropout probe OOMs at ~7.6 GB in use, while v13 trained that exact config at 6653 MB. - **What sources say:** The chain script calls the probe 'a conservative upper bound, not a prediction'. No source explains the mechanism. - **Probe design:** one forward+backward on the single longest example in the corpus (worst-case activation memory), no optimizer step, model.train() set so gradient checkpointing is active, wrapped in save_on_cpu like the trainer - **Length sensitivity:** probe peaks rise with the longest example and its response span: r112 fits at 6931 tokens (7055 MB) and 7068 tokens (6980 MB, short response) but OOMs at 7177 and 7288 tokens - **Measured on:** 2026-09-23 - **Status:** [inferred] ## Rank cost {#rank-cost} - **Real peak by rank MB:** 16: 4889-5117; 64: 5767-5796; 96 (+dropout): 6629; 112: 6643-6658 - **Probe cost per rank unit MB:** Value: 15; Evidence: dropout sweep: r80 6890.3, r88 7007.9, r96 7127.6 MB, i.e. ~118-120 MB per 8 ranks; measured on 2026-09-23 - **Real cost r64 to r112 MB:** Value: about 850-880; Evidence: v6 5796 vs v7 6647 (same corpus); v10 5767 vs v9 6643 (same corpus); measured on 2026-09-20 - **Measured on:** 2026-09-25 ## Dropout memory {#dropout-memory} - **Claim:** PEFT installs nn.Identity at lora_dropout=0 (allocates nothing) and nn.Dropout above 0, which saves a mask per call for backward across 7 target modules on 34 layers. Estimated cost about 350 MB of peak at rank 112, the whole remaining margin. The rate does not matter; the mask is allocated at any rate above 0. - **Estimate MB:** 350 - **Note:** 350 MB is the program notes's estimate from comparing crashes against v13's 6653 MB peak, not a direct measurement. It corrects an earlier ledger entry that said dropout cost no extra VRAM. - **Measured on:** 2026-09-23 - **Measured update:** Measured on 2026-10-15 at the one-step probe: dropout 0.1 adds 167 to 175 MB at ranks 96 to 256 (see Cross-GPU probes and finding F30), about half this estimate. - **Status:** [inferred] ## Inference footprint {#inference-footprint} | Model | GGUF size GB | Resident GB estimate | Note | Measured on | | --- | --- | --- | --- | --- | | fine-tuned router, Q8_0 | 4.14 | 4.4 | | 2026-09-25 | | fine-tuned router, Q4_0 | 2.37 | | | 2026-09-16 | | | | | untuned model resident sizes are in baselines.json | | ## Host RAM {#host-ram} - **Incident:** The first rank probe lacked model.train(), so HF gradient checkpointing was silently off and save_on_cpu spilled every activation to pinned host RAM. It thrashed the box and later hit a 32 GB cgroup memory cap. Probes now run under systemd-run --scope -p MemoryMax=12G. - **Measured on:** 2026-09-17 --- # Inference speed > Ollama load time, prompt speed, generation speed and resident VRAM for seven small models on an AMD RX 7900 XTX. Canonical: https://marvinlabs.org/inference Data: [inference.json](https://marvinlabs.org/data/inference.json) HTML: https://marvinlabs.org/inference Ollama inference speed for seven small models on one AMD Radeon RX 7900 XTX (24 GB, Windows 11, Ollama 0.30.10). Each model was unloaded first, then given synthetic prompts of about 1,550 and 4,550 tokens, 3 trials each, with num_ctx 9216, temperature 0 and 200 output tokens. Each trial changes the first line, so no prompt prefix is reused from cache. Cold load is the first call's load time. The speed medians leave out that first cold trial, because its prompt speed includes warm-up. Every model was fully resident in VRAM (size_vram equals size), and no prompt was truncated. In the summary, speeds are median tokens per second; short means a prompt of about 1,550 tokens and long about 4,550. Gen speed is output generation. ## GPU {#gpu} - **Model:** AMD Radeon RX 7900 XTX - **Arch:** gfx1100 - **Total MB:** 24560 - **OS:** Windows 11 (build 26200) - **Driver:** AMD Adrenalin 26.7.1 (32.0.31035.1003) - **Ollama:** 0.30.10 ## Summary {#summary} | Model | Resident MB | Cold load s | Prompt speed short | Prompt speed long | Gen speed short | Gen speed long | Measured on | | --- | --- | --- | --- | --- | --- | --- | --- | | gemma3:4b | 2891 | 7.74 | 4411.4 | 5039.9 | 132.6 | 128.6 | 2026-10-15 | | gemma4:e2b-it-qat | 1762 | 6.17 | 6982.2 | 7458.3 | 154.6 | 152.2 | 2026-10-15 | | granite4.2:3b-q8_0 | 4313 | 4.84 | 5707.4 | 5219.5 | 127.0 | 121.3 | 2026-10-15 | | gemma4:e4b-it-qat | 3133 | 7.69 | 3941.8 | 4258.9 | 116.5 | 113.5 | 2026-10-15 | | qwen3.5:9b | 5642 | 8.82 | 2588.9 | 2704.7 | 84.9 | 84.2 | 2026-10-15 | | gemma4:12b-it-qat | 7601 | 9.46 | 1526.1 | 1661.9 | 68.4 | 66.4 | 2026-10-15 | | gpt-oss:20b | 12261 | 14.69 | 3730.2 | 4118.1 | 123.0 | 116.2 | 2026-10-15 | ## Trials {#trials} | Model | Prompt words | Trial | Prompt tokens | Load s | Prompt s | Prompt tok per s | Gen tokens | Gen s | Gen tok per s | Total s | Size VRAM MB | Context length | Measured on | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | gemma3:4b | 1500 | 0 | 1551 | 7.74 | 0.532 | 2915.4 | 200 | 1.659 | 120.6 | 9.94 | 2891 | 9216 | 2026-10-15 | | gemma3:4b | 1500 | 1 | 1551 | 0.35 | 0.364 | 4261.0 | 200 | 1.511 | 132.4 | 2.45 | 2891 | 9216 | 2026-10-15 | | gemma3:4b | 1500 | 2 | 1551 | 0.32 | 0.34 | 4561.8 | 200 | 1.506 | 132.8 | 2.28 | 2891 | 9216 | 2026-10-15 | | gemma3:4b | 4500 | 0 | 4551 | 0.32 | 0.893 | 5096.3 | 200 | 1.547 | 129.3 | 2.89 | 2891 | 9216 | 2026-10-15 | | gemma3:4b | 4500 | 1 | 4551 | 0.33 | 0.93 | 4893.5 | 200 | 1.555 | 128.6 | 2.99 | 2891 | 9216 | 2026-10-15 | | gemma3:4b | 4500 | 2 | 4551 | 0.32 | 0.903 | 5039.9 | 200 | 1.571 | 127.3 | 2.97 | 2891 | 9216 | 2026-10-15 | | gemma4:e2b-it-qat | 1500 | 0 | 1557 | 6.17 | 0.336 | 4633.9 | 200 | 1.298 | 154.1 | 7.81 | 1762 | 9216 | 2026-10-15 | | gemma4:e2b-it-qat | 1500 | 1 | 1557 | 0.32 | 0.224 | 6950.9 | 200 | 1.29 | 155.0 | 1.86 | 1762 | 9216 | 2026-10-15 | | gemma4:e2b-it-qat | 1500 | 2 | 1557 | 0.32 | 0.222 | 7013.5 | 200 | 1.298 | 154.1 | 1.86 | 1762 | 9216 | 2026-10-15 | | gemma4:e2b-it-qat | 4500 | 0 | 4557 | 0.32 | 0.609 | 7482.8 | 200 | 1.314 | 152.2 | 2.29 | 1762 | 9216 | 2026-10-15 | | gemma4:e2b-it-qat | 4500 | 1 | 4557 | 0.32 | 0.611 | 7458.3 | 200 | 1.308 | 152.9 | 2.29 | 1762 | 9216 | 2026-10-15 | | gemma4:e2b-it-qat | 4500 | 2 | 4557 | 0.31 | 0.612 | 7446.1 | 200 | 1.316 | 152.0 | 2.3 | 1762 | 9216 | 2026-10-15 | | granite4.2:3b-q8_0 | 1500 | 0 | 1541 | 4.84 | 0.294 | 5241.5 | 200 | 1.564 | 127.9 | 6.71 | 4313 | 9216 | 2026-10-15 | | granite4.2:3b-q8_0 | 1500 | 1 | 1541 | 0.09 | 0.27 | 5707.4 | 200 | 1.567 | 127.6 | 2.0 | 4313 | 9216 | 2026-10-15 | | granite4.2:3b-q8_0 | 1500 | 2 | 1541 | 0.08 | 0.27 | 5707.4 | 200 | 1.582 | 126.4 | 2.0 | 4313 | 9216 | 2026-10-15 | | granite4.2:3b-q8_0 | 4500 | 0 | 4541 | 0.08 | 0.868 | 5231.6 | 200 | 1.65 | 121.2 | 2.68 | 4313 | 9216 | 2026-10-15 | | granite4.2:3b-q8_0 | 4500 | 1 | 4541 | 0.09 | 0.87 | 5219.5 | 200 | 1.649 | 121.3 | 2.78 | 4313 | 9216 | 2026-10-15 | | granite4.2:3b-q8_0 | 4500 | 2 | 4541 | 0.08 | 0.878 | 5172.0 | 200 | 1.649 | 121.3 | 2.79 | 4313 | 9216 | 2026-10-15 | | gemma4:e4b-it-qat | 1500 | 0 | 1557 | 7.69 | 0.496 | 3139.1 | 200 | 1.732 | 115.5 | 9.93 | 3133 | 9216 | 2026-10-15 | | gemma4:e4b-it-qat | 1500 | 1 | 1557 | 0.31 | 0.395 | 3941.8 | 200 | 1.718 | 116.4 | 2.48 | 3133 | 9216 | 2026-10-15 | | gemma4:e4b-it-qat | 1500 | 2 | 1557 | 0.31 | 0.395 | 3941.8 | 200 | 1.716 | 116.6 | 2.47 | 3133 | 9216 | 2026-10-15 | | gemma4:e4b-it-qat | 4500 | 0 | 4557 | 0.31 | 1.065 | 4278.9 | 200 | 1.778 | 112.5 | 3.23 | 3133 | 9216 | 2026-10-15 | | gemma4:e4b-it-qat | 4500 | 1 | 4557 | 0.31 | 1.07 | 4258.9 | 200 | 1.762 | 113.5 | 3.23 | 3133 | 9216 | 2026-10-15 | | gemma4:e4b-it-qat | 4500 | 2 | 4557 | 0.32 | 1.071 | 4254.9 | 200 | 1.762 | 113.5 | 3.25 | 3133 | 9216 | 2026-10-15 | | qwen3.5:9b | 1500 | 0 | 1552 | 8.82 | 0.68 | 2282.4 | 200 | 2.383 | 83.9 | 11.89 | 5642 | 9216 | 2026-10-15 | | qwen3.5:9b | 1500 | 1 | 1552 | 0.26 | 0.597 | 2599.7 | 200 | 2.359 | 84.8 | 3.3 | 5642 | 9216 | 2026-10-15 | | qwen3.5:9b | 1500 | 2 | 1552 | 0.23 | 0.602 | 2578.1 | 200 | 2.35 | 85.1 | 3.25 | 5642 | 9216 | 2026-10-15 | | qwen3.5:9b | 4500 | 0 | 4552 | 0.23 | 1.673 | 2720.9 | 200 | 2.375 | 84.2 | 4.37 | 5642 | 9216 | 2026-10-15 | | qwen3.5:9b | 4500 | 1 | 4552 | 0.24 | 1.683 | 2704.7 | 200 | 2.378 | 84.1 | 4.41 | 5642 | 9216 | 2026-10-15 | | qwen3.5:9b | 4500 | 2 | 4552 | 0.24 | 1.683 | 2704.7 | 200 | 2.374 | 84.2 | 4.41 | 5642 | 9216 | 2026-10-15 | | gemma4:12b-it-qat | 1500 | 0 | 1557 | 9.46 | 0.972 | 1601.9 | 200 | 2.851 | 70.2 | 13.29 | 7601 | 9216 | 2026-10-15 | | gemma4:12b-it-qat | 1500 | 1 | 1557 | 0.33 | 1.037 | 1501.4 | 200 | 2.851 | 70.2 | 4.5 | 7601 | 9216 | 2026-10-15 | | gemma4:12b-it-qat | 1500 | 2 | 1557 | 0.32 | 1.004 | 1550.8 | 200 | 3.003 | 66.6 | 4.62 | 7601 | 9216 | 2026-10-15 | | gemma4:12b-it-qat | 4500 | 0 | 4557 | 0.45 | 2.839 | 1605.1 | 200 | 2.964 | 67.5 | 6.68 | 7601 | 9216 | 2026-10-15 | | gemma4:12b-it-qat | 4500 | 1 | 4557 | 0.34 | 2.742 | 1661.9 | 200 | 3.119 | 64.1 | 6.57 | 7601 | 9216 | 2026-10-15 | | gemma4:12b-it-qat | 4500 | 2 | 4557 | 0.33 | 2.74 | 1663.1 | 200 | 3.01 | 66.4 | 6.47 | 7601 | 9216 | 2026-10-15 | | gpt-oss:20b | 1500 | 0 | 1604 | 14.69 | 0.81 | 1980.2 | 200 | 1.617 | 123.7 | 17.13 | 12261 | 9216 | 2026-10-15 | | gpt-oss:20b | 1500 | 1 | 1604 | 0.28 | 0.429 | 3738.9 | 200 | 1.623 | 123.2 | 2.37 | 12261 | 9216 | 2026-10-15 | | gpt-oss:20b | 1500 | 2 | 1604 | 0.27 | 0.431 | 3721.6 | 200 | 1.63 | 122.7 | 2.38 | 12261 | 9216 | 2026-10-15 | | gpt-oss:20b | 4500 | 0 | 4604 | 0.28 | 1.121 | 4107.0 | 200 | 1.721 | 116.2 | 3.19 | 12261 | 9216 | 2026-10-15 | | gpt-oss:20b | 4500 | 1 | 4604 | 0.28 | 1.117 | 4121.8 | 200 | 1.732 | 115.5 | 3.22 | 12261 | 9216 | 2026-10-15 | | gpt-oss:20b | 4500 | 2 | 4604 | 0.28 | 1.118 | 4118.1 | 200 | 1.715 | 116.6 | 3.21 | 12261 | 9216 | 2026-10-15 | ## Measured on {#measured-on} 2026-10-15 --- # Untuned baselines > The untuned model tournament and the untuned 12B scored on each test-set era. Canonical: https://marvinlabs.org/baselines Data: [baselines.json](https://marvinlabs.org/data/baselines.json) HTML: https://marvinlabs.org/baselines Untuned (off-the-shelf) models, measured before and during the fine-tune program through Ollama. 'routing' = the production command-routing prompt (~5.6k tokens); 'content' = 11 generation workloads (summaries, tagging, narration and similar) scored by a judge model. 'floor' tier = unambiguous cases every acceptable model must pass; 'hard' tier = deliberately ambiguous cases. Tournament numbers below were recomputed from the raw per-cycle score records by time window and checked against the program's written notes where they quote them. ## Untuned model tournament {#tournament} Counts are passed/total. Blended is the pass fraction across floor and hard tiers. Median seconds per case is for floor and hard tiers. | Model | Role and notes | Routing floor | Routing hard | Routing blended | Median s/case (floor, hard) | Content floor | Content hard | Content blended | Floor block | Hard block | Resident VRAM (GB) | Measured on | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | `gemma4:12b-it-qat` | production router before the fine-tune; baseline for every version | 150/150 | 130/150 | 0.933 | 3.0, 4.5 | 490/490 | 323/330 | 0.991 | Routing: 30/30; Content: 123/123 | Routing: 34/45; Content: 80/82 | Ledger GB: 8.8; Soak scores max size VRAM GB: 8.5; Soak scores typical GB: 4.76 | 2026-09-09 | | `gemma4:e2b-it-qat` | first-choice fine-tune base; abandoned for tooling reasons (see gotchas) | 150/150 | 120/150 | 0.9 | 1.0, 1.2 | 476/490 | 315/330 | 0.965 | unavailable (load errors in that block) | Routing: 35/45; Content: 77/82 | Soak scores max size VRAM GB: 1.79 | 2026-09-13 | | `granite4.2:3b-q8_0` | hard-tier mean 116.2 s vs median 27.6 s per workload record under contention; one record took 1357.88 s | 140/150 | 129/150 | 0.897 | 0.7, 0.9 | 488/490 | 306/330 | 0.968 | Routing: 28/30; Content: 120/123 | Routing: 35/45; Content: 78/82 | Ledger GB: 4.7; Soak scores max size VRAM GB: 4.44 | 2026-09-13 | | `qwen3.5:9b` | the original fine-tune plan, dropped; hybrid SSM/attention (qwen35.ssm.* fields in ollama show), not a plain dense model | 140/150 | 109/150 | 0.83 | 1.4, 1.5 | 476/480 | 304/330 | 0.963 | | Routing: 34/45; Content: 76/82 | Ledger GB: 5.8; Soak scores max size VRAM GB: 6.32 | 2026-09-09 | | `qwen3.5:4b-q8_0` | not in the program notes's or ledger's comparison table; numbers are from the raw records only | 150/150 | 120/150 | 0.9 | 1.2, 1.6 | 489/490 | 299/330 | 0.961 | | Routing: 36/45; Content: 77/82 | Soak scores max size VRAM GB: 5.31 | 2026-09-09 | | `nemotron-3-nano:4b-q8_0` | floor tier only; dropped after floor1 | | | | | | | | Routing: 26/30; Content: 122/123 | | Soak scores max size VRAM GB: 4.46 | 2026-09-09 | | `qwen3.5:4b-q4_K_M, qwen3.5:9b-q8_0, LFM2.5-2.6B (Q8_0, Q4_K_M)` | | | | | | | | | | | | 2026-09-08 | ### Details: qwen3.5:4b-q4_K_M, qwen3.5:9b-q8_0, LFM2.5-2.6B (Q8_0, Q4_K_M) {#details-qwen3-5-4b-q4-k-m-qwen3-5-9b-q8-0-lfm2-5-2-6b-q8-0-q4-k-m} #### Pre0908 summary {#details-qwen3-5-4b-q4-k-m-qwen3-5-9b-q8-0-lfm2-5-2-6b-q8-0-q4-k-m-pre0908-summary} | Model | Pass | S per case | VRAM GB | Routing workload | | --- | --- | --- | --- | --- | | qwen3.5:4b-q8_0 | 778/781 | 8.7 | 5.1 | | | qwen3.5:4b-q4_K_M | 774/781 | 10.9 | 2.2 | | | qwen3.5:9b | 768/781 | 2.9 | 5.8 | | | qwen3.5:9b-q8_0 | 760/778 | 6.3 | 9.1 | | | LFM2.5-2.6B Q8_0 | 621/703 | 12.4 | 3.0 | 0.68 | | LFM2.5-2.6B Q4_K_M | 607/703 | 6.5 | 1.8 | 0.58 | #### Pre0908 hard summary {#details-qwen3-5-4b-q4-k-m-qwen3-5-9b-q8-0-lfm2-5-2-6b-q8-0-q4-k-m-pre0908-hard-summary} | Model | Pass | S per case | | --- | --- | --- | | qwen3.5:4b-q8_0 | 530/607 | 9.9 | | qwen3.5:9b | 520/607 | 10.0 | - **Q8 vs q4 note:** qwen3.5:9b vs its q8_0: same 98% pass rate, q8 2.2x slower (6.3 vs 2.9 s/case) and 3.3 GB more VRAM - **Caveat:** Earlier, easier case set, all 12 workloads pooled; not comparable to the tiered blocks. LFM2 has no context-shift support and surfaced the num_ctx overflow as HTTP 400 errors (see gotchas), so its routing scores from this window are suspect. - **Measured on:** 2026-09-08 ### Tournament blocks {#tournament-blocks} Each block is a time window of the soak run that produced the scores above. | Block | Window | Note | | --- | --- | --- | | pre-0908 (12 workloads, no tier split) | 2026-09-07T22:48Z to 2026-09-08T19:32Z | | | pre-0908 hard | 2026-09-08T21:06Z to 2026-09-09T01:41Z | | | floor1 | 2026-09-09T03:24Z to 04:21Z | | | hard1 | 2026-09-09T04:23Z to 07:20Z | | | route1 | 2026-09-09T07:23Z to 15:38Z | superseded by route1b; heavy contention (median s/case 10-38) | | route1b | 2026-09-09T15:38Z to 16:29Z | the routing result the program notes and ledger quote | | content1 | 2026-09-09T17:07Z to 2026-09-10T04:14Z | | | e2bgranite_content1 | 2026-09-13T03:18Z to 10:38Z | filled the content gap for e2b and granite; ran under contention from the live router | ## The untuned 12B on each test-set era {#untuned-12b-on-gold-sets} | Test set | Passed | Accuracy | Wall clock s | S per case | Note | Held out passed | Held out total | Held out accuracy | flags | Measured on | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | 550 cases | 469 | 0.8527 | 1012.9 | | | | | | | 2026-09-14 | | 865 cases | 691 | 0.7988 | | 2.124 | | | | | | 2026-09-17 | | 865, train-flagged slice | | 0.76 | | | Ledger-only. Recomputing with today's train flags gives 79.5% (667/839) because flags changed after 09-18. | | | | [unverified] | 2026-09-18 | | 865, held-out slice | | 0.941 | | | Ledger-only. With today's train flags the held-out slice is 26 cases and the 12B scores 24/26 (92.3%). | | | | [unverified] | 2026-09-18 | | 1,125 runnable, harness fixed | 924 | 0.8213 | | 2.685 | | 96 | 108 | 0.8889 | | 2026-09-19 | ## Comparison to the fine-tunes {#comparison-to-finetunes} - **Statement:** On the 1,125-case set the rank-112 fine-tunes route about 2x faster than the untuned 12B (1.22-1.34 vs 2.69 s/case) and are 11-14 pp more accurate overall; v9 was the first to beat it on held-out (91.7% vs 88.9%). - **Measured on:** 2026-09-21 ## Warm head-to-head {#warm-head-to-head} - **Method:** evict, cold-load, then repeated warm calls through the production routing code path, under low system load ### Results {#warm-head-to-head-results} | Model | Cold load s | Warm route s | Warm 200 token generation s | | --- | --- | --- | --- | | qwen3.5:9b | 8.8 | Median: 6.03; Min: 6.03; Max: 6.79 | 6.73 | | gemma4:e2b-it-qat | 5.3 | Median: 6.03; Min: 6.03; Max: 6.03 | 3.04 | | granite4.2:3b-q8_0 | 2.9 | Median: 6.03; Min: 6.02; Max: 6.03 | 4.08 | - **Note:** warm-route times converge on ~6.03 s for all three with no speed-up across same-prefix calls, which the program notes reads as the static prompt prefix not being cached call to call. The 200-token generation column is the clean comparison. - **Measured on:** 2026-09-13 ## Sliding-window cache reuse test {#swa-cache-reuse-test} - **Method:** 6 trials per model, each with fresh ~300-char context through the real routing path; static ~5.6k tokens + dynamic tail = 6.2-6.5k tokens ### Results {#swa-cache-reuse-test-results} | Model | Sliding window | Trial0 s | Trials1 5 s | Median s | Fast plateau | | --- | --- | --- | --- | --- | --- | | gemma4:12b-it-qat | 1024 | 269.96 | 49-56 | 53.8 | no | | gemma4:e2b-it-qat | 512 | 60.74 | 11-12 | 11.76 | yes | | granite4.2:3b-q8_0 | | 142.14 | 23-33 | 29.2 | no | | qwen3.5:9b | | 142.79 | 24-36 | 26.0 | no | - **Confound:** system load rose from ~3.0 to ~6-8 between the gemma runs and the granite/qwen runs; a clean test would run all four back to back - **Measured on:** 2026-09-13 --- # Findings > Every finding with its evidence and measured date, plus the methods tried and closed and the ideas left untried. Canonical: https://marvinlabs.org/findings Data: [findings.json](https://marvinlabs.org/data/findings.json) HTML: https://marvinlabs.org/findings Lessons from 17+ LoRA fine-tunes of gemma-3-4b-it as a command router on one 8 GB consumer GPU. Each claim is limited to this task, this base model and this recipe. Accuracy numbers are passed / runnable cases; see versions.json for definitions. ## Findings {#findings} - [F01: LoRA rank, not more data, moved the hardest capability](#F01) - [F02: Rank 112 is the ceiling on an 8 GB card at ~7k-token examples](#F02) - [F03: Incremental patching on a trained adapter caused catastrophic forgetting](#F03) - [F04: Held-out eval loss does not pick the best checkpoint](#F04) - [F05: Accuracy plateaus well before the end, and the final checkpoint is as good as any](#F05) - [F06: A third epoch is worth about one point](#F06) - [F07: rsLoRA raised the headline and eroded the safety guards](#F07) - [F08: LoRA dropout did not fit at rank 112 and did not improve generalization where it fit](#F08) - [F09: Checkpoint averaging is noise on this recipe](#F09) - [F10: Data quantity beat data purity: a 25% disjoint holdout cost 8.5 pp](#F10) - [F11: The test set was mostly training data, but the plain recipe still generalized](#F11) - [F12: Negative guards need training targets](#F12) - [F13: Guard batches need counterweights or they teach over-refusal](#F13) - [F14: Label conflicts cost more than missing data](#F14) - [F15: A single bake pass over a small set overstated a margin by about 2x](#F15) - [F16: Run-to-run noise differs by model; repeat bakes before comparing](#F16) - [F17: The overall number can rank the worst router first; a guard veto is needed](#F17) - [F18: A harness channel bug scored every context case context-free for weeks](#F18) - [F19: Grammar-constrained decoding helps only when the enum matches what the router accepts](#F19) - [F20: One-line capability docs earn their prompt tokens](#F20) - [F21: Q8_0 beat Q4_0 by 3.2 pp on the fine-tuned router](#F21) - [F22: Inference speed is set by model size, not training choices](#F22) - [F23: A 4B fine-tune beat the untuned 12B it replaced](#F23) - [F24: Measure the memory ceiling with a calibrated probe instead of crashing into it](#F24) - [F25: Deterministic pre-router steps remove whole miss classes, invisibly to the bake](#F25) - [F26: Small models need the context window set explicitly](#F26) - [F27: New capabilities often need little or no training](#F27) - [F28: Fine-tunes trade cautious misses for confident ones](#F28) - [F29: AMD and NVIDIA need the same memory for the same QLoRA step](#F29) - [F30: LoRA dropout costs about 170 MB at the probe, not 350](#F30) - [F31: Rank cost is linear; DoRA and bf16 add large fixed costs](#F31) - [F32: A 20B mixture-of-experts model generates faster than dense 9B and 12B models](#F32) ## F01: LoRA rank, not more data, moved the hardest capability {#F01} Three rounds of more data (v3, v4, v5) left the clarify capability at 56-60%. Raising rank from 16 to 64 on the same 757 examples lifted overall accuracy 4 pp and clarify to ~75%; rank 112 lifted clarify to ~85%. Dropping back to rank 64 on a later corpus cost clarify 13 pp. - **Evidence:** Overall r16 vs r64: 0.8902, 0.9306; Clarify by rank 865set: R16: 0.521, R64: 0.755, R112: 0.851; V10 r64 vs v9 r112 clarify: 0.73, 0.86; V10 vs v9 overall 1125set: 0.9164, 0.9307 - **Measured on:** 2026-09-20 Cite: https://marvinlabs.org/findings#F01 ## F02: Rank 112 is the ceiling on an 8 GB card at ~7k-token examples {#F02} With chunked MLP, save_on_cpu and a bf16 lm_head, rank 112 peaks at ~6.65 GB in real runs; rank 128 and DoRA (at 64 and 112) OOM on the longest example. Rank 112 over 64 bought +0.5 pp on the 865 set, inside run-to-run noise there, but held clarify gains later. - **Evidence:** Real peak r112 MB: 6643-6658; Probe r128: OOM; Probe DoRA r64: OOM; Probe DoRA r112: OOM; V7 vs v6: 0.9434, 0.9387 - **Measured on:** 2026-09-18 Cite: https://marvinlabs.org/findings#F02 ## F03: Incremental patching on a trained adapter caused catastrophic forgetting {#F03} Continuing v3's adapter for 8 epochs on only 59 targeted examples dropped accuracy 7.3 pp on v3's own test set. Two targeted capabilities improved; the main target (clarify) got worse; four untouched capabilities collapsed to 25-56%. Mixing the same 59 examples into the full corpus and retraining from base (v5) recovered every collapsed capability and matched v3 exactly. - **Evidence:** V3: 709/790; V4: 651/790; V5: 709/790; V4 collapsed: Run agent: 0.25, Resume agent: 0.417, Dispatch code: 0.524, Calendar availability: 0.562; V5 same capabilities: 1.0 - **Measured on:** 2026-09-15 Cite: https://marvinlabs.org/findings#F03 ## F04: Held-out eval loss does not pick the best checkpoint {#F04} In every run with an eval set, eval loss bottomed early (step 75-125 of ~440 in most runs) and drifted up, while accuracy kept improving or held. The lowest-loss v9 checkpoint (step 175) scored 6.5 pp worse overall than the final. v12 (2 epochs) had the lowest final eval loss of the program and lower accuracy than v11, whose final eval loss was among the highest. v15 showed the 'better' loss curve and lost on accuracy. The eval targets carry partial arguments, which the program notes suggests rewards under-confident outputs. #### Evidence {#f04-evidence} - **V9 step175:** Eval loss: 0.1306; Accuracy: 0.8658 - **V9 final:** Eval loss: 0.1861; Accuracy: 0.9305 ##### Final eval loss {#f04-evidence-final-eval-loss} - **V11:** 0.3997 - **V12:** 0.2934 - **V13:** 0.3497 - **V15:** 0.3385 - **V17:** 0.4378 ##### Best eval step {#f04-evidence-best-eval-step} - **V9:** 175 - **V10:** 125 - **V11:** 75 - **V12:** 125 - **V13:** 100 - **V15:** 300 - **V16:** 100 - **V17:** 100 - **Accuracy 1131set:** V11: 0.9637; V12: 0.9558 - **Status:** [inferred] - **Measured on:** 2026-09-23 Cite: https://marvinlabs.org/findings#F04 ## F05: Accuracy plateaus well before the end, and the final checkpoint is as good as any {#F05} A held-out accuracy sweep of all 16 v9 checkpoints showed accuracy flat from step 200 of 405, with no mid-run peak worth picking. #### Evidence {#f05-evidence} ##### Held out 108 by step {#f05-evidence-held-out-108-by-step} - **25:** 86 - **50:** 94 - **75:** 96 - **100:** 86 - **125:** 95 - **150:** 94 - **175:** 96 - **200:** 100 - **225:** 99 - **250:** 100 - **275:** 97 - **300:** 99 - **325:** 99 - **350:** 100 - **375:** 100 - **400:** 99 - **405:** 99 - **Conflict:** The program notes say the plateau is '91.7-92.6% from step 200'; the sweep summary has step 275 at 97/108 (89.8%). The plateau claim holds loosely; the stated range is slightly too tight. Trust the summary. - **Measured on:** 2026-09-19 Cite: https://marvinlabs.org/findings#F05 ## F06: A third epoch is worth about one point {#F06} Same data and recipe at 2 epochs (v12) lost 0.8 pp overall and 4 held-out cases versus 3 epochs (v11), mostly on clarify. It saved about 4 hours of training, which buys nothing at inference. - **Evidence:** V11: Passed: 1090, Held out: 108/110; V12: Passed: 1081, Held out: 104/110; Runtime h: V11: 11.8, V12: 7.9 - **Measured on:** 2026-09-21 Cite: https://marvinlabs.org/findings#F06 ## F07: rsLoRA raised the headline and eroded the safety guards {#F07} rsLoRA at alpha = rank (scale 10.6) flatlined and was killed. At alpha 32 (scale 3.0) it posted the program's best overall accuracy, but every gained case was train-flagged; it lost 5 held-out cases and failed held-out guards the plain recipe passes, mostly cases with pasted or quoted content. Its checkpoint average did not recover them. - **Evidence:** V13 overall: 0.9752; V11 overall: 0.9637; V13 held out: 103/110; V11 held out: 108/110; V13 train flagged: 0.9794; Guards 174: V11: 169, 1 held-out breach, V13: 168, 3 held-out breaches (2 pasted/quoted content); A112 eval loss step50: 0.577; V11 eval loss step50: 0.33 - **Measured on:** 2026-09-22 Cite: https://marvinlabs.org/findings#F07 ## F08: LoRA dropout did not fit at rank 112 and did not improve generalization where it fit {#F08} Dropout 0.1 on top of v13 died three times at rank 112 (one Xid 31 fault, two OOMs). At equal steps (step 225) dropout raised overall accuracy 3.2 pp, all on train-flagged cases, and widened the train-to-held-out gap from 1.2 to 6.8 pp; it did improve guards (148 to 157 of 174). At rank 96 it trained clean but bought one held-out case over v13, cost 0.7 pp, and was vetoed on guards. - **Evidence:** Ck225 nodrop: Overall: 0.8382, Train: 0.8394, Held out: 91/110, Guards: 148/174, 10 held-out breaches; Ck225 drop10: Overall: 0.87, Train: 0.8766, Held out: 89/110, Guards: 157/174, 7 held-out breaches; V15: Overall: 0.9682, Held out: 104/110, Guards: 165/174, 4 held-out breaches, 3 pasted/quoted content - **Measured on:** 2026-09-23 Cite: https://marvinlabs.org/findings#F08 ## F09: Checkpoint averaging is noise on this recipe {#F09} An exact mean of v11's last seven checkpoints scored +0.4 pp on one bake and +0.0 on a re-bake, with identical held-out both times. The rsLoRA average scored 0.4 pp below its final checkpoint. - **Evidence:** V11 avg: 0.9691, 0.9637; V11: 0.9646, 0.9637; V13 avg: 0.9717; V13: 0.9761 - **Method note:** averaging done exactly by concatenating LoRA A/B along rank (r = n x r0, alpha fixed), with an extra 1/sqrt(n) on B for rsLoRA - **Measured on:** 2026-09-22 Cite: https://marvinlabs.org/findings#F09 ## F10: Data quantity beat data purity: a 25% disjoint holdout cost 8.5 pp {#F10} Holding a quarter of the gold set out of training (v16, 873 vs 1,167 examples, recipe unchanged) lost 8.5 pp on 82 cases neither model saw. The same experiment showed the plain recipe was not inflated by memorization: v11 scored 96.3% on those never-seen cases against a 96.4% headline. Recommendation recorded: hold back ~10%, not 25%. - **Evidence:** Fair 82: V11: 0.963, V16: 0.878; V11 trained v16 not 317: V11: 0.946, V16: 0.826; V16 overall 1148: 0.9016; V11 overall 1148: 0.9591 - **Measured on:** 2026-09-24 Cite: https://marvinlabs.org/findings#F10 ## F11: The test set was mostly training data, but the plain recipe still generalized {#F11} Through v7, ~79% of the 865-case test set was also in the training corpus and the held-out slice was almost all must-not-fire guards with no training target; the untuned 12B beat every fine-tune on that slice until v9. Once guards got targets (v8/v9) and the harness was fixed, fine-tunes passed the 12B on held-out. The later disjoint test (F10) then showed the plain recipe was not a memorization score. - **Evidence:** V7 865: Train flagged: 0.963, Held out: 0.871; Untuned 12b 865 held out: 0.941; V9 vs 12b held out 1125: 0.9167, 0.8889 - **Unverified parts:** the 79% overlap and the v7/12B slice numbers are from the ledger and could not be recomputed (train flags changed after 09-18) - **Status:** [unverified] - **Measured on:** 2026-09-19 Cite: https://marvinlabs.org/findings#F11 ## F12: Negative guards need training targets {#F12} Must-not-fire cases had no gold label, so nothing taught them. Authoring targets for 126 of them lifted the guard families 19-50 pp in v8, and they held in v9. A later 86-case adversarial guard batch cut held-out breaches in v17 relative to v11 over repeated passes. - **Evidence:** Guard family gain pp v8: 19-50; Not run pause guard: 0.67, 0.92, 0.96; Guards 267 three passes: V11: 250, V17: 256 - **Measured on:** 2026-09-25 Cite: https://marvinlabs.org/findings#F12 ## F13: Guard batches need counterweights or they teach over-refusal {#F13} On 127 never-seen cases the weakest group for both models was counterweights: quote something, then genuinely issue a command. The program notes concludes the real residual weakness is over-refusal, not over-action, and that a guard batch shipped without counterweights would have pushed further that way. - **Evidence:** Counterweights 24: V11: 0.833, V16: 0.667; Adversarial guards 62: V11: 0.855, V16: 0.903 - **Measured on:** 2026-09-24 Cite: https://marvinlabs.org/findings#F13 ## F14: Label conflicts cost more than missing data {#F14} Four new clarify labels on bare confirmation words contradicted older gold labels and dropped gate_approve from 100% to 70% in v8. Removing them restored it to 94% in v9 with no other change to that capability. - **Evidence:** Gate approve: 1.0, 0.7, 0.94 - **Measured on:** 2026-09-19 Cite: https://marvinlabs.org/findings#F14 ## F15: A single bake pass over a small set overstated a margin by about 2x {#F15} The one-pass guard bake that justified switching production to v17 scored it 257/267 with 1 held-out breach against v11's 247 with 4. Three fresh passes each, same conditions, gave v17 256 with 4 breaches and v11 250 with 5, identical every pass. The real margin is 6 cases, not 10, and both models still miss some pasted-text cases, including one the single pass had counted as a v17 pass. - **Evidence:** Single pass: V17: 257, 1 breach, V11: 247, 4 breaches; Three passes: V17: 256, 4 breaches (x3), V11: 250, 5 breaches (x3) - **Open question:** Why the single pass differed from four identical v11 passes (247 vs 250) at temperature 0 is not explained in the sources. - **Measured on:** 2026-09-25 Cite: https://marvinlabs.org/findings#F15 ## F16: Run-to-run noise differs by model; repeat bakes before comparing {#F16} Three all-caps passes: v11 landed within one case of itself (spread 0.09 pp); v9 (marvin-v1.2) swung 14 cases (1.18 pp). Sub-point deltas against a noisy model are meaningless without repeats. Per-capability slices with n of about 8 need intervals. - **Evidence:** V11 passes: 1145, 1144, 1145; V9 passes: 1098, 1112, 1112; Total: 1191; Example interval: Guard:open folder: Cases: 8, Accuracy: 0.875, Ci: 0.785, 0.931 - **Measured on:** 2026-09-22 Cite: https://marvinlabs.org/findings#F16 ## F17: The overall number can rank the worst router first; a guard veto is needed {#F17} v13 (97.6%) and v15 (96.8%) both beat v11 (96.4%) overall and both obey instructions hidden in pasted text more often. The 174-case guard bake (about 4 minutes) was the only check that separated them. The release rule was changed to: veto any held-out guard breach the incumbent does not have, then rank by held-out accuracy. - **Evidence:** Overall: V13: 0.9752, V15: 0.9682, V11: 0.9637; Held out guard breaches 174: V11: 1, V13: 3, V15: 4; Guard bake wall clock s: 220.6, 230.5 - **Measured on:** 2026-09-23 Cite: https://marvinlabs.org/findings#F17 ## F18: A harness channel bug scored every context case context-free for weeks {#F18} Bake scripts and both live-model test harnesses routed on one channel while each case's setup seeded context on another, so ~50 context-dependent cases were scored without context for every model. Fixing it moved v9's held-out from 86.8% to 91.7% on the same weights. - **Evidence:** V9 held out before fix: 0.8681; V9 held out after fix: 0.9167 - **Measured on:** 2026-09-19 Cite: https://marvinlabs.org/findings#F18 ## F19: Grammar-constrained decoding helps only when the enum matches what the router accepts {#F19} Forcing the JSON schema and capability enum at decode time first cost 0.7 pp: the enum was built from the visible capability docs and omitted one hidden capability, so all 8 of its cases were forced to clarify. With every accepted capability in the enum it was +0.1 pp and cut unparseable outputs from 3 to 1. - **Evidence:** First: 0.9573; Plain: 0.9637; Fixed: 0.9655; Unparseable: 3, 1 - **Measured on:** 2026-09-21 Cite: https://marvinlabs.org/findings#F19 ## F20: One-line capability docs earn their prompt tokens {#F20} Stripping the one-line capability descriptions (24% of the routing prompt, ~1,669 tokens) cost 2.9 pp. Guards were untouched; the losses were disambiguation between capabilities whose names do not explain themselves, and unparseable outputs rose from 3 to 8. - **Evidence:** With docs: 0.9637; Terse: 0.9355; Broke: 39; Fixed: 7; Guards 174 both: 169; Unparseable: 3, 8 - **Measured on:** 2026-09-22 Cite: https://marvinlabs.org/findings#F20 ## F21: Q8_0 beat Q4_0 by 3.2 pp on the fine-tuned router {#F21} Quantizing the same merged weights to Q4_0 cost 3.2 pp for about 15% speed. Q8_0 stayed for every release. - **Evidence:** Q8: 0.8936; Q4: 0.8613; S per case: Q8: 1.506, Q4: 1.278 - **Measured on:** 2026-09-16 Cite: https://marvinlabs.org/findings#F21 ## F22: Inference speed is set by model size, not training choices {#F22} Every rank-112 fine-tune ran 1.17-1.34 s per case on the same card; rank 64 was a few percent faster (time ratio 0.913 vs rank 112 in one head-to-head; the ledger says 4%); the untuned 12B was 2.1-2.7 s. A model that lands split across CPU and GPU (because other models are resident) slows to ~2.8 s per case or worse, so evict between bake halves. - **Evidence:** R112 s per case: 1.17-1.34; R64 vs r112 time ratio: 0.913; R64 vs r16 time ratio: 0.917; Untuned 12b: 2.124, 2.685; Split CPU GPU: 2.8 - **Measured on:** 2026-09-25 Cite: https://marvinlabs.org/findings#F22 ## F23: A 4B fine-tune beat the untuned 12B it replaced {#F23} On the 1,125-case set the untuned gemma4:12b-it-qat scored 82.1% (held-out 88.9%) at 2.7 s per case; the best 4B fine-tune scored 96.4% (held-out 96.3%) at 1.25 s. The first fine-tune (v1, 152 examples) had scored 4.9 pp below the 12B; the 490-example v2 was the first to beat it. - **Evidence:** Untuned 12b: Accuracy: 0.8213, Held out: 0.8889, S per case: 2.685; V11: Accuracy: 0.9636, Held out: 0.963, S per case: 1.245; V1 vs 12b 550: 0.8036, 0.8527; V2 vs 12b 550: 0.8964, 0.8527 - **Measured on:** 2026-09-21 Cite: https://marvinlabs.org/findings#F23 ## F24: Measure the memory ceiling with a calibrated probe instead of crashing into it {#F24} A one-step probe on the longest example over-estimates a real run's peak by ~4.7%. Scaling the probe by that factor predicted rank 96 + dropout at 6,793 MB; the real run peaked at 6,629 MB (2.5% error). Discovering the rank-112 dropout ceiling by training had cost three crashes and about 11 GPU-hours. - **Evidence:** Probe MB: 7127.6; Predicted MB: 6793; Actual MB: 6629; Error pct: 2.5 - **Measured on:** 2026-09-23 Cite: https://marvinlabs.org/findings#F24 ## F25: Deterministic pre-router steps remove whole miss classes, invisibly to the bake {#F25} v11's largest miss class was 14 short confirmations routed to clarify. A deterministic step before the router now resolves a bare yes or no against real pending state with no model call. Separately, regex shortcuts route some standard session phrasings before any model call. Bakes call the model directly, so production accuracy on those phrasings is better than the bake shows. - **Evidence:** Confirmation miss class: 14; Session positives 31: Routed by regex correctly: 10, Routed by regex wrongly: 0; Backstop covered gold cases dropped for v16: 58 - **Measured on:** 2026-09-24 Cite: https://marvinlabs.org/findings#F25 ## F26: Small models need the context window set explicitly {#F26} Ollama gave each model a VRAM-derived default context of 4096 tokens while the routing prompt was ~5,650. For one model Ollama silently trimmed the prompt to 2,050 tokens, cutting off the capability list at its front, with its truncated flag still 0; routing still returned plausible answers, so misses looked like flakiness. Sending num_ctx explicitly on every call and a test that bounds prompt tokens against it fixed it. - **Evidence:** Default num ctx: 4096; Routing prompt tokens: 5650; Trimmed to tokens: 2050; Num ctx now: 9216 - **Measured on:** 2026-09-07 Cite: https://marvinlabs.org/findings#F26 ## F27: New capabilities often need little or no training {#F27} v11 handled 25 session-capability paraphrases and discriminators at 88% having never seen those capabilities, and refused all 16 session-injection guard attempts. - **Evidence:** Session paraphrases 25: 0.88; Session injection guards 16: 1.0 - **Measured on:** 2026-09-24 Cite: https://marvinlabs.org/findings#F27 ## F28: Fine-tunes trade cautious misses for confident ones {#F28} v2 beat the untuned 12B by 4.3 pp, but a larger share of its remaining misses were confident picks of a forbidden capability rather than an honest clarify. v7 likewise fixed clarify but lost ground on context and must-not-route cases. - **Evidence:** Forbidden pick share of misses: Untuned 12b: 6/81, V1: 8/108, V2: 12/57; V7 context cases n31: 0.968, 0.871 - **Measured on:** 2026-09-18 Cite: https://marvinlabs.org/findings#F28 ## F29: AMD and NVIDIA need the same memory for the same QLoRA step {#F29} The identical 4-bit QLoRA probe at 7,177 tokens peaked at 5,721.3 MB (rank 16) and 6,406.2 MB (rank 64) on an RX 7900 XTX under ROCm 7.2.1 on Windows, and at 5,788.9 MB and 6,465.0 MB on an RTX 3060 Ti under CUDA on Linux: within 1.2%, AMD slightly lower. Memory before the step matched to within 0.5 MB. A VRAM budget measured on one vendor carries over to the other for this recipe. - **Evidence:** Peak MB r16: Rx 7900 xtx: 5721.3, Rtx 3060 ti: 5788.9; Peak MB r64: Rx 7900 xtx: 6406.2, Rtx 3060 ti: 6465.0; Pre step MB r64: Rx 7900 xtx: 3597.7, Rtx 3060 ti: 3597.3 - **Measured on:** 2026-10-15 Cite: https://marvinlabs.org/findings#F29 ## F30: LoRA dropout costs about 170 MB at the probe, not 350 {#F30} Turning lora_dropout from 0 to 0.1 raised the one-step peak by 174.5, 173.7, 166.8 and 168.6 MB at ranks 96, 112, 128 and 256 on the RX 7900 XTX. The cost is close to flat across rank. The earlier 350 MB figure on the VRAM page was an estimate from crash margins, not a measurement. - **Evidence:** Dropout cost MB by rank: 96: 174.5, 112: 173.7, 128: 166.8, 256: 168.6 - **Measured on:** 2026-10-15 Cite: https://marvinlabs.org/findings#F30 ## F31: Rank cost is linear; DoRA and bf16 add large fixed costs {#F31} On the RX 7900 XTX at 7,177 tokens, the peak rose from 5,721.3 MB at rank 16 to 9,338.3 MB at rank 256, about 15.1 MB per rank, which matches the 15 MB per rank measured earlier on the 8 GB NVIDIA card. DoRA added about 1.83 GB at ranks 64 and 112 and made the step about 38% slower. Loading the base model in bf16 instead of 4-bit added about 5.1 GB at every rank. Without the memory fixes, the step needed about 36 GB. #### Evidence {#f31-evidence} ##### Peak MB by rank 4bit {#f31-evidence-peak-mb-by-rank-4bit} - **16:** 5721.3 - **64:** 6406.2 - **96:** 6884.8 - **112:** 7194.5 - **128:** 7395.9 - **192:** 8383.4 - **256:** 9338.3 - **DoRA extra MB:** 64: 1834.5; 112: 1824.3 - **Bf16 extra MB:** 16: 5105.0; 64: 5103.3; 112: 5123.0; 256: 5104.9 - **No fixes peak MB:** 16: 35829.2; 112: 36581.3 ##### Peak MB by seq len r112 {#f31-evidence-peak-mb-by-seq-len-r112} - **2048:** 5585.8 - **4096:** 6207.9 - **7177:** 7194.5 - **8192:** 7486.0 - **12288:** 8917.2 - **16384:** 10523.6 - **Measured on:** 2026-10-15 Cite: https://marvinlabs.org/findings#F31 ## F32: A 20B mixture-of-experts model generates faster than dense 9B and 12B models {#F32} On an RX 7900 XTX under Ollama 0.30.10, gpt-oss:20b generated about 116-123 tokens per second, against 84 for qwen3.5:9b and 66-68 for gemma4:12b-it-qat, and processed prompts 1.4 to 2.5 times faster than either. It needs the most memory, 12.3 GB resident, and the slowest cold load at 14.7 s. For a card with room to hold it, size in parameters is a poor guide to speed. - **Evidence:** Gen tok per s short long: Gpt-oss:20b: 123.0, 116.2, Qwen3.5:9b: 84.9, 84.2, Gemma4:12b-it-qat: 68.4, 66.4; Prompt tok per s short long: Gpt-oss:20b: 3730.2, 4118.1, Qwen3.5:9b: 2588.9, 2704.7, Gemma4:12b-it-qat: 1526.1, 1661.9; Resident MB: Gpt-oss:20b: 12261, Qwen3.5:9b: 5642, Gemma4:12b-it-qat: 7601 - **Note:** The mixture-of-experts explanation (only a few billion parameters active per token) is the usual reading of gpt-oss's design; this bench measured speed only. - **Status:** [inferred] - **Measured on:** 2026-10-15 Cite: https://marvinlabs.org/findings#F32 ## Tried and closed {#tried-and-closed} Methods tried or considered, and why each is closed. | Method | Verdict | Finding | Flags | | --- | --- | --- | --- | | incremental patch on a trained adapter | forgetting; retrain from base | [F03](#F03) | | | Q4_0 quantization | -3.2 pp; Q8_0 stays | [F21](#F21) | | | rank 128 | OOM | [F02](#F02) | | | DoRA at rank 64 and 112 | OOM inside the step | [F02](#F02) | | | checkpoint selection by eval loss | picked a 6.5 pp worse checkpoint | [F04](#F04) | | | rank 64 on the enriched corpus | -1.4 pp, clarify -13 pp | [F01](#F01) | | | two epochs | -0.8 pp | [F06](#F06) | | | grammar-constrained decoding | +0.1 pp with the right enum; kept on | [F19](#F19) | | | checkpoint averaging | noise | [F09](#F09) | | | rsLoRA | higher headline, worse guards | [F07](#F07) | | | LoRA dropout | does not fit at r112; no generalization gain at r96 | [F08](#F08) | | | 25% disjoint holdout corpus | -8.5 pp on the fair slice | [F10](#F10) | | | terse prompt without capability docs | -2.9 pp | [F20](#F20) | | | decoding temperature as a lever | already 0 | | | | MoE upcycling / widening | considered, not tried: a 9B upcycled to 4 experts needs ~13.5 GB quantized for weights alone | | [inferred] | | two-model disagreement cascade | dropped: no untuned local model was left that beat the fine-tune on any slice | | | ## Untried {#untried} Ideas the program did not get to. - LR sweep (2e-4 linear never varied) - LoRA+ - NEFTune, higher weight decay - class-balanced loss - LoRA on embeddings or the output head (needs more VRAM) - a bigger base (gemma-3-12b, Qwen3.5-9B) on rented hardware - training on real logged corrections (none collected before the program closed) --- # Gotchas > Environment and tooling failures hit during the program, with symptom, cause, fix and versions. Canonical: https://marvinlabs.org/gotchas Data: [gotchas.json](https://marvinlabs.org/data/gotchas.json) HTML: https://marvinlabs.org/gotchas Environment and tooling failures hit during the program, with symptom, cause, fix and versions. Versions are those recorded at the time; see recipe.json for the final stack. ## Gotchas {#gotchas} - [G01: Ollama silently truncates prompts longer than its default context](#G01) - [G02: NVIDIA driver/library mismatch blocks new CUDA processes but not running ones](#G02) - [G03: torchao too new for the pinned torch](#G03) - [G04: Unsloth's Gemma 4 E2B MoE forward patch breaks when any weight is off-GPU](#G04) - [G05: Forcing device_map={'': 0} silently disables Unsloth's 4-bit quantization](#G05) - [G06: bitsandbytes refuses auto device maps with any CPU placement](#G06) - [G07: Unsloth multi-GPU is not supported; Gemma 4 crashes across two cards](#G07) - [G08: transformers 5.5.0 mask utilities require torch 2.6](#G08) - [G09: Unsloth's Gemma3 attention patch fails on long sequences](#G09) - [G10: datasets.map() crashes pickling an Unsloth-patched tokenizer](#G10) - [G11: SFTTrainer argument renames](#G11) - [G12: Gemma3 training requires token_type_ids](#G12) - [G13: Gemma3 decoder moved under model.model.language_model](#G13) - [G14: OOM in the MLP, not attention, on ~6.3k-token sequences](#G14) - [G15: uv served a stale cached wheel across rebuilds](#G15) - [G16: flash-attn needs a system CUDA toolkit to build](#G16) - [G17: PEFT LoRA dropout costs a backward mask, at any rate above zero](#G17) - [G18: Xid 31 MMU fault during a near-full-memory training run](#G18) - [G19: DoRA does not fit at rank 64 or 112 on 8 GB](#G19) - [G20: A probe without model.train() silently disables gradient checkpointing](#G20) - [G21: A live Ollama server reloads models onto the training GPU within seconds](#G21) - [G22: A partly CPU-resident model pollutes bake timings](#G22) - [G23: Merged Gemma 3 checkpoints need a tensor rename before GGUF conversion](#G23) - [G24: Template substitution built a model tag from the wrong weights](#G24) - [G25: Test harness routed on the wrong channel](#G25) - [G26: Schema enum built from the visible docs dropped a hidden capability](#G26) - [G27: rsLoRA with alpha left at rank is far too hot](#G27) - [G28: Static prompt prefix was not cached between calls](#G28) - [G29: AMD ROCm/Vulkan GPU wedges (before this program)](#G29) - [G30: HF Trainer's train_runtime and train_loss cover only the last resumed segment](#G30) - [G31: On Windows, a training step larger than VRAM spills to system memory instead of failing](#G31) - [G32: Python hangs at exit after GPU work on Windows ROCm](#G32) ## G01: Ollama silently truncates prompts longer than its default context {#G01} - **symptom:** Intermittent routing misses that looked like model flakiness; the model still returned plausible answers. - **cause:** Ollama assigned each model a VRAM-derived num_ctx of 4096 while the routing prompt had grown to ~5,650 tokens. For one model it pre-trimmed the prompt to 2,050 tokens (visible only as task.n_tokens in the llama-server log), dropping the capability list at the front of the prompt. The response's truncated flag stayed 0. A model without context-shift support (LFM2) surfaced the same overflow as HTTP 400, which is how it was noticed. - **fix:** Send num_ctx explicitly on every call (8192 from 2026-09-07, 9216 from 2026-09-19) and keep a deterministic test that bounds the measured prompt token count against it. - **versions:** Ollama: 0.32 - **Measured on:** 2026-09-07 Cite: https://marvinlabs.org/gotchas#G01 ## G02: NVIDIA driver/library mismatch blocks new CUDA processes but not running ones {#G02} - **symptom:** torch.cuda.is_available() returned False with 'Error 804: forward compatibility was attempted on non supported HW'; nvidia-smi reported 'Driver/library version mismatch'. Already-running Ollama processes kept working. - **cause:** Kernel driver module and on-disk user-space libraries were at different versions after an update; processes that created a CUDA context before the change kept it, new processes could not create one. - **fix:** Reboot to reload the kernel module. torch.cuda.is_available() was True immediately after. - **versions:** Nvml library: 580.178; Torch: 2.5.1+cu121 at the time - **Measured on:** 2026-09-13 Cite: https://marvinlabs.org/gotchas#G02 ## G03: torchao too new for the pinned torch {#G03} - **symptom:** import unsloth failed at module load, before touching the GPU. - **cause:** uv resolved torchao 0.18.0, which references torch.int1, a dtype missing from torch 2.5.1. - **fix:** Pin torchao==0.9.0, contemporary with torch 2.5.x. - **versions:** Torch: 2.5.1+cu121; Torchao bad: 0.18.0; Torchao good: 0.9.0 - **Measured on:** 2026-09-13 Cite: https://marvinlabs.org/gotchas#G03 ## G04: Unsloth's Gemma 4 E2B MoE forward patch breaks when any weight is off-GPU {#G04} - **symptom:** RuntimeError: tensors on cpu and cuda:0 in index_select during the forward pass. - **cause:** Unsloth's loader offloaded the embedding table to CPU to fit, and the patched Gemma 4 MoE forward in unsloth_zoo does not move inputs to match. Reproduced with and without explicit fp32 CPU offload. Gemma 4 E2B LoRA needs 8-10 GB per Unsloth's own guide, so it never fits one 8 GB card without offload. - **fix:** None found. Switched base model to unsloth/gemma-3-4b-it, which loads in 4.39 GB with no offload. - **versions:** Unsloth: 2026.9.4; Unsloth zoo: 2026.9.3; Transformers: 5.5.0 (the maximum Unsloth's metadata allowed); Model: unsloth/gemma-4-E2B-it - **Measured on:** 2026-09-13 Cite: https://marvinlabs.org/gotchas#G04 ## G05: Forcing device_map={'': 0} silently disables Unsloth's 4-bit quantization {#G05} - **symptom:** OOM loading a model that should fit in 4-bit; message 'Unsloth: QLoRA and full finetuning all not selected. Switching to 16bit LoRA'. - **cause:** An explicit single-device map bypasses Unsloth's quantization detection regardless of the quantization_config passed. - **fix:** Do not force device_map with Unsloth's loader. Plain transformers + BitsAndBytesConfig honours device_map={'': 0} and quantizes correctly. - **versions:** Unsloth: 2026.9.4 - **Measured on:** 2026-09-13 Cite: https://marvinlabs.org/gotchas#G05 ## G06: bitsandbytes refuses auto device maps with any CPU placement {#G06} - **symptom:** ValueError: Some modules are dispatched on the CPU or the disk, even with the whole 8 GB card free. - **cause:** Accelerate's auto planner placed some modules on CPU; bnb's validate_environment refuses that unless fp32 CPU offload is opted in. Why more free VRAM gave a stricter failure than less was not understood. - **fix:** Not pursued; superseded by the base-model change. - **Measured on:** 2026-09-13 Cite: https://marvinlabs.org/gotchas#G06 ## G07: Unsloth multi-GPU is not supported; Gemma 4 crashes across two cards {#G07} - **symptom:** With both GPUs visible the 4-bit split loaded cleanly (4.38 + 3.17 GB) but the forward raised 'tensors on cuda:0 and cuda:1' inside transformers' modeling_gemma4.py. - **cause:** Gemma 4's unified text/vision input path masks in a pad_embedding buffer that ends up on a different GPU from inputs_embeds; neither Unsloth's patches nor accelerate's dispatch hooks move it. Unsloth's multi-GPU PR #2074 was closed unmerged (2025-03-18). - **fix:** Single-GPU only. Every Gemma 4 size is unified multimodal, so none avoids this; stepped back to Gemma 3 4B. - **versions:** Transformers: 5.5.0; Unsloth: 2026.9.4 - **Measured on:** 2026-09-13 Cite: https://marvinlabs.org/gotchas#G07 ## G08: transformers 5.5.0 mask utilities require torch 2.6 {#G08} - **symptom:** ValueError: Using or_mask_function ... require torch>=2.6 on the first gemma-3-4b forward. - **cause:** torch was pinned at 2.5.1 from the earlier driver test. - **fix:** Upgrade in place to torch 2.6.0+cu124 with a matching xformers 0.0.29.post3, pinned together. A blind 'pip -U xformers' pulled torch 2.14.0, outside Unsloth's supported range (<2.13.0); reverted. - **versions:** Torch: 2.5.1+cu121 -> 2.6.0+cu124; Xformers: 0.0.29.post3; Transformers: 5.5.0 - **Measured on:** 2026-09-13 Cite: https://marvinlabs.org/gotchas#G08 ## G09: Unsloth's Gemma3 attention patch fails on long sequences {#G09} - **symptom:** 'too many values to unpack' on real ~6300-token examples; a 2048-token trial had passed. - **cause:** Length-dependent bug in unsloth_zoo/temporary_patches/gemma.py. - **fix:** Drop FastLanguageModel; use plain transformers AutoModelForCausalLM + BitsAndBytesConfig + peft with the same LoRA config. - **versions:** Unsloth zoo: 2026.9.3 - **Measured on:** 2026-09-14 Cite: https://marvinlabs.org/gotchas#G09 ## G10: datasets.map() crashes pickling an Unsloth-patched tokenizer {#G10} - **symptom:** cannot pickle 'ConfigModuleInstance' object, from both SFTTrainer._prepare_dataset and Unsloth's train_on_responses_only. - **cause:** Dill tries to pickle a closure over the patched tokenizer that captures a torch._dynamo/_inductor-style config singleton. - **fix:** Pre-tokenize and mask by hand in the main process; pass dataset_kwargs={'skip_prepare_dataset': True}; no .map() anywhere. - **versions:** Datasets: 4.3.0; Trl: 0.24.0 - **Measured on:** 2026-09-14 Cite: https://marvinlabs.org/gotchas#G10 ## G11: SFTTrainer argument renames {#G11} - **symptom:** TypeError on tokenizer= and max_seq_length=. - **cause:** Renamed to processing_class= and max_length= in this trl version. - **fix:** Use the new names (later replaced SFTTrainer with plain Trainer and a custom collator). - **versions:** Trl: 0.24.0 - **Measured on:** 2026-09-14 Cite: https://marvinlabs.org/gotchas#G11 ## G12: Gemma3 training requires token_type_ids {#G12} - **symptom:** Clear ValueError during training with plain transformers. - **cause:** Gemma3's forward expects token_type_ids in training mode (image/text segmenting). - **fix:** Supply all zeros for text-only data. - **versions:** Transformers: 5.5.0 - **Measured on:** 2026-09-14 Cite: https://marvinlabs.org/gotchas#G12 ## G13: Gemma3 decoder moved under model.model.language_model {#G13} - **symptom:** A trainer copy that iterated model.model.layers broke. - **cause:** transformers' Gemma3 code nests the decoder at model.model.language_model.layers. - **fix:** Find modules by type name (Gemma3MLP) via model.modules(), which ignores nesting depth. - **versions:** Transformers: 5.5.0 - **Measured on:** 2026-09-16 Cite: https://marvinlabs.org/gotchas#G13 ## G14: OOM in the MLP, not attention, on ~6.3k-token sequences {#G14} - **symptom:** First real training step OOMed (7.39 GB used, tried 248 MB) inside an early decoder layer; flash-attn, Liger, attention-only LoRA, logits_to_keep, paged optimizer, allocator tuning and a two-GPU layer split all left the same ~200-250 MB gap. - **cause:** gate_proj(x) and up_proj(x) over the full sequence must coexist for the gated product (the base model's tensors, not LoRA's). Also: 34 checkpoint-boundary hidden states stayed on GPU, and lm_head shipped in fp32 (2.68 GB). - **fix:** Chunk the MLP over 2048-token slices; wrap training in save_on_cpu(pin_memory=True); cast lm_head to bf16. First clean run peaked at 4.93 GB. - **Measured on:** 2026-09-14 Cite: https://marvinlabs.org/gotchas#G14 ## G15: uv served a stale cached wheel across rebuilds {#G15} - **symptom:** flash-attn rebuilt with different CFLAGS three times and kept producing the same wheel despite --reinstall and --no-cache. - **cause:** uv's build cache is keyed on source content, not environment variables. - **fix:** Delete the package's directory under uv's sdists cache directly. - **Measured on:** 2026-09-14 Cite: https://marvinlabs.org/gotchas#G15 ## G16: flash-attn needs a system CUDA toolkit to build {#G16} - **symptom:** pip install flash-attn failed immediately with 'CUDA_HOME environment variable is not set'. - **cause:** Only the pip-installed CUDA runtime was present, no nvcc. - **fix:** Installed the CUDA 13.0 toolkit only (no driver component, so the working 580.178.04 driver was untouched). The build then worked, but flash-attn turned out to be the wrong lever (G14). - **versions:** Cuda toolkit: 13.0; Driver: 580.178.04; Flash attn: 2.8.3.post1 - **Measured on:** 2026-09-14 Cite: https://marvinlabs.org/gotchas#G16 ## G17: PEFT LoRA dropout costs a backward mask, at any rate above zero {#G17} - **symptom:** Rank-112 runs with lora_dropout=0.1 died at steps 236 and 239 with CUDA OOM (~6.9-7.0 GiB allocated of 7.65 GiB) where the same config at dropout 0 peaked at 6,653 MB. - **cause:** At lora_dropout=0 PEFT installs nn.Identity (no allocation); above 0 it installs nn.Dropout, which saves a mask per call for backward across 7 target modules on 34 layers. Estimated ~350 MB of peak. Lowering the rate to 0.05 would not help. - **fix:** Keep dropout at 0, or drop rank (rank 96 + dropout trained at 6,629 MB). expandable_segments bought only 3 steps. - **versions:** Peft: 0.20.0 (peft/tuners/lora/layer.py:251) - **Measured on:** 2026-09-23 Cite: https://marvinlabs.org/gotchas#G17 ## G18: Xid 31 MMU fault during a near-full-memory training run {#G18} - **symptom:** Training died at step 184 with 'CUDA error: an illegal memory access was encountered' inside bitsandbytes' optimizer step; the program notes records it as Xid 31 (MMU fault on the graphics engine). The kernel log itself was not re-checked for this dataset. The GPU recovered by itself. - **cause:** Not established. It happened on the same dropout config that later OOMed twice at ~0 free memory, the first Xid on the box in 7 days. - **fix:** None needed beyond abandoning the config; nothing wedged. - **versions:** Bitsandbytes: 0.50.2; Driver: 580.178.04 - **Measured on:** 2026-09-22 Cite: https://marvinlabs.org/gotchas#G18 ## G19: DoRA does not fit at rank 64 or 112 on 8 GB {#G19} - **symptom:** OOM inside the DoRA weight-norm computation on the first step, with the GPU otherwise clear. - **cause:** DoRA's magnitude/weight-norm path allocates extra per-layer tensors on top of LoRA's. - **fix:** None on this card; needs shorter sequences or more VRAM. - **Status:** [inferred] - **Measured on:** 2026-09-18 Cite: https://marvinlabs.org/gotchas#G19 ## G20: A probe without model.train() silently disables gradient checkpointing {#G20} - **symptom:** A one-step memory probe thrashed the host and later hit a 32 GB cgroup memory cap. - **cause:** HF only checkpoints when module.training is True. Without it, save_on_cpu spilled every activation into pinned host RAM. - **fix:** Call model.train() in the probe; run probes under systemd-run --scope -p MemoryMax=12G -p MemorySwapMax=0. - **Measured on:** 2026-09-17 Cite: https://marvinlabs.org/gotchas#G20 ## G21: A live Ollama server reloads models onto the training GPU within seconds {#G21} - **symptom:** Training launches OOMed before reaching the loop, or were killed mid-run by the host's low-memory monitor (4 times for v5, 9 relaunches for v2). - **cause:** Production traffic reloaded an evicted model onto the training GPU within 1-2 s of 'ollama stop', winning the race against the trainer's allocation. save_on_cpu's pinned host memory also raised host-RAM pressure. - **fix:** A launcher that retries stop+launch up to 15 times and declares success only when the training loop starts; save a checkpoint every 25 steps and auto-resume from the latest. Do not evict the embedding model (its eviction livelocked its client). - **Measured on:** 2026-09-15 Cite: https://marvinlabs.org/gotchas#G21 ## G22: A partly CPU-resident model pollutes bake timings {#G22} - **symptom:** Mean seconds per case jumped (1.71 for one model, ~2.8 s per case for split placement, up to 4.2 in a contended bake) while accuracy looked normal. - **cause:** With two other models resident, Ollama placed the new model split across CPU and GPU. - **fix:** Evict finished models between bake halves; read speed only from bakes with the model fully on GPU. - **Measured on:** 2026-09-18 Cite: https://marvinlabs.org/gotchas#G22 ## G23: Merged Gemma 3 checkpoints need a tensor rename before GGUF conversion {#G23} - **symptom:** Not recorded in the sources; the step exists in every export chain. - **cause:** The merged checkpoint carries the vision tower and projector plus a model.language_model.* prefix on text tensors. - **fix:** Before convert_hf_to_gguf.py, drop the 439 vision/projector tensors and rename the 444 text tensors to language_model.* (883 total). Verify the count every time. - **Status:** [inferred] - **Measured on:** 2026-09-15 Cite: https://marvinlabs.org/gotchas#G23 ## G24: Template substitution built a model tag from the wrong weights {#G24} - **symptom:** The v3 Ollama tag would have served v2's GGUF. - **cause:** A sed substitution on the Modelfile assumed a filename pattern the new version did not follow. - **fix:** Verify the model-layer sha256 and Ollama model id differ from the previous version before baking; grep-verify every path. - **Measured on:** 2026-09-15 Cite: https://marvinlabs.org/gotchas#G24 ## G25: Test harness routed on the wrong channel {#G25} - **symptom:** Every context-dependent case scored as if it had no context, for every model, since July. - **cause:** Bake scripts routed on a default channel while each case's setup exchange seeded context on the case's own channel. - **fix:** Route each case on its setup's channel. v9's held-out rose from 86.8% to 91.7% on the same weights. - **Measured on:** 2026-09-19 Cite: https://marvinlabs.org/gotchas#G25 ## G26: Schema enum built from the visible docs dropped a hidden capability {#G26} - **symptom:** Turning on grammar-constrained decoding broke 10 cases, 8 of them one capability forced to clarify. - **cause:** The JSON-schema enum came from the prompt's documented capability list, which hides one capability the router still accepts. - **fix:** Build the enum from every available capability, hidden ones included. - **Measured on:** 2026-09-21 Cite: https://marvinlabs.org/gotchas#G26 ## G27: rsLoRA with alpha left at rank is far too hot {#G27} - **symptom:** Held-out eval loss flat at ~0.57 by step 50 (plain recipe ~0.33); train loss peaked at 3.85. - **cause:** rsLoRA scales by alpha/sqrt(r): 112/sqrt(112) = 10.6, versus 1.0 for the plain recipe at lr 2e-4. - **fix:** Lower alpha (32 gives scale 3.0). When averaging rsLoRA checkpoints by concatenation, also scale B by 1/sqrt(n). - **Measured on:** 2026-09-21 Cite: https://marvinlabs.org/gotchas#G27 ## G28: Static prompt prefix was not cached between calls {#G28} - **symptom:** Warm routing calls took ~6.03 s for three different models with no speed-up across repeated same-prefix calls; one model (gemma4:12b) never reused cache once the dynamic tail varied (49-270 s per call in a reproduction). - **cause:** Partly sliding-window-attention / hybrid-memory cache restore limits in llama.cpp; partly contention from other models sharing the server. Not fully separated. - **fix:** Raised the router timeout (60 to 90 s) so a slow call is not turned into a wrong answer; planned (not done during the program) a dedicated GPU-pinned router instance. - **Measured on:** 2026-09-13 Cite: https://marvinlabs.org/gotchas#G28 ## G29: AMD ROCm/Vulkan GPU wedges (before this program) {#G29} - **symptom:** Three independent kernel wedge or device-loss incidents with the earlier AMD RX 7900 XTX serving Ollama. - **cause:** ROCm/Vulkan driver stack instability under an always-on local LLM. - **fix:** Hardware swap to NVIDIA/CUDA on 2026-08-07/08; no recurrence since. Context only: all fine-tuning ran on NVIDIA. - **Measured on:** 2026-08-08 Cite: https://marvinlabs.org/gotchas#G29 ## G30: HF Trainer's train_runtime and train_loss cover only the last resumed segment {#G30} - **symptom:** v2 reported train_loss 0.0274, far below every other run. - **cause:** After resume_from_checkpoint the reported aggregate covers only the steps run in that process. - **fix:** Read per-step losses from log_history across segments, or report peak and loss only for uninterrupted runs. - **Status:** [inferred] - **Measured on:** 2026-09-14 Cite: https://marvinlabs.org/gotchas#G30 ## G31: On Windows, a training step larger than VRAM spills to system memory instead of failing {#G31} - **symptom:** With the memory fixes off, the probe on a 24 GB RX 7900 XTX reported a 35.8 to 36.6 GB peak and finished, with no out-of-memory error. The step took 58 to 69 s instead of about 14 s. - **cause:** The Windows display driver model lets GPU allocations overflow into shared system memory. The allocator's peak can exceed the card's capacity, and the only sign is the slowdown. - **fix:** On Windows, treat a peak above the card's total memory, or a step several times slower than expected, as a failure. A config that 'fits' on Windows this way runs out of memory on Linux. - **versions:** Windows 11 build 26200, Adrenalin 26.7.1, ROCm 7.2.1, torch 2.9.1 - **Status:** [inferred] - **Measured on:** 2026-10-15 Cite: https://marvinlabs.org/gotchas#G31 ## G32: Python hangs at exit after GPU work on Windows ROCm {#G32} - **symptom:** A probe printed its result and then never exited; the process had to be killed. - **cause:** Interpreter shutdown hangs after ROCm GPU work on Windows. The cause is not known. - **fix:** Print results and flush, then call os._exit() to skip normal teardown, and run each probe in its own process with a watchdog that kills it shortly after the result line. - **versions:** Windows 11, ROCm 7.2.1, torch 2.9.1+rocm7.2.1, Python 3.12 - **Measured on:** 2026-10-15 Cite: https://marvinlabs.org/gotchas#G32 --- # Training recipe > The shared training recipe: model, quantization, LoRA settings, hyperparameters, memory fixes, export path and release rule. Canonical: https://marvinlabs.org/recipe Data: [recipe.json](https://marvinlabs.org/data/recipe.json) HTML: https://marvinlabs.org/recipe The training recipe shared by every fresh-from-base version (v1-v3, v5-v17). v4 differs only in continuing from v3's adapter at 8 epochs. Values come from the training scripts and the run headers printed in each training log. ## Task {#task} - **Description:** Supervised LoRA fine-tune. Input is the production routing prompt (system + user, ~5.6k-6.3k tokens incl. capability docs and recent context); target is the assistant's single JSON capability call (13-128 tokens, median 26). - **Loss:** response-only: prompt tokens masked; labels and logits computed only for the last response_len + 1 positions via logits_to_keep - **Example length tokens:** Min: 6277; Max: 6447; Corpus: v1-v5 era; measured on 2026-09-14 ### Longest example later corpora tokens {#task-longest-example-later-corpora-tokens} | Corpus | Tokens | Measured on | | --- | --- | --- | | 2026-09-15 | 7037 | 2026-09-17 | | 2026-09-18 unfiltered | 7288 | 2026-09-18 | | 2026-09-18 filtered | 6931 | 2026-09-18 | | 2026-09-19 filtered | 7068 | 2026-09-20 | | 2026-09-21 | 7177 | 2026-09-23 | - **Response span tokens:** Min: 13; Max: 128; Median: 26; measured on 2026-09-13 ## Base model {#base-model} - **HF repo:** unsloth/gemma-3-4b-it - **Text layers:** 34 - **Vocab:** 262208 - **Hidden:** 2560 - **Measured on:** 2026-09-14 ## Quantization for training {#quantization-for-training} - **Method:** bitsandbytes 4-bit QLoRA - **Load in 4bit:** yes - **Bnb 4bit quant type:** nf4 - **Bnb 4bit use double quant:** yes - **Bnb 4bit compute dtype:** bfloat16 - **Lm head:** cast from fp32 to bf16 after get_peft_model - **Measured on:** 2026-09-18 ## LoRA {#lora} **Target modules** - q_proj - k_proj - v_proj - o_proj - gate_proj - up_proj - down_proj - **Rank:** Shipped: 112; Tried: 16, 64, 96, 112; Probed oom: 128; measured on 2026-09-25 - **Alpha:** equal to rank (scale 1.0) in every shipped version; rsLoRA runs used alpha 32 at r112 and 30 at r96 - **Dropout:** 0 (0.1 tried in v14/v15, closed) - **RsLoRA:** off (tried v13-v15, closed) - **DoRA:** off (OOM at r64 and r112) - **Bias:** none - **Measured on:** 2026-09-25 ## Hyperparameters {#hyperparameters} - **Epochs:** 3 - **Learning rate:** 0.0002 - **Lr scheduler:** linear - **Warmup steps:** 5 - **Per device train batch size:** 1 - **Gradient accumulation steps:** 8 - **Effective batch:** 8 - **Optimizer:** paged_adamw_8bit - **Weight decay:** 0.01 - **Seed:** 3407 - **Bf16:** yes - **Max seq length:** 8192 - **Gradient checkpointing:** on, use_reentrant=False, via prepare_model_for_kbit_training - **Attn implementation:** sdpa - **Save and eval steps:** 25 - **Decoding temperature at inference:** 0 - **Measured on:** 2026-09-18 ## Memory fixes {#memory-fixes} | Fix | Why | Measured on | | --- | --- | --- | | chunked MLP: Gemma3MLP.forward monkey-patched to run gate/up/down over 2048-token chunks of the sequence | gate_proj(x) and up_proj(x) over ~6300 tokens must coexist for the gated product; that pair was the measured OOM point | 2026-09-14 | | torch.autograd.graph.save_on_cpu(pin_memory=True) around trainer.train() | gradient-checkpoint boundary tensors for 34 layers stayed on GPU; memory climbed 4.5 to 6.99 GB in one forward, 6.39 GB with the offload | 2026-09-14 | | lm_head cast fp32 to bf16 | fp32 lm_head was 2.68 GB and forced a mixed-precision matmul; pre-training VRAM fell 4502 MB to 3221 MB | 2026-09-14 | | logits_to_keep = response_len + 1 | full [1, 6390, 262208] fp32 logits would be 6.7 GB vs 135 MB for the response span; did not fix the first OOM (that was the MLP) but removes the tensor whenever the forward reaches lm_head | 2026-09-14 | | PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True | allocator setting only; bought 3 steps in the v14 dropout runs, not a fix | 2026-09-23 | | pre-tokenize and mask by hand; no datasets.map() | map() crashed dill-pickling a closure over the Unsloth-patched tokenizer | 2026-09-14 | | token_type_ids all zeros | plain transformers Gemma3 requires them during training (clear ValueError) | 2026-09-14 | ## Did not help {#did-not-help} | Lever | | --- | | expandable_segments allocator tuning | | paged_adamw_8bit (kept, but the first OOM was in forward, before any optimizer step) | | non-reentrant gradient checkpointing alone | | splitting layers across both GPUs with device_map=auto (pipeline split does not shrink one layer's activations) | | attention-only LoRA targets (moved the OOM one line; the floor is the base MLP) | | Liger Kernel fused GEGLU (fuses only the elementwise step after both projections exist) | | flash-attn (attention was never the bottleneck; built from source after installing a CUDA 13.0 toolkit, then abandoned) | ## Why plain transformers peft not unsloth {#why-plain-transformers-peft-not-unsloth} - **Decision:** train with transformers.AutoModelForCausalLM + BitsAndBytesConfig + peft.get_peft_model; Unsloth's FastLanguageModel loader dropped on 2026-09-14 **Reasons** - Unsloth's patched Gemma 4 E2B MoE forward crashed whenever any part of the model sat off-GPU (embedding offloaded to CPU, index_select device mismatch); E2B needs 8-10 GB and does not fit one 8 GB card. - Unsloth multi-GPU was not supported (PR #2074 closed unmerged); spanning both cards loaded cleanly but transformers' own Gemma 4 forward then hit a cuda:0/cuda:1 mismatch. - Forcing device_map={'': 0} silently disabled Unsloth's 4-bit quantization ('Switching to 16bit LoRA') and OOMed. - On gemma-3-4b-it, Unsloth's Gemma3 attention patch raised 'too many values to unpack' on real ~6300-token sequences (a 2048-token trial never reached that path). - datasets.map() via Unsloth's train_on_responses_only crashed pickling the patched tokenizer. - **Measured on:** 2026-09-14 ## Libraries {#libraries} ### Versions {#libraries-versions} - **Python:** 3.12.13 - **Torch:** 2.6.0+cu124 - **Transformers:** 5.5.0 - **Peft:** 0.20.0 - **Trl:** 0.24.0 - **Accelerate:** 1.15.0 - **Bitsandbytes:** 0.50.2 - **Datasets:** 4.3.0 - **Safetensors:** 0.8.0 - **Xformers:** 0.0.29.post3 - **Unsloth:** 2026.9.4 - **Unsloth zoo:** 2026.9.3 - **Liger kernel:** 0.8.2 - **Flash attn:** 2.8.3.post1 (built, unused) - **Torchao:** 0.9.0 (pinned; not present in the venv on 2026-10-15) - **Note:** Read from the training venv's installed packages on 2026-10-15. Install dates are 2026-09-13; no later upgrade is recorded, so these are taken to be the training versions. torch was upgraded in place from 2.5.1+cu121 to 2.6.0+cu124 on 2026-09-13. - **Measured on:** 2026-10-15 - **Status:** [inferred] ## Driver and hardware {#driver-and-hardware} - **Nvidia driver:** 580.178.04 - **Cuda toolkit installed:** 13.0 (toolkit only, for the flash-attn build) - **Training GPU:** CUDA device 1 under CUDA_DEVICE_ORDER=PCI_BUS_ID, which is an RTX 3070 Ti 8 GB on this box; torch reports 7.65 GiB total capacity - **Other GPU:** RTX 3060 Ti 8 GB. Not used for training. The live Ollama server shared the cards and repeatedly reloaded models onto the training GPU (see gotchas). - **Note:** GPU model names read from nvidia-smi on 2026-10-15; no September log names the card. The program notes confirms 'two 8GB GPUs' and ~7.65 GB usable. - **Measured on:** 2026-10-15 - **Status:** [inferred] ## Runtime per run {#runtime-per-run} - **Rank112 full corpus hours:** Value: 11.8, 12.5; Runs: v11 (1,167 ex, 438 steps), v17 (1,189 ex, 447 steps); measured on 2026-09-25 - **Rank112 two epochs hours:** Value: 7.9; measured on 2026-09-21 - **Seconds per optimizer step r16:** Value: 81; measured on 2026-09-15 - **Note:** HF Trainer's train_runtime covers only the last resumed segment when a run was killed and resumed (v2, v5). ## Export path {#export-path} **Steps** - merge: load base in bf16 on CPU, PeftModel.from_pretrained, merge_and_unload, save safetensors (merge_adapter_generic.py) - tensor-name fix: drop the 439 vision/projector tensors and rename the 444 text tensors from model.language_model.* to language_model.* (fix_tensor_names_generic.py; 883 tensors total) - llama.cpp convert_hf_to_gguf.py --outtype f16 - llama.cpp llama-quantize ... Q8_0 - ollama create from a Modelfile (below) - **Tensor fix reason:** not stated in the sources; the merged checkpoint carries Gemma 3's vision tower and a nested text-model prefix, and only the text tensors are kept - **Llama cpp checkout:** commit ad6c668, committed 2026-09-14 (checked 2026-10-15) - **GGUF sizes GB:** F16: 7.78; Q8 0: 4.14; Q4 0: 2.37 - **Verification lesson:** a sed substitution once built the v3 Ollama tag from v2's GGUF; caught by comparing model-layer sha256 and Ollama model id before baking - **Measured on:** 2026-09-25 - **Status:** [inferred] ## Ollama modelfile {#ollama-modelfile} **Text** ``` FROM /gemma3-4b-router-.Q8_0.gguf TEMPLATE """{{- range $i, $_ := .Messages }} {{- $last := eq (len (slice $.Messages $i)) 1 }} {{- if or (eq .Role "user") (eq .Role "system") }}user {{ .Content }} {{ if $last }}model {{ end }} {{- else if eq .Role "assistant" }}model {{ .Content }}{{ if not $last }} {{ end }} {{- end }} {{- end }}""" PARAMETER stop PARAMETER temperature 1 PARAMETER top_k 64 PARAMETER top_p 0.95 PARAMETER num_ctx 8192 ``` - **Notes:** system and user turns both render as a Gemma user turn, the router overrides temperature to 0 per request, the client sends num_ctx 9216 per request in production (code default since 2026-09-19); the Modelfile's 8192 is a fallback - **Measured on:** 2026-09-25 ## Q8 0 vs q4 0 {#q8-0-vs-q4-0} - **Model:** v5 (rank 16) - **Test set:** 865 - **Q8 0:** Passed: 773; Accuracy: 0.8936; S per case: 1.506 - **Q4 0:** Passed: 745; Accuracy: 0.8613; S per case: 1.278 - **Delta pp:** -3.23 - **Speed ratio q4 over q8:** 0.849 - **Verdict:** Q8_0 stays: -3.2 pp is not worth ~15% speed. - **Measured on:** 2026-09-16 ## Release rule {#release-rule} - **Rule:** veto any model that fails a held-out guard an incumbent passes; rank the rest by held-out accuracy; break ties within 0.5 pp by overall accuracy. For v17 the scripted rule was: beat the incumbent on the fair slice and add no new held-out guard breach. - **Why:** the earlier 'highest overall accuracy wins' rule picked v13 (rsLoRA) on 22 Sep and had to be overruled - **Measured on:** 2026-09-25 --- # About: method and scope > How the numbers were measured, what the terms mean, what is not published, and how to report a correction. Canonical: https://marvinlabs.org/about Data: [baselines.json](https://marvinlabs.org/data/baselines.json), [findings.json](https://marvinlabs.org/data/findings.json), [gotchas.json](https://marvinlabs.org/data/gotchas.json), [inference.json](https://marvinlabs.org/data/inference.json), [recipe.json](https://marvinlabs.org/data/recipe.json), [versions.json](https://marvinlabs.org/data/versions.json), [vram.json](https://marvinlabs.org/data/vram.json) HTML: https://marvinlabs.org/about ## Method {#method} Each version was trained, served through Ollama and scored against a routing test set. Dates are UTC. Every record carries a measured-on date. ### Definitions {#definitions} In short: accuracy is the share of routing test cases answered with the right capability and args. Held-out and fair-slice cases are those with no training label. - **Accuracy:** passed / runnable cases. A case passes when the router returns the asserted capability and arguments, and none of the case's forbidden capabilities. - **Held out:** cases with no training label in the corpus the model was trained on. The held-out slice changed size as the case file grew, so compare held-out numbers only within one test-set era. - **Fair slice:** cases carrying no train key at all, which neither compared model ever saw (used for v16 and v17). - **Guard:** must-not-fire cases: utterances that must NOT route to a named capability (negations, quoted or pasted content, vague phrasing). Counts only; case text is excluded. - **Speed:** mean seconds per case through the production routing prompt (~5.6k tokens), model fully on GPU unless noted. - **Dates:** UTC. measured_on is the file timestamp of the result, or the date the program notes records. ### Scope notes from the data {#scope-notes} - **findings.json:** Lessons from 17+ LoRA fine-tunes of gemma-3-4b-it as a command router on one 8 GB consumer GPU. Each claim is limited to this task, this base model and this recipe. Accuracy numbers are passed / runnable cases; see versions.json for definitions. - **recipe.json:** The training recipe shared by every fresh-from-base version (v1-v3, v5-v17). v4 differs only in continuing from v3's adapter at 8 epochs. Values come from the training scripts and the run headers printed in each training log. - **vram.json:** Every VRAM measurement from the program. The cross_gpu section adds one-step probes on an AMD RX 7900 XTX (24 GB) and a second NVIDIA card. Training GPU: one 8 GB NVIDIA consumer card (an RTX 3070 Ti by today's nvidia-smi; torch reports 7.65 GiB total capacity). 'mode' is probe (one forward+backward on the single longest example, no optimizer step) or real (a full Trainer run; peak = torch.cuda.max_memory_allocated at the end). All training configs are 4-bit nf4 QLoRA on unsloth/gemma-3-4b-it with the recipe.json memory fixes unless noted. - **baselines.json:** Untuned (off-the-shelf) models, measured before and during the fine-tune program through Ollama. 'routing' = the production command-routing prompt (~5.6k tokens); 'content' = 11 generation workloads (summaries, tagging, narration and similar) scored by a judge model. 'floor' tier = unambiguous cases every acceptable model must pass; 'hard' tier = deliberately ambiguous cases. Tournament numbers below were recomputed from the raw per-cycle score records by time window and checked against the program's written notes where they quote them. - **gotchas.json:** Environment and tooling failures hit during the program, with symptom, cause, fix and versions. Versions are those recorded at the time; see recipe.json for the final stack. ## What is not published {#not-published} The training corpus, the text of the test cases and the exact text of the safety-guard cases are deliberately not published. They appear only as counts and broad categories. ## Corrections {#corrections} Numbers marked unverified or inferred are flagged in the data. An inferred number was derived by the authors, not read straight from a result file. An unverified number could not be checked against a primary result. Contact: marvinmediaai@gmail.com ## License {#license} The data and text on this site are licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Use them for anything, with credit to MarvinLabs and a link to marvinlabs.org. ## Data files {#data-files} - [baselines.json](data/baselines.json) - [findings.json](data/findings.json) - [gotchas.json](data/gotchas.json) - [inference.json](data/inference.json) - [recipe.json](data/recipe.json) - [versions.json](data/versions.json) - [vram.json](data/vram.json)