# Findings > Every finding with its evidence and measured date, plus the methods tried and closed and the ideas left untried. Canonical: https://marvinlabs.org/findings Data: [findings.json](https://marvinlabs.org/data/findings.json) HTML: https://marvinlabs.org/findings Lessons from 17+ LoRA fine-tunes of gemma-3-4b-it as a command router on one 8 GB consumer GPU. Each claim is limited to this task, this base model and this recipe. Accuracy numbers are passed / runnable cases; see versions.json for definitions. ## Findings {#findings} - [F01: LoRA rank, not more data, moved the hardest capability](#F01) - [F02: Rank 112 is the ceiling on an 8 GB card at ~7k-token examples](#F02) - [F03: Incremental patching on a trained adapter caused catastrophic forgetting](#F03) - [F04: Held-out eval loss does not pick the best checkpoint](#F04) - [F05: Accuracy plateaus well before the end, and the final checkpoint is as good as any](#F05) - [F06: A third epoch is worth about one point](#F06) - [F07: rsLoRA raised the headline and eroded the safety guards](#F07) - [F08: LoRA dropout did not fit at rank 112 and did not improve generalization where it fit](#F08) - [F09: Checkpoint averaging is noise on this recipe](#F09) - [F10: Data quantity beat data purity: a 25% disjoint holdout cost 8.5 pp](#F10) - [F11: The test set was mostly training data, but the plain recipe still generalized](#F11) - [F12: Negative guards need training targets](#F12) - [F13: Guard batches need counterweights or they teach over-refusal](#F13) - [F14: Label conflicts cost more than missing data](#F14) - [F15: A single bake pass over a small set overstated a margin by about 2x](#F15) - [F16: Run-to-run noise differs by model; repeat bakes before comparing](#F16) - [F17: The overall number can rank the worst router first; a guard veto is needed](#F17) - [F18: A harness channel bug scored every context case context-free for weeks](#F18) - [F19: Grammar-constrained decoding helps only when the enum matches what the router accepts](#F19) - [F20: One-line capability docs earn their prompt tokens](#F20) - [F21: Q8_0 beat Q4_0 by 3.2 pp on the fine-tuned router](#F21) - [F22: Inference speed is set by model size, not training choices](#F22) - [F23: A 4B fine-tune beat the untuned 12B it replaced](#F23) - [F24: Measure the memory ceiling with a calibrated probe instead of crashing into it](#F24) - [F25: Deterministic pre-router steps remove whole miss classes, invisibly to the bake](#F25) - [F26: Small models need the context window set explicitly](#F26) - [F27: New capabilities often need little or no training](#F27) - [F28: Fine-tunes trade cautious misses for confident ones](#F28) - [F29: AMD and NVIDIA need the same memory for the same QLoRA step](#F29) - [F30: LoRA dropout costs about 170 MB at the probe, not 350](#F30) - [F31: Rank cost is linear; DoRA and bf16 add large fixed costs](#F31) - [F32: A 20B mixture-of-experts model generates faster than dense 9B and 12B models](#F32) ## F01: LoRA rank, not more data, moved the hardest capability {#F01} Three rounds of more data (v3, v4, v5) left the clarify capability at 56-60%. Raising rank from 16 to 64 on the same 757 examples lifted overall accuracy 4 pp and clarify to ~75%; rank 112 lifted clarify to ~85%. Dropping back to rank 64 on a later corpus cost clarify 13 pp. - **Evidence:** Overall r16 vs r64: 0.8902, 0.9306; Clarify by rank 865set: R16: 0.521, R64: 0.755, R112: 0.851; V10 r64 vs v9 r112 clarify: 0.73, 0.86; V10 vs v9 overall 1125set: 0.9164, 0.9307 - **Measured on:** 2026-09-20 Cite: https://marvinlabs.org/findings#F01 ## F02: Rank 112 is the ceiling on an 8 GB card at ~7k-token examples {#F02} With chunked MLP, save_on_cpu and a bf16 lm_head, rank 112 peaks at ~6.65 GB in real runs; rank 128 and DoRA (at 64 and 112) OOM on the longest example. Rank 112 over 64 bought +0.5 pp on the 865 set, inside run-to-run noise there, but held clarify gains later. - **Evidence:** Real peak r112 MB: 6643-6658; Probe r128: OOM; Probe DoRA r64: OOM; Probe DoRA r112: OOM; V7 vs v6: 0.9434, 0.9387 - **Measured on:** 2026-09-18 Cite: https://marvinlabs.org/findings#F02 ## F03: Incremental patching on a trained adapter caused catastrophic forgetting {#F03} Continuing v3's adapter for 8 epochs on only 59 targeted examples dropped accuracy 7.3 pp on v3's own test set. Two targeted capabilities improved; the main target (clarify) got worse; four untouched capabilities collapsed to 25-56%. Mixing the same 59 examples into the full corpus and retraining from base (v5) recovered every collapsed capability and matched v3 exactly. - **Evidence:** V3: 709/790; V4: 651/790; V5: 709/790; V4 collapsed: Run agent: 0.25, Resume agent: 0.417, Dispatch code: 0.524, Calendar availability: 0.562; V5 same capabilities: 1.0 - **Measured on:** 2026-09-15 Cite: https://marvinlabs.org/findings#F03 ## F04: Held-out eval loss does not pick the best checkpoint {#F04} In every run with an eval set, eval loss bottomed early (step 75-125 of ~440 in most runs) and drifted up, while accuracy kept improving or held. The lowest-loss v9 checkpoint (step 175) scored 6.5 pp worse overall than the final. v12 (2 epochs) had the lowest final eval loss of the program and lower accuracy than v11, whose final eval loss was among the highest. v15 showed the 'better' loss curve and lost on accuracy. The eval targets carry partial arguments, which the program notes suggests rewards under-confident outputs. #### Evidence {#f04-evidence} - **V9 step175:** Eval loss: 0.1306; Accuracy: 0.8658 - **V9 final:** Eval loss: 0.1861; Accuracy: 0.9305 ##### Final eval loss {#f04-evidence-final-eval-loss} - **V11:** 0.3997 - **V12:** 0.2934 - **V13:** 0.3497 - **V15:** 0.3385 - **V17:** 0.4378 ##### Best eval step {#f04-evidence-best-eval-step} - **V9:** 175 - **V10:** 125 - **V11:** 75 - **V12:** 125 - **V13:** 100 - **V15:** 300 - **V16:** 100 - **V17:** 100 - **Accuracy 1131set:** V11: 0.9637; V12: 0.9558 - **Status:** [inferred] - **Measured on:** 2026-09-23 Cite: https://marvinlabs.org/findings#F04 ## F05: Accuracy plateaus well before the end, and the final checkpoint is as good as any {#F05} A held-out accuracy sweep of all 16 v9 checkpoints showed accuracy flat from step 200 of 405, with no mid-run peak worth picking. #### Evidence {#f05-evidence} ##### Held out 108 by step {#f05-evidence-held-out-108-by-step} - **25:** 86 - **50:** 94 - **75:** 96 - **100:** 86 - **125:** 95 - **150:** 94 - **175:** 96 - **200:** 100 - **225:** 99 - **250:** 100 - **275:** 97 - **300:** 99 - **325:** 99 - **350:** 100 - **375:** 100 - **400:** 99 - **405:** 99 - **Conflict:** The program notes say the plateau is '91.7-92.6% from step 200'; the sweep summary has step 275 at 97/108 (89.8%). The plateau claim holds loosely; the stated range is slightly too tight. Trust the summary. - **Measured on:** 2026-09-19 Cite: https://marvinlabs.org/findings#F05 ## F06: A third epoch is worth about one point {#F06} Same data and recipe at 2 epochs (v12) lost 0.8 pp overall and 4 held-out cases versus 3 epochs (v11), mostly on clarify. It saved about 4 hours of training, which buys nothing at inference. - **Evidence:** V11: Passed: 1090, Held out: 108/110; V12: Passed: 1081, Held out: 104/110; Runtime h: V11: 11.8, V12: 7.9 - **Measured on:** 2026-09-21 Cite: https://marvinlabs.org/findings#F06 ## F07: rsLoRA raised the headline and eroded the safety guards {#F07} rsLoRA at alpha = rank (scale 10.6) flatlined and was killed. At alpha 32 (scale 3.0) it posted the program's best overall accuracy, but every gained case was train-flagged; it lost 5 held-out cases and failed held-out guards the plain recipe passes, mostly cases with pasted or quoted content. Its checkpoint average did not recover them. - **Evidence:** V13 overall: 0.9752; V11 overall: 0.9637; V13 held out: 103/110; V11 held out: 108/110; V13 train flagged: 0.9794; Guards 174: V11: 169, 1 held-out breach, V13: 168, 3 held-out breaches (2 pasted/quoted content); A112 eval loss step50: 0.577; V11 eval loss step50: 0.33 - **Measured on:** 2026-09-22 Cite: https://marvinlabs.org/findings#F07 ## F08: LoRA dropout did not fit at rank 112 and did not improve generalization where it fit {#F08} Dropout 0.1 on top of v13 died three times at rank 112 (one Xid 31 fault, two OOMs). At equal steps (step 225) dropout raised overall accuracy 3.2 pp, all on train-flagged cases, and widened the train-to-held-out gap from 1.2 to 6.8 pp; it did improve guards (148 to 157 of 174). At rank 96 it trained clean but bought one held-out case over v13, cost 0.7 pp, and was vetoed on guards. - **Evidence:** Ck225 nodrop: Overall: 0.8382, Train: 0.8394, Held out: 91/110, Guards: 148/174, 10 held-out breaches; Ck225 drop10: Overall: 0.87, Train: 0.8766, Held out: 89/110, Guards: 157/174, 7 held-out breaches; V15: Overall: 0.9682, Held out: 104/110, Guards: 165/174, 4 held-out breaches, 3 pasted/quoted content - **Measured on:** 2026-09-23 Cite: https://marvinlabs.org/findings#F08 ## F09: Checkpoint averaging is noise on this recipe {#F09} An exact mean of v11's last seven checkpoints scored +0.4 pp on one bake and +0.0 on a re-bake, with identical held-out both times. The rsLoRA average scored 0.4 pp below its final checkpoint. - **Evidence:** V11 avg: 0.9691, 0.9637; V11: 0.9646, 0.9637; V13 avg: 0.9717; V13: 0.9761 - **Method note:** averaging done exactly by concatenating LoRA A/B along rank (r = n x r0, alpha fixed), with an extra 1/sqrt(n) on B for rsLoRA - **Measured on:** 2026-09-22 Cite: https://marvinlabs.org/findings#F09 ## F10: Data quantity beat data purity: a 25% disjoint holdout cost 8.5 pp {#F10} Holding a quarter of the gold set out of training (v16, 873 vs 1,167 examples, recipe unchanged) lost 8.5 pp on 82 cases neither model saw. The same experiment showed the plain recipe was not inflated by memorization: v11 scored 96.3% on those never-seen cases against a 96.4% headline. Recommendation recorded: hold back ~10%, not 25%. - **Evidence:** Fair 82: V11: 0.963, V16: 0.878; V11 trained v16 not 317: V11: 0.946, V16: 0.826; V16 overall 1148: 0.9016; V11 overall 1148: 0.9591 - **Measured on:** 2026-09-24 Cite: https://marvinlabs.org/findings#F10 ## F11: The test set was mostly training data, but the plain recipe still generalized {#F11} Through v7, ~79% of the 865-case test set was also in the training corpus and the held-out slice was almost all must-not-fire guards with no training target; the untuned 12B beat every fine-tune on that slice until v9. Once guards got targets (v8/v9) and the harness was fixed, fine-tunes passed the 12B on held-out. The later disjoint test (F10) then showed the plain recipe was not a memorization score. - **Evidence:** V7 865: Train flagged: 0.963, Held out: 0.871; Untuned 12b 865 held out: 0.941; V9 vs 12b held out 1125: 0.9167, 0.8889 - **Unverified parts:** the 79% overlap and the v7/12B slice numbers are from the ledger and could not be recomputed (train flags changed after 09-18) - **Status:** [unverified] - **Measured on:** 2026-09-19 Cite: https://marvinlabs.org/findings#F11 ## F12: Negative guards need training targets {#F12} Must-not-fire cases had no gold label, so nothing taught them. Authoring targets for 126 of them lifted the guard families 19-50 pp in v8, and they held in v9. A later 86-case adversarial guard batch cut held-out breaches in v17 relative to v11 over repeated passes. - **Evidence:** Guard family gain pp v8: 19-50; Not run pause guard: 0.67, 0.92, 0.96; Guards 267 three passes: V11: 250, V17: 256 - **Measured on:** 2026-09-25 Cite: https://marvinlabs.org/findings#F12 ## F13: Guard batches need counterweights or they teach over-refusal {#F13} On 127 never-seen cases the weakest group for both models was counterweights: quote something, then genuinely issue a command. The program notes concludes the real residual weakness is over-refusal, not over-action, and that a guard batch shipped without counterweights would have pushed further that way. - **Evidence:** Counterweights 24: V11: 0.833, V16: 0.667; Adversarial guards 62: V11: 0.855, V16: 0.903 - **Measured on:** 2026-09-24 Cite: https://marvinlabs.org/findings#F13 ## F14: Label conflicts cost more than missing data {#F14} Four new clarify labels on bare confirmation words contradicted older gold labels and dropped gate_approve from 100% to 70% in v8. Removing them restored it to 94% in v9 with no other change to that capability. - **Evidence:** Gate approve: 1.0, 0.7, 0.94 - **Measured on:** 2026-09-19 Cite: https://marvinlabs.org/findings#F14 ## F15: A single bake pass over a small set overstated a margin by about 2x {#F15} The one-pass guard bake that justified switching production to v17 scored it 257/267 with 1 held-out breach against v11's 247 with 4. Three fresh passes each, same conditions, gave v17 256 with 4 breaches and v11 250 with 5, identical every pass. The real margin is 6 cases, not 10, and both models still miss some pasted-text cases, including one the single pass had counted as a v17 pass. - **Evidence:** Single pass: V17: 257, 1 breach, V11: 247, 4 breaches; Three passes: V17: 256, 4 breaches (x3), V11: 250, 5 breaches (x3) - **Open question:** Why the single pass differed from four identical v11 passes (247 vs 250) at temperature 0 is not explained in the sources. - **Measured on:** 2026-09-25 Cite: https://marvinlabs.org/findings#F15 ## F16: Run-to-run noise differs by model; repeat bakes before comparing {#F16} Three all-caps passes: v11 landed within one case of itself (spread 0.09 pp); v9 (marvin-v1.2) swung 14 cases (1.18 pp). Sub-point deltas against a noisy model are meaningless without repeats. Per-capability slices with n of about 8 need intervals. - **Evidence:** V11 passes: 1145, 1144, 1145; V9 passes: 1098, 1112, 1112; Total: 1191; Example interval: Guard:open folder: Cases: 8, Accuracy: 0.875, Ci: 0.785, 0.931 - **Measured on:** 2026-09-22 Cite: https://marvinlabs.org/findings#F16 ## F17: The overall number can rank the worst router first; a guard veto is needed {#F17} v13 (97.6%) and v15 (96.8%) both beat v11 (96.4%) overall and both obey instructions hidden in pasted text more often. The 174-case guard bake (about 4 minutes) was the only check that separated them. The release rule was changed to: veto any held-out guard breach the incumbent does not have, then rank by held-out accuracy. - **Evidence:** Overall: V13: 0.9752, V15: 0.9682, V11: 0.9637; Held out guard breaches 174: V11: 1, V13: 3, V15: 4; Guard bake wall clock s: 220.6, 230.5 - **Measured on:** 2026-09-23 Cite: https://marvinlabs.org/findings#F17 ## F18: A harness channel bug scored every context case context-free for weeks {#F18} Bake scripts and both live-model test harnesses routed on one channel while each case's setup seeded context on another, so ~50 context-dependent cases were scored without context for every model. Fixing it moved v9's held-out from 86.8% to 91.7% on the same weights. - **Evidence:** V9 held out before fix: 0.8681; V9 held out after fix: 0.9167 - **Measured on:** 2026-09-19 Cite: https://marvinlabs.org/findings#F18 ## F19: Grammar-constrained decoding helps only when the enum matches what the router accepts {#F19} Forcing the JSON schema and capability enum at decode time first cost 0.7 pp: the enum was built from the visible capability docs and omitted one hidden capability, so all 8 of its cases were forced to clarify. With every accepted capability in the enum it was +0.1 pp and cut unparseable outputs from 3 to 1. - **Evidence:** First: 0.9573; Plain: 0.9637; Fixed: 0.9655; Unparseable: 3, 1 - **Measured on:** 2026-09-21 Cite: https://marvinlabs.org/findings#F19 ## F20: One-line capability docs earn their prompt tokens {#F20} Stripping the one-line capability descriptions (24% of the routing prompt, ~1,669 tokens) cost 2.9 pp. Guards were untouched; the losses were disambiguation between capabilities whose names do not explain themselves, and unparseable outputs rose from 3 to 8. - **Evidence:** With docs: 0.9637; Terse: 0.9355; Broke: 39; Fixed: 7; Guards 174 both: 169; Unparseable: 3, 8 - **Measured on:** 2026-09-22 Cite: https://marvinlabs.org/findings#F20 ## F21: Q8_0 beat Q4_0 by 3.2 pp on the fine-tuned router {#F21} Quantizing the same merged weights to Q4_0 cost 3.2 pp for about 15% speed. Q8_0 stayed for every release. - **Evidence:** Q8: 0.8936; Q4: 0.8613; S per case: Q8: 1.506, Q4: 1.278 - **Measured on:** 2026-09-16 Cite: https://marvinlabs.org/findings#F21 ## F22: Inference speed is set by model size, not training choices {#F22} Every rank-112 fine-tune ran 1.17-1.34 s per case on the same card; rank 64 was a few percent faster (time ratio 0.913 vs rank 112 in one head-to-head; the ledger says 4%); the untuned 12B was 2.1-2.7 s. A model that lands split across CPU and GPU (because other models are resident) slows to ~2.8 s per case or worse, so evict between bake halves. - **Evidence:** R112 s per case: 1.17-1.34; R64 vs r112 time ratio: 0.913; R64 vs r16 time ratio: 0.917; Untuned 12b: 2.124, 2.685; Split CPU GPU: 2.8 - **Measured on:** 2026-09-25 Cite: https://marvinlabs.org/findings#F22 ## F23: A 4B fine-tune beat the untuned 12B it replaced {#F23} On the 1,125-case set the untuned gemma4:12b-it-qat scored 82.1% (held-out 88.9%) at 2.7 s per case; the best 4B fine-tune scored 96.4% (held-out 96.3%) at 1.25 s. The first fine-tune (v1, 152 examples) had scored 4.9 pp below the 12B; the 490-example v2 was the first to beat it. - **Evidence:** Untuned 12b: Accuracy: 0.8213, Held out: 0.8889, S per case: 2.685; V11: Accuracy: 0.9636, Held out: 0.963, S per case: 1.245; V1 vs 12b 550: 0.8036, 0.8527; V2 vs 12b 550: 0.8964, 0.8527 - **Measured on:** 2026-09-21 Cite: https://marvinlabs.org/findings#F23 ## F24: Measure the memory ceiling with a calibrated probe instead of crashing into it {#F24} A one-step probe on the longest example over-estimates a real run's peak by ~4.7%. Scaling the probe by that factor predicted rank 96 + dropout at 6,793 MB; the real run peaked at 6,629 MB (2.5% error). Discovering the rank-112 dropout ceiling by training had cost three crashes and about 11 GPU-hours. - **Evidence:** Probe MB: 7127.6; Predicted MB: 6793; Actual MB: 6629; Error pct: 2.5 - **Measured on:** 2026-09-23 Cite: https://marvinlabs.org/findings#F24 ## F25: Deterministic pre-router steps remove whole miss classes, invisibly to the bake {#F25} v11's largest miss class was 14 short confirmations routed to clarify. A deterministic step before the router now resolves a bare yes or no against real pending state with no model call. Separately, regex shortcuts route some standard session phrasings before any model call. Bakes call the model directly, so production accuracy on those phrasings is better than the bake shows. - **Evidence:** Confirmation miss class: 14; Session positives 31: Routed by regex correctly: 10, Routed by regex wrongly: 0; Backstop covered gold cases dropped for v16: 58 - **Measured on:** 2026-09-24 Cite: https://marvinlabs.org/findings#F25 ## F26: Small models need the context window set explicitly {#F26} Ollama gave each model a VRAM-derived default context of 4096 tokens while the routing prompt was ~5,650. For one model Ollama silently trimmed the prompt to 2,050 tokens, cutting off the capability list at its front, with its truncated flag still 0; routing still returned plausible answers, so misses looked like flakiness. Sending num_ctx explicitly on every call and a test that bounds prompt tokens against it fixed it. - **Evidence:** Default num ctx: 4096; Routing prompt tokens: 5650; Trimmed to tokens: 2050; Num ctx now: 9216 - **Measured on:** 2026-09-07 Cite: https://marvinlabs.org/findings#F26 ## F27: New capabilities often need little or no training {#F27} v11 handled 25 session-capability paraphrases and discriminators at 88% having never seen those capabilities, and refused all 16 session-injection guard attempts. - **Evidence:** Session paraphrases 25: 0.88; Session injection guards 16: 1.0 - **Measured on:** 2026-09-24 Cite: https://marvinlabs.org/findings#F27 ## F28: Fine-tunes trade cautious misses for confident ones {#F28} v2 beat the untuned 12B by 4.3 pp, but a larger share of its remaining misses were confident picks of a forbidden capability rather than an honest clarify. v7 likewise fixed clarify but lost ground on context and must-not-route cases. - **Evidence:** Forbidden pick share of misses: Untuned 12b: 6/81, V1: 8/108, V2: 12/57; V7 context cases n31: 0.968, 0.871 - **Measured on:** 2026-09-18 Cite: https://marvinlabs.org/findings#F28 ## F29: AMD and NVIDIA need the same memory for the same QLoRA step {#F29} The identical 4-bit QLoRA probe at 7,177 tokens peaked at 5,721.3 MB (rank 16) and 6,406.2 MB (rank 64) on an RX 7900 XTX under ROCm 7.2.1 on Windows, and at 5,788.9 MB and 6,465.0 MB on an RTX 3060 Ti under CUDA on Linux: within 1.2%, AMD slightly lower. Memory before the step matched to within 0.5 MB. A VRAM budget measured on one vendor carries over to the other for this recipe. - **Evidence:** Peak MB r16: Rx 7900 xtx: 5721.3, Rtx 3060 ti: 5788.9; Peak MB r64: Rx 7900 xtx: 6406.2, Rtx 3060 ti: 6465.0; Pre step MB r64: Rx 7900 xtx: 3597.7, Rtx 3060 ti: 3597.3 - **Measured on:** 2026-10-15 Cite: https://marvinlabs.org/findings#F29 ## F30: LoRA dropout costs about 170 MB at the probe, not 350 {#F30} Turning lora_dropout from 0 to 0.1 raised the one-step peak by 174.5, 173.7, 166.8 and 168.6 MB at ranks 96, 112, 128 and 256 on the RX 7900 XTX. The cost is close to flat across rank. The earlier 350 MB figure on the VRAM page was an estimate from crash margins, not a measurement. - **Evidence:** Dropout cost MB by rank: 96: 174.5, 112: 173.7, 128: 166.8, 256: 168.6 - **Measured on:** 2026-10-15 Cite: https://marvinlabs.org/findings#F30 ## F31: Rank cost is linear; DoRA and bf16 add large fixed costs {#F31} On the RX 7900 XTX at 7,177 tokens, the peak rose from 5,721.3 MB at rank 16 to 9,338.3 MB at rank 256, about 15.1 MB per rank, which matches the 15 MB per rank measured earlier on the 8 GB NVIDIA card. DoRA added about 1.83 GB at ranks 64 and 112 and made the step about 38% slower. Loading the base model in bf16 instead of 4-bit added about 5.1 GB at every rank. Without the memory fixes, the step needed about 36 GB. #### Evidence {#f31-evidence} ##### Peak MB by rank 4bit {#f31-evidence-peak-mb-by-rank-4bit} - **16:** 5721.3 - **64:** 6406.2 - **96:** 6884.8 - **112:** 7194.5 - **128:** 7395.9 - **192:** 8383.4 - **256:** 9338.3 - **DoRA extra MB:** 64: 1834.5; 112: 1824.3 - **Bf16 extra MB:** 16: 5105.0; 64: 5103.3; 112: 5123.0; 256: 5104.9 - **No fixes peak MB:** 16: 35829.2; 112: 36581.3 ##### Peak MB by seq len r112 {#f31-evidence-peak-mb-by-seq-len-r112} - **2048:** 5585.8 - **4096:** 6207.9 - **7177:** 7194.5 - **8192:** 7486.0 - **12288:** 8917.2 - **16384:** 10523.6 - **Measured on:** 2026-10-15 Cite: https://marvinlabs.org/findings#F31 ## F32: A 20B mixture-of-experts model generates faster than dense 9B and 12B models {#F32} On an RX 7900 XTX under Ollama 0.30.10, gpt-oss:20b generated about 116-123 tokens per second, against 84 for qwen3.5:9b and 66-68 for gemma4:12b-it-qat, and processed prompts 1.4 to 2.5 times faster than either. It needs the most memory, 12.3 GB resident, and the slowest cold load at 14.7 s. For a card with room to hold it, size in parameters is a poor guide to speed. - **Evidence:** Gen tok per s short long: Gpt-oss:20b: 123.0, 116.2, Qwen3.5:9b: 84.9, 84.2, Gemma4:12b-it-qat: 68.4, 66.4; Prompt tok per s short long: Gpt-oss:20b: 3730.2, 4118.1, Qwen3.5:9b: 2588.9, 2704.7, Gemma4:12b-it-qat: 1526.1, 1661.9; Resident MB: Gpt-oss:20b: 12261, Qwen3.5:9b: 5642, Gemma4:12b-it-qat: 7601 - **Note:** The mixture-of-experts explanation (only a few billion parameters active per token) is the usual reading of gpt-oss's design; this bench measured speed only. - **Status:** [inferred] - **Measured on:** 2026-10-15 Cite: https://marvinlabs.org/findings#F32 ## Tried and closed {#tried-and-closed} Methods tried or considered, and why each is closed. | Method | Verdict | Finding | Flags | | --- | --- | --- | --- | | incremental patch on a trained adapter | forgetting; retrain from base | [F03](#F03) | | | Q4_0 quantization | -3.2 pp; Q8_0 stays | [F21](#F21) | | | rank 128 | OOM | [F02](#F02) | | | DoRA at rank 64 and 112 | OOM inside the step | [F02](#F02) | | | checkpoint selection by eval loss | picked a 6.5 pp worse checkpoint | [F04](#F04) | | | rank 64 on the enriched corpus | -1.4 pp, clarify -13 pp | [F01](#F01) | | | two epochs | -0.8 pp | [F06](#F06) | | | grammar-constrained decoding | +0.1 pp with the right enum; kept on | [F19](#F19) | | | checkpoint averaging | noise | [F09](#F09) | | | rsLoRA | higher headline, worse guards | [F07](#F07) | | | LoRA dropout | does not fit at r112; no generalization gain at r96 | [F08](#F08) | | | 25% disjoint holdout corpus | -8.5 pp on the fair slice | [F10](#F10) | | | terse prompt without capability docs | -2.9 pp | [F20](#F20) | | | decoding temperature as a lever | already 0 | | | | MoE upcycling / widening | considered, not tried: a 9B upcycled to 4 experts needs ~13.5 GB quantized for weights alone | | [inferred] | | two-model disagreement cascade | dropped: no untuned local model was left that beat the fine-tune on any slice | | | ## Untried {#untried} Ideas the program did not get to. - LR sweep (2e-4 linear never varied) - LoRA+ - NEFTune, higher weight decay - class-balanced loss - LoRA on embeddings or the output head (needs more VRAM) - a bigger base (gemma-3-12b, Qwen3.5-9B) on rented hardware - training on real logged corrections (none collected before the program closed)