MarvinLabs

Findings

Markdown: findings.md · Data: findings.json

Lessons from 17+ LoRA fine-tunes of gemma-3-4b-it as a command router on one 8 GB consumer GPU. Each claim is limited to this task, this base model and this recipe. Accuracy numbers are passed / runnable cases; see versions.json for definitions.

Findings

F01: LoRA rank, not more data, moved the hardest capability

Three rounds of more data (v3, v4, v5) left the clarify capability at 56-60%. Raising rank from 16 to 64 on the same 757 examples lifted overall accuracy 4 pp and clarify to ~75%; rank 112 lifted clarify to ~85%. Dropping back to rank 64 on a later corpus cost clarify 13 pp.

Evidence
Overall r16 vs r64: 0.8902, 0.9306; Clarify by rank 865set: R16: 0.521, R64: 0.755, R112: 0.851; V10 r64 vs v9 r112 clarify: 0.73, 0.86; V10 vs v9 overall 1125set: 0.9164, 0.9307
Measured on

Cite: https://marvinlabs.org/findings#F01

F02: Rank 112 is the ceiling on an 8 GB card at ~7k-token examples

With chunked MLP, save_on_cpu and a bf16 lm_head, rank 112 peaks at ~6.65 GB in real runs; rank 128 and DoRA (at 64 and 112) OOM on the longest example. Rank 112 over 64 bought +0.5 pp on the 865 set, inside run-to-run noise there, but held clarify gains later.

Evidence
Real peak r112 MB: 6643-6658; Probe r128: OOM; Probe DoRA r64: OOM; Probe DoRA r112: OOM; V7 vs v6: 0.9434, 0.9387
Measured on

Cite: https://marvinlabs.org/findings#F02

F03: Incremental patching on a trained adapter caused catastrophic forgetting

Continuing v3's adapter for 8 epochs on only 59 targeted examples dropped accuracy 7.3 pp on v3's own test set. Two targeted capabilities improved; the main target (clarify) got worse; four untouched capabilities collapsed to 25-56%. Mixing the same 59 examples into the full corpus and retraining from base (v5) recovered every collapsed capability and matched v3 exactly.

Evidence
V3: 709/790; V4: 651/790; V5: 709/790; V4 collapsed: Run agent: 0.25, Resume agent: 0.417, Dispatch code: 0.524, Calendar availability: 0.562; V5 same capabilities: 1.0
Measured on

Cite: https://marvinlabs.org/findings#F03

F04: Held-out eval loss does not pick the best checkpoint

In every run with an eval set, eval loss bottomed early (step 75-125 of ~440 in most runs) and drifted up, while accuracy kept improving or held. The lowest-loss v9 checkpoint (step 175) scored 6.5 pp worse overall than the final. v12 (2 epochs) had the lowest final eval loss of the program and lower accuracy than v11, whose final eval loss was among the highest. v15 showed the 'better' loss curve and lost on accuracy. The eval targets carry partial arguments, which the program notes suggests rewards under-confident outputs.

Evidence

V9 step175
Eval loss: 0.1306; Accuracy: 0.8658
V9 final
Eval loss: 0.1861; Accuracy: 0.9305
Final eval loss
V11
0.3997
V12
0.2934
V13
0.3497
V15
0.3385
V17
0.4378
Best eval step
V9
175
V10
125
V11
75
V12
125
V13
100
V15
300
V16
100
V17
100
Accuracy 1131set
V11: 0.9637; V12: 0.9558
Status
inferred
Measured on

Cite: https://marvinlabs.org/findings#F04

F05: Accuracy plateaus well before the end, and the final checkpoint is as good as any

A held-out accuracy sweep of all 16 v9 checkpoints showed accuracy flat from step 200 of 405, with no mid-run peak worth picking.

Evidence

Held out 108 by step
25
86
50
94
75
96
100
86
125
95
150
94
175
96
200
100
225
99
250
100
275
97
300
99
325
99
350
100
375
100
400
99
405
99
Conflict
The program notes say the plateau is '91.7-92.6% from step 200'; the sweep summary has step 275 at 97/108 (89.8%). The plateau claim holds loosely; the stated range is slightly too tight. Trust the summary.
Measured on

Cite: https://marvinlabs.org/findings#F05

F06: A third epoch is worth about one point

Same data and recipe at 2 epochs (v12) lost 0.8 pp overall and 4 held-out cases versus 3 epochs (v11), mostly on clarify. It saved about 4 hours of training, which buys nothing at inference.

Evidence
V11: Passed: 1090, Held out: 108/110; V12: Passed: 1081, Held out: 104/110; Runtime h: V11: 11.8, V12: 7.9
Measured on

Cite: https://marvinlabs.org/findings#F06

F07: rsLoRA raised the headline and eroded the safety guards

rsLoRA at alpha = rank (scale 10.6) flatlined and was killed. At alpha 32 (scale 3.0) it posted the program's best overall accuracy, but every gained case was train-flagged; it lost 5 held-out cases and failed held-out guards the plain recipe passes, mostly cases with pasted or quoted content. Its checkpoint average did not recover them.

Evidence
V13 overall: 0.9752; V11 overall: 0.9637; V13 held out: 103/110; V11 held out: 108/110; V13 train flagged: 0.9794; Guards 174: V11: 169, 1 held-out breach, V13: 168, 3 held-out breaches (2 pasted/quoted content); A112 eval loss step50: 0.577; V11 eval loss step50: 0.33
Measured on

Cite: https://marvinlabs.org/findings#F07

F08: LoRA dropout did not fit at rank 112 and did not improve generalization where it fit

Dropout 0.1 on top of v13 died three times at rank 112 (one Xid 31 fault, two OOMs). At equal steps (step 225) dropout raised overall accuracy 3.2 pp, all on train-flagged cases, and widened the train-to-held-out gap from 1.2 to 6.8 pp; it did improve guards (148 to 157 of 174). At rank 96 it trained clean but bought one held-out case over v13, cost 0.7 pp, and was vetoed on guards.

Evidence
Ck225 nodrop: Overall: 0.8382, Train: 0.8394, Held out: 91/110, Guards: 148/174, 10 held-out breaches; Ck225 drop10: Overall: 0.87, Train: 0.8766, Held out: 89/110, Guards: 157/174, 7 held-out breaches; V15: Overall: 0.9682, Held out: 104/110, Guards: 165/174, 4 held-out breaches, 3 pasted/quoted content
Measured on

Cite: https://marvinlabs.org/findings#F08

F09: Checkpoint averaging is noise on this recipe

An exact mean of v11's last seven checkpoints scored +0.4 pp on one bake and +0.0 on a re-bake, with identical held-out both times. The rsLoRA average scored 0.4 pp below its final checkpoint.

Evidence
V11 avg: 0.9691, 0.9637; V11: 0.9646, 0.9637; V13 avg: 0.9717; V13: 0.9761
Method note
averaging done exactly by concatenating LoRA A/B along rank (r = n x r0, alpha fixed), with an extra 1/sqrt(n) on B for rsLoRA
Measured on

Cite: https://marvinlabs.org/findings#F09

F10: Data quantity beat data purity: a 25% disjoint holdout cost 8.5 pp

Holding a quarter of the gold set out of training (v16, 873 vs 1,167 examples, recipe unchanged) lost 8.5 pp on 82 cases neither model saw. The same experiment showed the plain recipe was not inflated by memorization: v11 scored 96.3% on those never-seen cases against a 96.4% headline. Recommendation recorded: hold back ~10%, not 25%.

Evidence
Fair 82: V11: 0.963, V16: 0.878; V11 trained v16 not 317: V11: 0.946, V16: 0.826; V16 overall 1148: 0.9016; V11 overall 1148: 0.9591
Measured on

Cite: https://marvinlabs.org/findings#F10

F11: The test set was mostly training data, but the plain recipe still generalized

Through v7, ~79% of the 865-case test set was also in the training corpus and the held-out slice was almost all must-not-fire guards with no training target; the untuned 12B beat every fine-tune on that slice until v9. Once guards got targets (v8/v9) and the harness was fixed, fine-tunes passed the 12B on held-out. The later disjoint test (F10) then showed the plain recipe was not a memorization score.

Evidence
V7 865: Train flagged: 0.963, Held out: 0.871; Untuned 12b 865 held out: 0.941; V9 vs 12b held out 1125: 0.9167, 0.8889
Unverified parts
the 79% overlap and the v7/12B slice numbers are from the ledger and could not be recomputed (train flags changed after 09-18)
Status
unverified
Measured on

Cite: https://marvinlabs.org/findings#F11

F12: Negative guards need training targets

Must-not-fire cases had no gold label, so nothing taught them. Authoring targets for 126 of them lifted the guard families 19-50 pp in v8, and they held in v9. A later 86-case adversarial guard batch cut held-out breaches in v17 relative to v11 over repeated passes.

Evidence
Guard family gain pp v8: 19-50; Not run pause guard: 0.67, 0.92, 0.96; Guards 267 three passes: V11: 250, V17: 256
Measured on

Cite: https://marvinlabs.org/findings#F12

F13: Guard batches need counterweights or they teach over-refusal

On 127 never-seen cases the weakest group for both models was counterweights: quote something, then genuinely issue a command. The program notes concludes the real residual weakness is over-refusal, not over-action, and that a guard batch shipped without counterweights would have pushed further that way.

Evidence
Counterweights 24: V11: 0.833, V16: 0.667; Adversarial guards 62: V11: 0.855, V16: 0.903
Measured on

Cite: https://marvinlabs.org/findings#F13

F14: Label conflicts cost more than missing data

Four new clarify labels on bare confirmation words contradicted older gold labels and dropped gate_approve from 100% to 70% in v8. Removing them restored it to 94% in v9 with no other change to that capability.

Evidence
Gate approve: 1.0, 0.7, 0.94
Measured on

Cite: https://marvinlabs.org/findings#F14

F15: A single bake pass over a small set overstated a margin by about 2x

The one-pass guard bake that justified switching production to v17 scored it 257/267 with 1 held-out breach against v11's 247 with 4. Three fresh passes each, same conditions, gave v17 256 with 4 breaches and v11 250 with 5, identical every pass. The real margin is 6 cases, not 10, and both models still miss some pasted-text cases, including one the single pass had counted as a v17 pass.

Evidence
Single pass: V17: 257, 1 breach, V11: 247, 4 breaches; Three passes: V17: 256, 4 breaches (x3), V11: 250, 5 breaches (x3)
Open question
Why the single pass differed from four identical v11 passes (247 vs 250) at temperature 0 is not explained in the sources.
Measured on

Cite: https://marvinlabs.org/findings#F15

F16: Run-to-run noise differs by model; repeat bakes before comparing

Three all-caps passes: v11 landed within one case of itself (spread 0.09 pp); v9 (marvin-v1.2) swung 14 cases (1.18 pp). Sub-point deltas against a noisy model are meaningless without repeats. Per-capability slices with n of about 8 need intervals.

Evidence
V11 passes: 1145, 1144, 1145; V9 passes: 1098, 1112, 1112; Total: 1191; Example interval: Guard:open folder: Cases: 8, Accuracy: 0.875, Ci: 0.785, 0.931
Measured on

Cite: https://marvinlabs.org/findings#F16

F17: The overall number can rank the worst router first; a guard veto is needed

v13 (97.6%) and v15 (96.8%) both beat v11 (96.4%) overall and both obey instructions hidden in pasted text more often. The 174-case guard bake (about 4 minutes) was the only check that separated them. The release rule was changed to: veto any held-out guard breach the incumbent does not have, then rank by held-out accuracy.

Evidence
Overall: V13: 0.9752, V15: 0.9682, V11: 0.9637; Held out guard breaches 174: V11: 1, V13: 3, V15: 4; Guard bake wall clock s: 220.6, 230.5
Measured on

Cite: https://marvinlabs.org/findings#F17

F18: A harness channel bug scored every context case context-free for weeks

Bake scripts and both live-model test harnesses routed on one channel while each case's setup seeded context on another, so ~50 context-dependent cases were scored without context for every model. Fixing it moved v9's held-out from 86.8% to 91.7% on the same weights.

Evidence
V9 held out before fix: 0.8681; V9 held out after fix: 0.9167
Measured on

Cite: https://marvinlabs.org/findings#F18

F19: Grammar-constrained decoding helps only when the enum matches what the router accepts

Forcing the JSON schema and capability enum at decode time first cost 0.7 pp: the enum was built from the visible capability docs and omitted one hidden capability, so all 8 of its cases were forced to clarify. With every accepted capability in the enum it was +0.1 pp and cut unparseable outputs from 3 to 1.

Evidence
First: 0.9573; Plain: 0.9637; Fixed: 0.9655; Unparseable: 3, 1
Measured on

Cite: https://marvinlabs.org/findings#F19

F20: One-line capability docs earn their prompt tokens

Stripping the one-line capability descriptions (24% of the routing prompt, ~1,669 tokens) cost 2.9 pp. Guards were untouched; the losses were disambiguation between capabilities whose names do not explain themselves, and unparseable outputs rose from 3 to 8.

Evidence
With docs: 0.9637; Terse: 0.9355; Broke: 39; Fixed: 7; Guards 174 both: 169; Unparseable: 3, 8
Measured on

Cite: https://marvinlabs.org/findings#F20

F21: Q8_0 beat Q4_0 by 3.2 pp on the fine-tuned router

Quantizing the same merged weights to Q4_0 cost 3.2 pp for about 15% speed. Q8_0 stayed for every release.

Evidence
Q8: 0.8936; Q4: 0.8613; S per case: Q8: 1.506, Q4: 1.278
Measured on

Cite: https://marvinlabs.org/findings#F21

F22: Inference speed is set by model size, not training choices

Every rank-112 fine-tune ran 1.17-1.34 s per case on the same card; rank 64 was a few percent faster (time ratio 0.913 vs rank 112 in one head-to-head; the ledger says 4%); the untuned 12B was 2.1-2.7 s. A model that lands split across CPU and GPU (because other models are resident) slows to ~2.8 s per case or worse, so evict between bake halves.

Evidence
R112 s per case: 1.17-1.34; R64 vs r112 time ratio: 0.913; R64 vs r16 time ratio: 0.917; Untuned 12b: 2.124, 2.685; Split CPU GPU: 2.8
Measured on

Cite: https://marvinlabs.org/findings#F22

F23: A 4B fine-tune beat the untuned 12B it replaced

On the 1,125-case set the untuned gemma4:12b-it-qat scored 82.1% (held-out 88.9%) at 2.7 s per case; the best 4B fine-tune scored 96.4% (held-out 96.3%) at 1.25 s. The first fine-tune (v1, 152 examples) had scored 4.9 pp below the 12B; the 490-example v2 was the first to beat it.

Evidence
Untuned 12b: Accuracy: 0.8213, Held out: 0.8889, S per case: 2.685; V11: Accuracy: 0.9636, Held out: 0.963, S per case: 1.245; V1 vs 12b 550: 0.8036, 0.8527; V2 vs 12b 550: 0.8964, 0.8527
Measured on

Cite: https://marvinlabs.org/findings#F23

F24: Measure the memory ceiling with a calibrated probe instead of crashing into it

A one-step probe on the longest example over-estimates a real run's peak by ~4.7%. Scaling the probe by that factor predicted rank 96 + dropout at 6,793 MB; the real run peaked at 6,629 MB (2.5% error). Discovering the rank-112 dropout ceiling by training had cost three crashes and about 11 GPU-hours.

Evidence
Probe MB: 7127.6; Predicted MB: 6793; Actual MB: 6629; Error pct: 2.5
Measured on

Cite: https://marvinlabs.org/findings#F24

F25: Deterministic pre-router steps remove whole miss classes, invisibly to the bake

v11's largest miss class was 14 short confirmations routed to clarify. A deterministic step before the router now resolves a bare yes or no against real pending state with no model call. Separately, regex shortcuts route some standard session phrasings before any model call. Bakes call the model directly, so production accuracy on those phrasings is better than the bake shows.

Evidence
Confirmation miss class: 14; Session positives 31: Routed by regex correctly: 10, Routed by regex wrongly: 0; Backstop covered gold cases dropped for v16: 58
Measured on

Cite: https://marvinlabs.org/findings#F25

F26: Small models need the context window set explicitly

Ollama gave each model a VRAM-derived default context of 4096 tokens while the routing prompt was ~5,650. For one model Ollama silently trimmed the prompt to 2,050 tokens, cutting off the capability list at its front, with its truncated flag still 0; routing still returned plausible answers, so misses looked like flakiness. Sending num_ctx explicitly on every call and a test that bounds prompt tokens against it fixed it.

Evidence
Default num ctx: 4096; Routing prompt tokens: 5650; Trimmed to tokens: 2050; Num ctx now: 9216
Measured on

Cite: https://marvinlabs.org/findings#F26

F27: New capabilities often need little or no training

v11 handled 25 session-capability paraphrases and discriminators at 88% having never seen those capabilities, and refused all 16 session-injection guard attempts.

Evidence
Session paraphrases 25: 0.88; Session injection guards 16: 1.0
Measured on

Cite: https://marvinlabs.org/findings#F27

F28: Fine-tunes trade cautious misses for confident ones

v2 beat the untuned 12B by 4.3 pp, but a larger share of its remaining misses were confident picks of a forbidden capability rather than an honest clarify. v7 likewise fixed clarify but lost ground on context and must-not-route cases.

Evidence
Forbidden pick share of misses: Untuned 12b: 6/81, V1: 8/108, V2: 12/57; V7 context cases n31: 0.968, 0.871
Measured on

Cite: https://marvinlabs.org/findings#F28

F29: AMD and NVIDIA need the same memory for the same QLoRA step

The identical 4-bit QLoRA probe at 7,177 tokens peaked at 5,721.3 MB (rank 16) and 6,406.2 MB (rank 64) on an RX 7900 XTX under ROCm 7.2.1 on Windows, and at 5,788.9 MB and 6,465.0 MB on an RTX 3060 Ti under CUDA on Linux: within 1.2%, AMD slightly lower. Memory before the step matched to within 0.5 MB. A VRAM budget measured on one vendor carries over to the other for this recipe.

Evidence
Peak MB r16: Rx 7900 xtx: 5721.3, Rtx 3060 ti: 5788.9; Peak MB r64: Rx 7900 xtx: 6406.2, Rtx 3060 ti: 6465.0; Pre step MB r64: Rx 7900 xtx: 3597.7, Rtx 3060 ti: 3597.3
Measured on

Cite: https://marvinlabs.org/findings#F29

F30: LoRA dropout costs about 170 MB at the probe, not 350

Turning lora_dropout from 0 to 0.1 raised the one-step peak by 174.5, 173.7, 166.8 and 168.6 MB at ranks 96, 112, 128 and 256 on the RX 7900 XTX. The cost is close to flat across rank. The earlier 350 MB figure on the VRAM page was an estimate from crash margins, not a measurement.

Evidence
Dropout cost MB by rank: 96: 174.5, 112: 173.7, 128: 166.8, 256: 168.6
Measured on

Cite: https://marvinlabs.org/findings#F30

F31: Rank cost is linear; DoRA and bf16 add large fixed costs

On the RX 7900 XTX at 7,177 tokens, the peak rose from 5,721.3 MB at rank 16 to 9,338.3 MB at rank 256, about 15.1 MB per rank, which matches the 15 MB per rank measured earlier on the 8 GB NVIDIA card. DoRA added about 1.83 GB at ranks 64 and 112 and made the step about 38% slower. Loading the base model in bf16 instead of 4-bit added about 5.1 GB at every rank. Without the memory fixes, the step needed about 36 GB.

Evidence

Peak MB by rank 4bit
16
5721.3
64
6406.2
96
6884.8
112
7194.5
128
7395.9
192
8383.4
256
9338.3
DoRA extra MB
64: 1834.5; 112: 1824.3
Bf16 extra MB
16: 5105.0; 64: 5103.3; 112: 5123.0; 256: 5104.9
No fixes peak MB
16: 35829.2; 112: 36581.3
Peak MB by seq len r112
2048
5585.8
4096
6207.9
7177
7194.5
8192
7486.0
12288
8917.2
16384
10523.6
Measured on

Cite: https://marvinlabs.org/findings#F31

F32: A 20B mixture-of-experts model generates faster than dense 9B and 12B models

On an RX 7900 XTX under Ollama 0.30.10, gpt-oss:20b generated about 116-123 tokens per second, against 84 for qwen3.5:9b and 66-68 for gemma4:12b-it-qat, and processed prompts 1.4 to 2.5 times faster than either. It needs the most memory, 12.3 GB resident, and the slowest cold load at 14.7 s. For a card with room to hold it, size in parameters is a poor guide to speed.

Evidence
Gen tok per s short long: Gpt-oss:20b: 123.0, 116.2, Qwen3.5:9b: 84.9, 84.2, Gemma4:12b-it-qat: 68.4, 66.4; Prompt tok per s short long: Gpt-oss:20b: 3730.2, 4118.1, Qwen3.5:9b: 2588.9, 2704.7, Gemma4:12b-it-qat: 1526.1, 1661.9; Resident MB: Gpt-oss:20b: 12261, Qwen3.5:9b: 5642, Gemma4:12b-it-qat: 7601
Note
The mixture-of-experts explanation (only a few billion parameters active per token) is the usual reading of gpt-oss's design; this bench measured speed only.
Status
inferred
Measured on

Cite: https://marvinlabs.org/findings#F32

Tried and closed

Methods tried or considered, and why each is closed.

MethodVerdictFindingFlags
incremental patch on a trained adapterforgetting; retrain from baseF03
Q4_0 quantization-3.2 pp; Q8_0 staysF21
rank 128OOMF02
DoRA at rank 64 and 112OOM inside the stepF02
checkpoint selection by eval losspicked a 6.5 pp worse checkpointF04
rank 64 on the enriched corpus-1.4 pp, clarify -13 ppF01
two epochs-0.8 ppF06
grammar-constrained decoding+0.1 pp with the right enum; kept onF19
checkpoint averagingnoiseF09
rsLoRAhigher headline, worse guardsF07
LoRA dropoutdoes not fit at r112; no generalization gain at r96F08
25% disjoint holdout corpus-8.5 pp on the fair sliceF10
terse prompt without capability docs-2.9 ppF20
decoding temperature as a leveralready 0
MoE upcycling / wideningconsidered, not tried: a 9B upcycled to 4 experts needs ~13.5 GB quantized for weights aloneinferred
two-model disagreement cascadedropped: no untuned local model was left that beat the fine-tune on any slice

Untried

Ideas the program did not get to.