MarvinLabs

Markdown: index.md · Data: baselines.json · Data: findings.json · Data: gotchas.json · Data: inference.json · Data: recipe.json · Data: versions.json · Data: vram.json

MarvinLabs is a public dataset of measured local-LLM fine-tuning and inference findings. It covers a 17-version LoRA fine-tune program of a 4B command router (unsloth/gemma-3-4b-it) on 8 GB consumer NVIDIA cards, in September 2026, plus cross-GPU memory probes of the same recipe on an AMD RX 7900 XTX (October 2026). Every number is measured and dated, and every page has a Markdown twin and a JSON file.

Headline findings

LoRA rank, not more data, moved the hardest capability

Three rounds of more data (v3, v4, v5) left the clarify capability at 56-60%. Raising rank from 16 to 64 on the same 757 examples lifted overall accuracy 4 pp and clarify to ~75%; rank 112 lifted clarify to ~85%. Dropping back to rank 64 on a later corpus cost clarify 13 pp. F01 in full (measured on 2026-09-20)

Rank 112 is the ceiling on an 8 GB card at ~7k-token examples

With chunked MLP, save_on_cpu and a bf16 lm_head, rank 112 peaks at ~6.65 GB in real runs; rank 128 and DoRA (at 64 and 112) OOM on the longest example. Rank 112 over 64 bought +0.5 pp on the 865 set, inside run-to-run noise there, but held clarify gains later. F02 in full (measured on 2026-09-18)

The overall number can rank the worst router first; a guard veto is needed

v13 (97.6%) and v15 (96.8%) both beat v11 (96.4%) overall and both obey instructions hidden in pasted text more often. The 174-case guard bake (about 4 minutes) was the only check that separated them. The release rule was changed to: veto any held-out guard breach the incumbent does not have, then rank by held-out accuracy. F17 in full (measured on 2026-09-23)

A 4B fine-tune beat the untuned 12B it replaced

On the 1,125-case set the untuned gemma4:12b-it-qat scored 82.1% (held-out 88.9%) at 2.7 s per case; the best 4B fine-tune scored 96.4% (held-out 96.3%) at 1.25 s. The first fine-tune (v1, 152 examples) had scored 4.9 pp below the 12B; the 490-example v2 was the first to beat it. F23 in full (measured on 2026-09-21)

AMD and NVIDIA need the same memory for the same QLoRA step

The identical 4-bit QLoRA probe at 7,177 tokens peaked at 5,721.3 MB (rank 16) and 6,406.2 MB (rank 64) on an RX 7900 XTX under ROCm 7.2.1 on Windows, and at 5,788.9 MB and 6,465.0 MB on an RTX 3060 Ti under CUDA on Linux: within 1.2%, AMD slightly lower. Memory before the step matched to within 0.5 MB. A VRAM budget measured on one vendor carries over to the other for this recipe. F29 in full (measured on 2026-10-15)

A single bake pass over a small set overstated a margin by about 2x

The one-pass guard bake that justified switching production to v17 scored it 257/267 with 1 held-out breach against v11's 247 with 4. Three fresh passes each, same conditions, gave v17 256 with 4 breaches and v11 250 with 5, identical every pass. The real margin is 6 cases, not 10, and both models still miss some pasted-text cases, including one the single pass had counted as a v17 pass. F15 in full (measured on 2026-09-25)

Pages

• Version ledger: The ledger of every trained version: config, training stats and measured results, plus the test-set eras. Markdown
• VRAM measurements: Every GPU memory measurement from the program: peak MB by rank, dropout, DoRA, bf16 and sequence length on an 8 GB NVIDIA card and a 24 GB AMD card, plus the probe calibration. Markdown
• Inference speed: Ollama load time, prompt speed, generation speed and resident VRAM for seven small models on an AMD RX 7900 XTX. Markdown
• Untuned baselines: The untuned model tournament and the untuned 12B scored on each test-set era. Markdown
• Findings: Every finding with its evidence and measured date, plus the methods tried and closed and the ideas left untried. Markdown
• Gotchas: Environment and tooling failures hit during the program, with symptom, cause, fix and versions. Markdown
• Training recipe: The shared training recipe: model, quantization, LoRA settings, hyperparameters, memory fixes, export path and release rule. Markdown
• About: method and scope: How the numbers were measured, what the terms mean, what is not published, and how to report a correction. Markdown

Dataset downloads

Each file is plain JSON. Records keep their original fields, including internal source references.

• baselines.json: Untuned (off-the-shelf) models, measured before and during the fine-tune program through Ollama. 'routing' = the production command-routing prompt (~5.6k...
• findings.json: Lessons from 17+ LoRA fine-tunes of gemma-3-4b-it as a command router on one 8 GB consumer GPU. Each claim is limited to this task, this base model and this...
• gotchas.json: Environment and tooling failures hit during the program, with symptom, cause, fix and versions. Versions are those recorded at the time; see recipe.json for...
• inference.json: Ollama inference speed for seven small models on one AMD Radeon RX 7900 XTX (24 GB, Windows 11, Ollama 0.30.10). Each model was unloaded first, then given...
• recipe.json: The training recipe shared by every fresh-from-base version (v1-v3, v5-v17). v4 differs only in continuing from v3's adapter at 8 epochs. Values come from the...
• versions.json: LoRA fine-tunes of unsloth/gemma-3-4b-it as a closed-set command router (one JSON tool call per utterance), trained and served on 8 GB NVIDIA consumer cards...
• vram.json: Every VRAM measurement from the program. The cross_gpu section adds one-step probes on an AMD RX 7900 XTX (24 GB) and a second NVIDIA card. Training GPU: one 8...

For agents

• llms.txt: index of every Markdown page and data file.
• llms-full.txt: every Markdown page in one file.
• sitemap.xml: all URLs.