# Training recipe

> The shared training recipe: model, quantization, LoRA settings, hyperparameters, memory fixes, export path and release rule.

Canonical: https://marvinlabs.org/recipe
Data: [recipe.json](https://marvinlabs.org/data/recipe.json)
HTML: https://marvinlabs.org/recipe

The training recipe shared by every fresh-from-base version (v1-v3, v5-v17). v4 differs only in continuing from v3's adapter at 8 epochs. Values come from the training scripts and the run headers printed in each training log.

## Task {#task}

- **Description:** Supervised LoRA fine-tune. Input is the production routing prompt (system + user, ~5.6k-6.3k tokens incl. capability docs and recent context); target is the assistant's single JSON capability call (13-128 tokens, median 26).
- **Loss:** response-only: prompt tokens masked; labels and logits computed only for the last response_len + 1 positions via logits_to_keep
- **Example length tokens:** Min: 6277; Max: 6447; Corpus: v1-v5 era; measured on 2026-09-14

### Longest example later corpora tokens {#task-longest-example-later-corpora-tokens}

| Corpus | Tokens | Measured on |
| --- | --- | --- |
| 2026-09-15 | 7037 | 2026-09-17 |
| 2026-09-18 unfiltered | 7288 | 2026-09-18 |
| 2026-09-18 filtered | 6931 | 2026-09-18 |
| 2026-09-19 filtered | 7068 | 2026-09-20 |
| 2026-09-21 | 7177 | 2026-09-23 |

- **Response span tokens:** Min: 13; Max: 128; Median: 26; measured on 2026-09-13

## Base model {#base-model}

- **HF repo:** unsloth/gemma-3-4b-it
- **Text layers:** 34
- **Vocab:** 262208
- **Hidden:** 2560
- **Measured on:** 2026-09-14

## Quantization for training {#quantization-for-training}

- **Method:** bitsandbytes 4-bit QLoRA
- **Load in 4bit:** yes
- **Bnb 4bit quant type:** nf4
- **Bnb 4bit use double quant:** yes
- **Bnb 4bit compute dtype:** bfloat16
- **Lm head:** cast from fp32 to bf16 after get_peft_model
- **Measured on:** 2026-09-18

## LoRA {#lora}

**Target modules**

- q_proj
- k_proj
- v_proj
- o_proj
- gate_proj
- up_proj
- down_proj

- **Rank:** Shipped: 112; Tried: 16, 64, 96, 112; Probed oom: 128; measured on 2026-09-25
- **Alpha:** equal to rank (scale 1.0) in every shipped version; rsLoRA runs used alpha 32 at r112 and 30 at r96
- **Dropout:** 0 (0.1 tried in v14/v15, closed)
- **RsLoRA:** off (tried v13-v15, closed)
- **DoRA:** off (OOM at r64 and r112)
- **Bias:** none
- **Measured on:** 2026-09-25

## Hyperparameters {#hyperparameters}

- **Epochs:** 3
- **Learning rate:** 0.0002
- **Lr scheduler:** linear
- **Warmup steps:** 5
- **Per device train batch size:** 1
- **Gradient accumulation steps:** 8
- **Effective batch:** 8
- **Optimizer:** paged_adamw_8bit
- **Weight decay:** 0.01
- **Seed:** 3407
- **Bf16:** yes
- **Max seq length:** 8192
- **Gradient checkpointing:** on, use_reentrant=False, via prepare_model_for_kbit_training
- **Attn implementation:** sdpa
- **Save and eval steps:** 25
- **Decoding temperature at inference:** 0
- **Measured on:** 2026-09-18

## Memory fixes {#memory-fixes}

| Fix | Why | Measured on |
| --- | --- | --- |
| chunked MLP: Gemma3MLP.forward monkey-patched to run gate/up/down over 2048-token chunks of the sequence | gate_proj(x) and up_proj(x) over ~6300 tokens must coexist for the gated product; that pair was the measured OOM point | 2026-09-14 |
| torch.autograd.graph.save_on_cpu(pin_memory=True) around trainer.train() | gradient-checkpoint boundary tensors for 34 layers stayed on GPU; memory climbed 4.5 to 6.99 GB in one forward, 6.39 GB with the offload | 2026-09-14 |
| lm_head cast fp32 to bf16 | fp32 lm_head was 2.68 GB and forced a mixed-precision matmul; pre-training VRAM fell 4502 MB to 3221 MB | 2026-09-14 |
| logits_to_keep = response_len + 1 | full [1, 6390, 262208] fp32 logits would be 6.7 GB vs 135 MB for the response span; did not fix the first OOM (that was the MLP) but removes the tensor whenever the forward reaches lm_head | 2026-09-14 |
| PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True | allocator setting only; bought 3 steps in the v14 dropout runs, not a fix | 2026-09-23 |
| pre-tokenize and mask by hand; no datasets.map() | map() crashed dill-pickling a closure over the Unsloth-patched tokenizer | 2026-09-14 |
| token_type_ids all zeros | plain transformers Gemma3 requires them during training (clear ValueError) | 2026-09-14 |

## Did not help {#did-not-help}

| Lever |
| --- |
| expandable_segments allocator tuning |
| paged_adamw_8bit (kept, but the first OOM was in forward, before any optimizer step) |
| non-reentrant gradient checkpointing alone |
| splitting layers across both GPUs with device_map=auto (pipeline split does not shrink one layer's activations) |
| attention-only LoRA targets (moved the OOM one line; the floor is the base MLP) |
| Liger Kernel fused GEGLU (fuses only the elementwise step after both projections exist) |
| flash-attn (attention was never the bottleneck; built from source after installing a CUDA 13.0 toolkit, then abandoned) |

## Why plain transformers peft not unsloth {#why-plain-transformers-peft-not-unsloth}

- **Decision:** train with transformers.AutoModelForCausalLM + BitsAndBytesConfig + peft.get_peft_model; Unsloth's FastLanguageModel loader dropped on 2026-09-14

**Reasons**

- Unsloth's patched Gemma 4 E2B MoE forward crashed whenever any part of the model sat off-GPU (embedding offloaded to CPU, index_select device mismatch); E2B needs 8-10 GB and does not fit one 8 GB card.
- Unsloth multi-GPU was not supported (PR #2074 closed unmerged); spanning both cards loaded cleanly but transformers' own Gemma 4 forward then hit a cuda:0/cuda:1 mismatch.
- Forcing device_map={'': 0} silently disabled Unsloth's 4-bit quantization ('Switching to 16bit LoRA') and OOMed.
- On gemma-3-4b-it, Unsloth's Gemma3 attention patch raised 'too many values to unpack' on real ~6300-token sequences (a 2048-token trial never reached that path).
- datasets.map() via Unsloth's train_on_responses_only crashed pickling the patched tokenizer.

- **Measured on:** 2026-09-14

## Libraries {#libraries}

### Versions {#libraries-versions}

- **Python:** 3.12.13
- **Torch:** 2.6.0+cu124
- **Transformers:** 5.5.0
- **Peft:** 0.20.0
- **Trl:** 0.24.0
- **Accelerate:** 1.15.0
- **Bitsandbytes:** 0.50.2
- **Datasets:** 4.3.0
- **Safetensors:** 0.8.0
- **Xformers:** 0.0.29.post3
- **Unsloth:** 2026.9.4
- **Unsloth zoo:** 2026.9.3
- **Liger kernel:** 0.8.2
- **Flash attn:** 2.8.3.post1 (built, unused)
- **Torchao:** 0.9.0 (pinned; not present in the venv on 2026-10-15)

- **Note:** Read from the training venv's installed packages on 2026-10-15. Install dates are 2026-09-13; no later upgrade is recorded, so these are taken to be the training versions. torch was upgraded in place from 2.5.1+cu121 to 2.6.0+cu124 on 2026-09-13.
- **Measured on:** 2026-10-15
- **Status:** [inferred]

## Driver and hardware {#driver-and-hardware}

- **Nvidia driver:** 580.178.04
- **Cuda toolkit installed:** 13.0 (toolkit only, for the flash-attn build)
- **Training GPU:** CUDA device 1 under CUDA_DEVICE_ORDER=PCI_BUS_ID, which is an RTX 3070 Ti 8 GB on this box; torch reports 7.65 GiB total capacity
- **Other GPU:** RTX 3060 Ti 8 GB. Not used for training. The live Ollama server shared the cards and repeatedly reloaded models onto the training GPU (see gotchas).
- **Note:** GPU model names read from nvidia-smi on 2026-10-15; no September log names the card. The program notes confirms 'two 8GB GPUs' and ~7.65 GB usable.
- **Measured on:** 2026-10-15
- **Status:** [inferred]

## Runtime per run {#runtime-per-run}

- **Rank112 full corpus hours:** Value: 11.8, 12.5; Runs: v11 (1,167 ex, 438 steps), v17 (1,189 ex, 447 steps); measured on 2026-09-25
- **Rank112 two epochs hours:** Value: 7.9; measured on 2026-09-21
- **Seconds per optimizer step r16:** Value: 81; measured on 2026-09-15
- **Note:** HF Trainer's train_runtime covers only the last resumed segment when a run was killed and resumed (v2, v5).

## Export path {#export-path}

**Steps**

- merge: load base in bf16 on CPU, PeftModel.from_pretrained, merge_and_unload, save safetensors (merge_adapter_generic.py)
- tensor-name fix: drop the 439 vision/projector tensors and rename the 444 text tensors from model.language_model.* to language_model.* (fix_tensor_names_generic.py; 883 tensors total)
- llama.cpp convert_hf_to_gguf.py --outtype f16
- llama.cpp llama-quantize ... Q8_0
- ollama create from a Modelfile (below)

- **Tensor fix reason:** not stated in the sources; the merged checkpoint carries Gemma 3's vision tower and a nested text-model prefix, and only the text tensors are kept
- **Llama cpp checkout:** commit ad6c668, committed 2026-09-14 (checked 2026-10-15)
- **GGUF sizes GB:** F16: 7.78; Q8 0: 4.14; Q4 0: 2.37
- **Verification lesson:** a sed substitution once built the v3 Ollama tag from v2's GGUF; caught by comparing model-layer sha256 and Ollama model id before baking
- **Measured on:** 2026-09-25
- **Status:** [inferred]

## Ollama modelfile {#ollama-modelfile}

**Text**

```
FROM <path-to>/gemma3-4b-router-<version>.Q8_0.gguf
TEMPLATE """{{- range $i, $_ := .Messages }}
{{- $last := eq (len (slice $.Messages $i)) 1 }}
{{- if or (eq .Role "user") (eq .Role "system") }}<start_of_turn>user
{{ .Content }}<end_of_turn>
{{ if $last }}<start_of_turn>model
{{ end }}
{{- else if eq .Role "assistant" }}<start_of_turn>model
{{ .Content }}{{ if not $last }}<end_of_turn>
{{ end }}
{{- end }}
{{- end }}"""
PARAMETER stop <end_of_turn>
PARAMETER temperature 1
PARAMETER top_k 64
PARAMETER top_p 0.95
PARAMETER num_ctx 8192
```

- **Notes:** system and user turns both render as a Gemma user turn, the router overrides temperature to 0 per request, the client sends num_ctx 9216 per request in production (code default since 2026-09-19); the Modelfile's 8192 is a fallback
- **Measured on:** 2026-09-25

## Q8 0 vs q4 0 {#q8-0-vs-q4-0}

- **Model:** v5 (rank 16)
- **Test set:** 865
- **Q8 0:** Passed: 773; Accuracy: 0.8936; S per case: 1.506
- **Q4 0:** Passed: 745; Accuracy: 0.8613; S per case: 1.278
- **Delta pp:** -3.23
- **Speed ratio q4 over q8:** 0.849
- **Verdict:** Q8_0 stays: -3.2 pp is not worth ~15% speed.
- **Measured on:** 2026-09-16

## Release rule {#release-rule}

- **Rule:** veto any model that fails a held-out guard an incumbent passes; rank the rest by held-out accuracy; break ties within 0.5 pp by overall accuracy. For v17 the scripted rule was: beat the incumbent on the fair slice and add no new held-out guard breach.
- **Why:** the earlier 'highest overall accuracy wins' rule picked v13 (rsLoRA) on 22 Sep and had to be overruled
- **Measured on:** 2026-09-25