# VRAM measurements

> Every GPU memory measurement from the program: peak MB by rank, dropout, DoRA, bf16 and sequence length on an 8 GB NVIDIA card and a 24 GB AMD card, plus the probe calibration.

Canonical: https://marvinlabs.org/vram
Data: [vram.json](https://marvinlabs.org/data/vram.json)
HTML: https://marvinlabs.org/vram

Every VRAM measurement from the program. The cross_gpu section adds one-step probes on an AMD RX 7900 XTX (24 GB) and a second NVIDIA card. Training GPU: one 8 GB NVIDIA consumer card (an RTX 3070 Ti by today's nvidia-smi; torch reports 7.65 GiB total capacity). 'mode' is probe (one forward+backward on the single longest example, no optimizer step) or real (a full Trainer run; peak = torch.cuda.max_memory_allocated at the end). All training configs are 4-bit nf4 QLoRA on unsloth/gemma-3-4b-it with the recipe.json memory fixes unless noted.

## GPU {#gpu}

- **Model:** NVIDIA GeForce RTX 3070 Ti
- **Total MiB:** 8192
- **Usable GiB torch:** 7.65
- **Second card:** NVIDIA GeForce RTX 3060 Ti, 8192 MiB
- **Driver:** 580.178.04
- **Note:** Card names read on 2026-10-15 and mapped through CUDA_DEVICE_ORDER=PCI_BUS_ID, CUDA_VISIBLE_DEVICES=1 as set in every training script (the probe scripts document the same pin). This holds as long as neither card changed PCIe slot since September; the owner confirms both cards were in the box then. The 7.65 GiB figure is printed in every OOM message.
- **Measured on:** 2026-10-15
- **Status:** [inferred]

## Measurements {#measurements}

Mode is real (a training run), probe (one forward and backward pass on the longest example) or load (model load only). Peak is the highest GPU memory seen. OOM means the run ran out of memory.

| ID | Mode | Model | Rank | rsLoRA | Alpha | Dropout | DoRA | Seq length (tokens) | Peak | Status | Config, result and notes | Measured on |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| `e2b-contended-load` | load | unsloth/gemma-4-E2B-it |  |  |  |  |  |  |  |  | Config: Unsloth load_in_4bit, ~3.6 GB free (live models resident) \| Result: loaded, then forward crashed (embedding offloaded to CPU) | 2026-09-13 |
| `e2b-two-gpu-split` | load | unsloth/gemma-4-E2B-it |  |  |  |  |  |  | 4.38 GB on GPU0 + 3.17 GB on GPU1, zero CPU offload |  | Config: Unsloth 4-bit, both GPUs visible \| Result: forward crashed on a cross-device tensor in transformers' Gemma 4 forward | 2026-09-13 |
| `e2b-lora-params` | load | unsloth/gemma-4-E2B-it |  |  |  |  |  |  |  |  | Result: LoRA 25.3M trainable of 5.15B total (0.49%); vendor guide says E2B LoRA needs 8-10 GB | 2026-09-13 |
| `g3-trial-load` | load | unsloth/gemma-3-4b-it |  |  |  |  |  |  | 4.39 GB | [ok] | Config: Unsloth load_in_4bit, single GPU \| Result: fits, zero CPU offload | 2026-09-13 |
| `g3-trial-step` | probe | unsloth/gemma-3-4b-it |  |  |  |  |  |  | 5.59 GB | [ok] | Config: one forward+backward, example truncated to 2048 tokens | 2026-09-13 |
| `first-real-step-oom` | real |  |  |  |  |  |  |  | OOM | [oom] | Config: plain transformers+peft, r16, full ~6300-token examples, before the memory fixes \| Result: OOM: 7.39 GB used, 256 MB free, tried to allocate 248 MB | 2026-09-13 |
| `sdpa-forward-probe` | probe |  |  |  |  |  |  |  | 7.16 GB | [ok] | Config: bare no-grad forward, mem_efficient SDPA forced, 6390-token example \| Note: proved attention was not the bottleneck | 2026-09-13 |
| `logits-tensor-size` | computed |  |  |  |  |  |  |  |  | [inferred] | Config: [1, 6390, 262208] fp32 logits vs response span only \| Result: 6.7 GB full vs 135 MB response-only \| Note: arithmetic, not a profiler reading | 2026-09-13 |
| `checkpoint-boundary-climb` | probe |  |  |  |  |  |  |  |  |  | Config: one forward across 34 layers with gradient checkpointing \| Result: 4.5 GB to 6.99 GB without offload; 6.39 GB with save_on_cpu | 2026-09-14 |
| `lm-head-cast` | load |  |  |  |  |  |  |  |  |  | Config: pre-training VRAM, lm_head fp32 vs bf16 \| Result: 4502 MB to 3221 MB | 2026-09-14 |
| `real-v1` | real |  | 16 |  |  |  |  |  | 4926 MB | [ok] | Dataset examples: 152 | 2026-09-14 |
| `real-v2` | real |  | 16 |  |  |  |  |  | 5053 MB | [ok] | Note: last resumed segment \| Dataset examples: 490 | 2026-09-14 |
| `real-v3` | real |  | 16 |  |  |  |  |  | 5040 MB | [ok] | Dataset examples: 696 | 2026-09-14 |
| `real-v4` | real |  | 16 |  |  |  |  |  | 4889 MB | [ok] | Dataset examples: 59 | 2026-09-15 |
| `real-v5` | real |  | 16 |  |  |  |  |  | 5117 MB | [ok] | Note: last resumed segment \| Dataset examples: 757 | 2026-09-15 |
| `real-v6` | real |  | 64 |  |  |  |  |  | 5796 MB | [ok] | Dataset examples: 757 | 2026-09-17 |
| `real-v7` | real |  | 112 |  |  |  |  |  | 6647 MB | [ok] | Dataset examples: 757 | 2026-09-17 |
| `real-dryrun` | real |  | 112 |  |  |  |  |  | 6634 MB | [ok] | Note: 2-step dry run of the generic trainer \| Dataset examples: 3 | 2026-09-18 |
| `real-v8` | real |  | 112 |  |  |  |  |  | 6643 MB | [ok] | Dataset examples: 1050 | 2026-09-18 |
| `real-v9` | real |  | 112 |  |  |  |  |  | 6643 MB | [ok] | Dataset examples: 1080 | 2026-09-19 |
| `real-v10` | real |  | 64 |  |  |  |  |  | 5767 MB | [ok] | Dataset examples: 1080 | 2026-09-20 |
| `real-v11` | real |  | 112 |  |  |  |  |  | 6648 MB | [ok] | Dataset examples: 1167 | 2026-09-21 |
| `real-v12` | real |  | 112 |  |  |  |  |  | 6648 MB | [ok] | Epochs: 2; Dataset examples: 1167 | 2026-09-21 |
| `real-v13` | real |  | 112 | yes | 32 |  |  | 7177 | 6653 MB | [ok] | Dataset examples: 1172 | 2026-09-22 |
| `real-v14-run1` | real |  | 112 | yes |  | 0.1 |  |  |  |  | Result: died at step 184: CUDA illegal memory access (Xid 31 MMU fault) \| Dataset examples: 1172 | 2026-09-22 |
| `real-v14-run2` | real |  | 112 | yes |  | 0.1 |  |  | OOM | [oom] | Result: OOM at step 236: 6.98 GiB allocated, 394 MiB reserved-unallocated, 58 MiB free | 2026-09-23 |
| `real-v14-run3` | real |  | 112 | yes |  | 0.1 |  |  | OOM | [oom] | Result: OOM at step 239: 6.90 GiB allocated, 457 MiB reserved-unallocated, 78 MiB free | 2026-09-23 |
| `real-v15` | real |  | 96 | yes | 30 | 0.1 |  |  | 6629 MB | [ok] | Dataset examples: 1172; Predicted MB: 6793 | 2026-09-23 |
| `real-v16` | real |  | 112 |  |  |  |  |  | 6653 MB | [ok] | Dataset examples: 873 | 2026-09-24 |
| `real-v17` | real |  | 112 |  |  |  |  |  | 6658 MB | [ok] | Dataset examples: 1189 | 2026-09-25 |
| `probe-0917-r96` | probe |  | 96 |  |  |  |  | 7037 | 6977.6 MB | [ok] | Dataset: 2026-09-15 corpus; Logits to keep: 319; Pre step MB: 3847 | 2026-09-17 |
| `probe-0917-r112` | probe |  | 112 |  |  |  |  | 7037 | 7216.0 MB | [ok] | Dataset: 2026-09-15 corpus; Pre step MB: 3973 | 2026-09-17 |
| `probe-0917-r128` | probe |  | 128 |  |  |  |  | 7037 | OOM | [oom] | Result: OOM (tried 70 MiB, 48 MiB free) \| Dataset: 2026-09-15 corpus; Pre step MB: 4098 | 2026-09-17 |
| `probe-0918-r96` | probe |  | 96 |  |  |  |  | 7288 | 7058.9 MB | [ok] | Dataset: 2026-09-18 unfiltered | 2026-09-18 |
| `probe-0918-r104` | probe |  | 104 |  |  |  |  | 7288 | 7178.1 MB | [ok] | Dataset: 2026-09-18 unfiltered | 2026-09-18 |
| `probe-0918-r112` | probe |  | 112 |  |  |  |  | 7288 | OOM | [oom] | Result: OOM (tried 72 MiB, 66 MiB free) \| Dataset: 2026-09-18 unfiltered | 2026-09-18 |
| `probe-0918f-r112` | probe |  | 112 |  |  |  |  | 6931 | 7054.8 MB | [ok] | Dataset: 2026-09-18 filtered; Logits to keep: 192 | 2026-09-18 |
| `probe-dora-r112` | probe |  | 112 |  |  |  | yes | 7037 | OOM | [oom] | Result: OOM inside the DoRA weight-norm step (tried 100 MiB, 30 MiB free) \| Pre step MB: 3985 | 2026-09-18 |
| `probe-dora-r64` | probe |  | 64 |  |  |  | yes | 7037 | OOM | [oom] | Result: OOM inside the DoRA weight-norm step (tried 100 MiB, 76 MiB free) \| Pre step MB: 3610 | 2026-09-18 |
| `probe-0919f-r112` | probe |  | 112 |  |  |  |  | 7068 | 6980.0 MB | [ok] | Note: the calibration anchor \| Dataset: 2026-09-19 filtered; Logits to keep: 73 | 2026-09-20 |
| `sweep-r112-drop` | probe |  | 112 | yes | 32 | 0.1 |  | 7177 | OOM | [oom] | Result: OOM \| Dataset: 2026-09-21; Logits to keep: 237 | 2026-09-23 |
| `sweep-r104-drop` | probe |  | 104 | yes |  | 0.1 |  | 7177 | OOM | [oom] | Result: OOM | 2026-09-23 |
| `sweep-r96-drop` | probe |  | 96 | yes |  | 0.1 |  | 7177 | 7127.6 MB | [ok] |  | 2026-09-23 |
| `sweep-r88-drop` | probe |  | 88 | yes |  | 0.1 |  | 7177 | 7007.9 MB | [ok] |  | 2026-09-23 |
| `sweep-r80-drop` | probe |  | 80 | yes |  | 0.1 |  | 7177 | 6890.3 MB | [ok] |  | 2026-09-23 |
| `sweep-r112-nodrop` | probe |  | 112 | yes | 32 | 0 |  | 7177 | OOM | [oom] | Result: OOM, although v13 trained this exact config at a 6653 MB real peak | 2026-09-23 |

## Cross-GPU probes {#cross-gpu}

One-step probes of the training recipe on two GPU vendors. Each probe runs one forward and backward pass of unsloth/gemma-3-4b-it with LoRA on all 7 attention and MLP projections, on synthetic random tokens: seq_len tokens of input with the loss on the last response_len. Peak memory depends on shapes, not on what the tokens say, so the synthetic input stands in for the real worst-case example (checked on the RTX 3060 Ti: pre-step memory at rank 112 was 3,972.6 MB synthetic vs 3,972.55 MB real). No optimizer step, so a real training run's peak differs; see Calibration above. Memory fixes 'all' = chunked MLP, save_on_cpu activation offload, bf16 lm_head and logits_to_keep (see the recipe). Peak allocated is torch.cuda.max_memory_allocated; reserved adds the allocator's cache.

### AMD Radeon RX 7900 XTX {#gpu-rx-7900-xtx}

- **Arch:** gfx1100
- **Total MB:** 24560
- **OS:** Windows 11 (build 26200)
- **Driver:** AMD Adrenalin 26.7.1 (32.0.31035.1003)
- **Stack:** ROCm 7.2.1 Windows SDK wheels, torch 2.9.1+rocm7.2.1, HIP 7.2.53211
- **Libraries:** transformers 5.19.0, peft 0.21.2, accelerate 1.15.0, bitsandbytes 0.50.2
- **Python:** 3.12.10

### NVIDIA GeForce RTX 3060 Ti {#gpu-rtx-3060-ti}

- **Total MB:** 8192
- **Usable GiB torch:** 7.66
- **OS:** Ubuntu, Linux 7.0
- **Driver:** 580.178.04
- **Note:** About 0.55 GB of the card was held by an Ollama process during the probes, so the rank 96 OOM is for about 7.1 GB free, not 7.66.

| GPU | Quant | Memory fixes | Rank | Dropout | DoRA | Seq length (tokens) | Pre-step MB | Peak allocated MB | Peak reserved MB | Step s | Status | Measured on |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| RX 7900 XTX | 4bit | all | 16 | 0.0 | no | 7177 | 3222.4 | 5721.3 | 6694.0 | 16.92 | [ok] | 2026-10-15 |
| RX 7900 XTX | 4bit | all | 64 | 0.0 | no | 7177 | 3597.7 | 6406.2 | 6752.0 | 14.13 | [ok] | 2026-10-15 |
| RX 7900 XTX | 4bit | all | 96 | 0.0 | no | 7177 | 3847.8 | 6884.8 | 7318.0 | 14.28 | [ok] | 2026-10-15 |
| RX 7900 XTX | 4bit | all | 112 | 0.0 | no | 7177 | 3974.5 | 7194.5 | 7578.0 | 14.27 | [ok] | 2026-10-15 |
| RX 7900 XTX | 4bit | all | 128 | 0.0 | no | 7177 | 4099.7 | 7395.9 | 7750.0 | 14.07 | [ok] | 2026-10-15 |
| RX 7900 XTX | 4bit | all | 192 | 0.0 | no | 7177 | 4611.9 | 8383.4 | 8842.0 | 14.8 | [ok] | 2026-10-15 |
| RX 7900 XTX | 4bit | all | 256 | 0.0 | no | 7177 | 5129.1 | 9338.3 | 9732.0 | 14.88 | [ok] | 2026-10-15 |
| RX 7900 XTX | 4bit | all | 96 | 0.1 | no | 7177 | 3847.8 | 7059.3 | 7520.0 | 14.64 | [ok] | 2026-10-15 |
| RX 7900 XTX | 4bit | all | 112 | 0.1 | no | 7177 | 3974.5 | 7368.2 | 7794.0 | 15.46 | [ok] | 2026-10-15 |
| RX 7900 XTX | 4bit | all | 128 | 0.1 | no | 7177 | 4099.7 | 7562.7 | 8038.0 | 14.21 | [ok] | 2026-10-15 |
| RX 7900 XTX | 4bit | all | 256 | 0.1 | no | 7177 | 5129.1 | 9506.9 | 9890.0 | 14.99 | [ok] | 2026-10-15 |
| RX 7900 XTX | 4bit | all | 64 | 0.0 | yes | 7177 | 3633.9 | 8240.7 | 8564.0 | 19.53 | [ok] | 2026-10-15 |
| RX 7900 XTX | 4bit | all | 112 | 0.0 | yes | 7177 | 4010.7 | 9018.8 | 9290.0 | 19.61 | [ok] | 2026-10-15 |
| RX 7900 XTX | bf16 | all | 16 | 0.0 | no | 7177 | 8327.1 | 10826.3 | 11818.0 | 14.7 | [ok] | 2026-10-15 |
| RX 7900 XTX | bf16 | all | 64 | 0.0 | no | 7177 | 8702.3 | 11509.5 | 12290.0 | 13.3 | [ok] | 2026-10-15 |
| RX 7900 XTX | bf16 | all | 112 | 0.0 | no | 7177 | 9089.1 | 12317.5 | 12658.0 | 13.43 | [ok] | 2026-10-15 |
| RX 7900 XTX | bf16 | all | 256 | 0.0 | no | 7177 | 10244.5 | 14443.2 | 14840.0 | 14.38 | [ok] | 2026-10-15 |
| RX 7900 XTX | 4bit | all | 112 | 0.0 | no | 2048 | 3974.4 | 5585.8 | 7288.0 | 4.46 | [ok] | 2026-10-15 |
| RX 7900 XTX | 4bit | all | 112 | 0.0 | no | 4096 | 3974.4 | 6207.9 | 7286.0 | 7.33 | [ok] | 2026-10-15 |
| RX 7900 XTX | 4bit | all | 112 | 0.0 | no | 8192 | 3974.5 | 7486.0 | 7852.0 | 15.91 | [ok] | 2026-10-15 |
| RX 7900 XTX | 4bit | all | 112 | 0.0 | no | 12288 | 3974.6 | 8917.2 | 9498.0 | 30.49 | [ok] | 2026-10-15 |
| RX 7900 XTX | 4bit | all | 112 | 0.0 | no | 16384 | 3974.7 | 10523.6 | 11162.0 | 53.5 | [ok] | 2026-10-15 |
| RX 7900 XTX | 4bit | none | 16 | 0.0 | no | 7177 | 4502.8 | 35829.2 | 37502.0 | 68.68 | [spilled] Finished only because Windows spilled past the 24 GB card into shared system memory; about 4x slower. Would not fit on this card under Linux. See gotcha G31. | 2026-10-15 |
| RX 7900 XTX | 4bit | none | 112 | 0.0 | no | 7177 | 5254.9 | 36581.3 | 38334.0 | 57.79 | [spilled] Finished only because Windows spilled past the 24 GB card into shared system memory; about 4x slower. Would not fit on this card under Linux. See gotcha G31. | 2026-10-15 |
| RTX 3060 Ti | 4bit | all | 16 | 0.0 | no | 7177 | 3222.1 | 5788.9 | 6076.0 | 17.28 | [ok] | 2026-10-15 |
| RTX 3060 Ti | 4bit | all | 64 | 0.0 | no | 7177 | 3597.3 | 6465.0 | 6826.0 | 17.26 | [ok] | 2026-10-15 |
| RTX 3060 Ti | 4bit | all | 96 | 0.0 | no | 7177 | 3847.5 | 6757.3 |  |  | [oom] Ollama held about 0.55 GB of this 8 GB card during the probe | 2026-10-15 |

Source: probe_kit/probe_vram.py and run_sweep.py (this site's probe kit)

## Calibration {#calibration}

- **Method:** Run the one-step probe on the worst-case example, then scale by the probe's measured over-estimate on a config that also has a real run.

### Anchor {#calibration-anchor}

- **Probe MB:** 6980.0
- **Probe:** the 2026-09-19-corpus probe at r112, no dropout
- **Real MB:** 6653
- **Real:** v11/v13 r112 runs (6648/6653)
- **Probe overestimate pct:** 4.7

### Prediction {#calibration-prediction}

- **Config:** r96, rsLoRA alpha 30, dropout 0.1
- **Probe MB:** 7127.6
- **Predicted MB:** 6793
- **Actual MB:** 6629
- **Error pct:** 2.5

- **Derivation note:** The chain script states the 4.7% over-estimate and the 6,793 MB prediction; it does not name the anchor probe. 6980 x (1 - 0.047) = 6652 and 7127.6 x 0.953 = 6793 match, so the anchor is taken to be the 2026-09-19-corpus r112 probe.
- **Value:** replaced three crashes and about 11 GPU-hours that found the rank-112 dropout ceiling the hard way
- **Measured on:** 2026-09-23
- **Status:** [inferred]

## Probe versus real gap {#probe-vs-real-gap}

- **Observed:** The probe over-estimates a real run's peak by roughly 4.7% (anchor) and up to >0.9 GB in the worst case: on the 21 Sep corpus the r112 no-dropout probe OOMs at ~7.6 GB in use, while v13 trained that exact config at 6653 MB.
- **What sources say:** The chain script calls the probe 'a conservative upper bound, not a prediction'. No source explains the mechanism.
- **Probe design:** one forward+backward on the single longest example in the corpus (worst-case activation memory), no optimizer step, model.train() set so gradient checkpointing is active, wrapped in save_on_cpu like the trainer
- **Length sensitivity:** probe peaks rise with the longest example and its response span: r112 fits at 6931 tokens (7055 MB) and 7068 tokens (6980 MB, short response) but OOMs at 7177 and 7288 tokens
- **Measured on:** 2026-09-23
- **Status:** [inferred]

## Rank cost {#rank-cost}

- **Real peak by rank MB:** 16: 4889-5117; 64: 5767-5796; 96 (+dropout): 6629; 112: 6643-6658
- **Probe cost per rank unit MB:** Value: 15; Evidence: dropout sweep: r80 6890.3, r88 7007.9, r96 7127.6 MB, i.e. ~118-120 MB per 8 ranks; measured on 2026-09-23
- **Real cost r64 to r112 MB:** Value: about 850-880; Evidence: v6 5796 vs v7 6647 (same corpus); v10 5767 vs v9 6643 (same corpus); measured on 2026-09-20
- **Measured on:** 2026-09-25

## Dropout memory {#dropout-memory}

- **Claim:** PEFT installs nn.Identity at lora_dropout=0 (allocates nothing) and nn.Dropout above 0, which saves a mask per call for backward across 7 target modules on 34 layers. Estimated cost about 350 MB of peak at rank 112, the whole remaining margin. The rate does not matter; the mask is allocated at any rate above 0.
- **Estimate MB:** 350
- **Note:** 350 MB is the program notes's estimate from comparing crashes against v13's 6653 MB peak, not a direct measurement. It corrects an earlier ledger entry that said dropout cost no extra VRAM.
- **Measured on:** 2026-09-23
- **Measured update:** Measured on 2026-10-15 at the one-step probe: dropout 0.1 adds 167 to 175 MB at ranks 96 to 256 (see Cross-GPU probes and finding F30), about half this estimate.
- **Status:** [inferred]

## Inference footprint {#inference-footprint}

| Model | GGUF size GB | Resident GB estimate | Note | Measured on |
| --- | --- | --- | --- | --- |
| fine-tuned router, Q8_0 | 4.14 | 4.4 |  | 2026-09-25 |
| fine-tuned router, Q4_0 | 2.37 |  |  | 2026-09-16 |
|  |  |  | untuned model resident sizes are in baselines.json |  |

## Host RAM {#host-ram}

- **Incident:** The first rank probe lacked model.train(), so HF gradient checkpointing was silently off and save_on_cpu spilled every activation to pinned host RAM. It thrashed the box and later hit a 32 GB cgroup memory cap. Probes now run under systemd-run --scope -p MemoryMax=12G.
- **Measured on:** 2026-09-17