VRAM measurements

Markdown: vram.md · Data: vram.json

Every VRAM measurement from the program. The cross_gpu section adds one-step probes on an AMD RX 7900 XTX (24 GB) and a second NVIDIA card. Training GPU: one 8 GB NVIDIA consumer card (an RTX 3070 Ti by today's nvidia-smi; torch reports 7.65 GiB total capacity). 'mode' is probe (one forward+backward on the single longest example, no optimizer step) or real (a full Trainer run; peak = torch.cuda.max_memory_allocated at the end). All training configs are 4-bit nf4 QLoRA on unsloth/gemma-3-4b-it with the recipe.json memory fixes unless noted.

GPU

Model

NVIDIA GeForce RTX 3070 Ti

Total MiB

8192

Usable GiB torch

7.65

Second card

NVIDIA GeForce RTX 3060 Ti, 8192 MiB

Driver

580.178.04

Note

Card names read on 2026-10-15 and mapped through CUDA_DEVICE_ORDER=PCI_BUS_ID, CUDA_VISIBLE_DEVICES=1 as set in every training script (the probe scripts document the same pin). This holds as long as neither card changed PCIe slot since September; the owner confirms both cards were in the box then. The 7.65 GiB figure is printed in every OOM message.

Measured on

2026-10-15

Status

inferred

Measurements

Mode is real (a training run), probe (one forward and backward pass on the longest example) or load (model load only). Peak is the highest GPU memory seen. OOM means the run ran out of memory.

Cross-GPU probes

One-step probes of the training recipe on two GPU vendors. Each probe runs one forward and backward pass of unsloth/gemma-3-4b-it with LoRA on all 7 attention and MLP projections, on synthetic random tokens: seq_len tokens of input with the loss on the last response_len. Peak memory depends on shapes, not on what the tokens say, so the synthetic input stands in for the real worst-case example (checked on the RTX 3060 Ti: pre-step memory at rank 112 was 3,972.6 MB synthetic vs 3,972.55 MB real). No optimizer step, so a real training run's peak differs; see Calibration above. Memory fixes 'all' = chunked MLP, save_on_cpu activation offload, bf16 lm_head and logits_to_keep (see the recipe). Peak allocated is torch.cuda.max_memory_allocated; reserved adds the allocator's cache.

AMD Radeon RX 7900 XTX

Arch

gfx1100

Total MB

24560

OS

Windows 11 (build 26200)

Driver

AMD Adrenalin 26.7.1 (32.0.31035.1003)

Stack

ROCm 7.2.1 Windows SDK wheels, torch 2.9.1+rocm7.2.1, HIP 7.2.53211

Libraries

transformers 5.19.0, peft 0.21.2, accelerate 1.15.0, bitsandbytes 0.50.2

Python

3.12.10

NVIDIA GeForce RTX 3060 Ti

Total MB

8192

Usable GiB torch

7.66

OS

Ubuntu, Linux 7.0

Driver

580.178.04

Note

About 0.55 GB of the card was held by an Ollama process during the probes, so the rank 96 OOM is for about 7.1 GB free, not 7.66.

Source: probe_kit/probe_vram.py and run_sweep.py (this site's probe kit)

Calibration

Method

Run the one-step probe on the worst-case example, then scale by the probe's measured over-estimate on a config that also has a real run.

Anchor

Probe MB

6980.0

Probe

the 2026-09-19-corpus probe at r112, no dropout

Real MB

6653

Real

v11/v13 r112 runs (6648/6653)

Probe overestimate pct

4.7

Prediction

Config

r96, rsLoRA alpha 30, dropout 0.1

Probe MB

7127.6

Predicted MB

6793

Actual MB

6629

Error pct

2.5

Derivation note

The chain script states the 4.7% over-estimate and the 6,793 MB prediction; it does not name the anchor probe. 6980 x (1 - 0.047) = 6652 and 7127.6 x 0.953 = 6793 match, so the anchor is taken to be the 2026-09-19-corpus r112 probe.

Value

replaced three crashes and about 11 GPU-hours that found the rank-112 dropout ceiling the hard way

Measured on

2026-09-23

Status

inferred

Probe versus real gap

Observed

The probe over-estimates a real run's peak by roughly 4.7% (anchor) and up to >0.9 GB in the worst case: on the 21 Sep corpus the r112 no-dropout probe OOMs at ~7.6 GB in use, while v13 trained that exact config at 6653 MB.

What sources say

The chain script calls the probe 'a conservative upper bound, not a prediction'. No source explains the mechanism.

Probe design

one forward+backward on the single longest example in the corpus (worst-case activation memory), no optimizer step, model.train() set so gradient checkpointing is active, wrapped in save_on_cpu like the trainer

Length sensitivity

probe peaks rise with the longest example and its response span: r112 fits at 6931 tokens (7055 MB) and 7068 tokens (6980 MB, short response) but OOMs at 7177 and 7288 tokens

Measured on

2026-09-23

Status

inferred

Rank cost

Real peak by rank MB

16: 4889-5117; 64: 5767-5796; 96 (+dropout): 6629; 112: 6643-6658

Probe cost per rank unit MB

Value: 15; Evidence: dropout sweep: r80 6890.3, r88 7007.9, r96 7127.6 MB, i.e. ~118-120 MB per 8 ranks; measured on 2026-09-23

Real cost r64 to r112 MB

Value: about 850-880; Evidence: v6 5796 vs v7 6647 (same corpus); v10 5767 vs v9 6643 (same corpus); measured on 2026-09-20

Measured on

2026-09-25

Dropout memory

Claim

PEFT installs nn.Identity at lora_dropout=0 (allocates nothing) and nn.Dropout above 0, which saves a mask per call for backward across 7 target modules on 34 layers. Estimated cost about 350 MB of peak at rank 112, the whole remaining margin. The rate does not matter; the mask is allocated at any rate above 0.

Estimate MB

350

Note

350 MB is the program notes's estimate from comparing crashes against v13's 6653 MB peak, not a direct measurement. It corrects an earlier ledger entry that said dropout cost no extra VRAM.

Measured on

2026-09-23

Measured update

Measured on 2026-10-15 at the one-step probe: dropout 0.1 adds 167 to 175 MB at ranks 96 to 256 (see Cross-GPU probes and finding F30), about half this estimate.

Status

inferred

Inference footprint

Host RAM

Incident

The first rank probe lacked model.train(), so HF gradient checkpointing was silently off and save_on_cpu spilled every activation to pinned host RAM. It thrashed the box and later hit a 32 GB cgroup memory cap. Probes now run under systemd-run --scope -p MemoryMax=12G.

Measured on

2026-09-17