# About: method and scope

> How the numbers were measured, what the terms mean, what is not published, and how to report a correction.

Canonical: https://marvinlabs.org/about
Data: [baselines.json](https://marvinlabs.org/data/baselines.json), [findings.json](https://marvinlabs.org/data/findings.json), [gotchas.json](https://marvinlabs.org/data/gotchas.json), [inference.json](https://marvinlabs.org/data/inference.json), [recipe.json](https://marvinlabs.org/data/recipe.json), [versions.json](https://marvinlabs.org/data/versions.json), [vram.json](https://marvinlabs.org/data/vram.json)
HTML: https://marvinlabs.org/about

## Method {#method}

Each version was trained, served through Ollama and scored against a routing test set. Dates are UTC. Every record carries a measured-on date.

### Definitions {#definitions}

In short: accuracy is the share of routing test cases answered with the right capability and args. Held-out and fair-slice cases are those with no training label.

- **Accuracy:** passed / runnable cases. A case passes when the router returns the asserted capability and arguments, and none of the case's forbidden capabilities.
- **Held out:** cases with no training label in the corpus the model was trained on. The held-out slice changed size as the case file grew, so compare held-out numbers only within one test-set era.
- **Fair slice:** cases carrying no train key at all, which neither compared model ever saw (used for v16 and v17).
- **Guard:** must-not-fire cases: utterances that must NOT route to a named capability (negations, quoted or pasted content, vague phrasing). Counts only; case text is excluded.
- **Speed:** mean seconds per case through the production routing prompt (~5.6k tokens), model fully on GPU unless noted.
- **Dates:** UTC. measured_on is the file timestamp of the result, or the date the program notes records.

### Scope notes from the data {#scope-notes}

- **findings.json:** Lessons from 17+ LoRA fine-tunes of gemma-3-4b-it as a command router on one 8 GB consumer GPU. Each claim is limited to this task, this base model and this recipe. Accuracy numbers are passed / runnable cases; see versions.json for definitions.
- **recipe.json:** The training recipe shared by every fresh-from-base version (v1-v3, v5-v17). v4 differs only in continuing from v3's adapter at 8 epochs. Values come from the training scripts and the run headers printed in each training log.
- **vram.json:** Every VRAM measurement from the program. The cross_gpu section adds one-step probes on an AMD RX 7900 XTX (24 GB) and a second NVIDIA card. Training GPU: one 8 GB NVIDIA consumer card (an RTX 3070 Ti by today's nvidia-smi; torch reports 7.65 GiB total capacity). 'mode' is probe (one forward+backward on the single longest example, no optimizer step) or real (a full Trainer run; peak = torch.cuda.max_memory_allocated at the end). All training configs are 4-bit nf4 QLoRA on unsloth/gemma-3-4b-it with the recipe.json memory fixes unless noted.
- **baselines.json:** Untuned (off-the-shelf) models, measured before and during the fine-tune program through Ollama. 'routing' = the production command-routing prompt (~5.6k tokens); 'content' = 11 generation workloads (summaries, tagging, narration and similar) scored by a judge model. 'floor' tier = unambiguous cases every acceptable model must pass; 'hard' tier = deliberately ambiguous cases. Tournament numbers below were recomputed from the raw per-cycle score records by time window and checked against the program's written notes where they quote them.
- **gotchas.json:** Environment and tooling failures hit during the program, with symptom, cause, fix and versions. Versions are those recorded at the time; see recipe.json for the final stack.

## What is not published {#not-published}

The training corpus, the text of the test cases and the exact text of the safety-guard cases are deliberately not published. They appear only as counts and broad categories.

## Corrections {#corrections}

Numbers marked unverified or inferred are flagged in the data. An inferred number was derived by the authors, not read straight from a result file. An unverified number could not be checked against a primary result.

Contact: marvinmediaai@gmail.com

## License {#license}

The data and text on this site are licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Use them for anything, with credit to MarvinLabs and a link to marvinlabs.org.

## Data files {#data-files}

- [baselines.json](data/baselines.json)
- [findings.json](data/findings.json)
- [gotchas.json](data/gotchas.json)
- [inference.json](data/inference.json)
- [recipe.json](data/recipe.json)
- [versions.json](data/versions.json)
- [vram.json](data/vram.json)