One Engineer Trained a 3.8B LLM to 0.384 CORE for $998

Hugo Vergnes trained a 3.8B-parameter decoder-only LLM from scratch to 0.384 CORE on 65.3B tokens in 43 hours for $998 on 8x B200s. The project, little-lm, is a config-driven framework inspired by Karpathy's nanochat. The headline isn't the score — it's the cost-per-capability curve it implies for solo engineers.

For context: GPT-2 1.5B scores 0.2565 CORE. nanochat d32 (~1B params) scores 0.310 for roughly $1,000. little-lm lands meaningfully ahead of both at the same budget, largely because B200s delivered better value per unit of work than the H100s used in the nanochat runs.

Architecture

The final model is Llama-style: RMSNorm, RoPE, GQA (24 query heads, 8 KV heads), relu² MLPs, QK-norm, logit softcap, per-layer learnable residual scalars, and ResFormer-style value embeddings. The value embeddings are 19% of the parameter count — 14 tables of vocab × kv_dim, one on every other layer. Token embeddings and the untied LM head each contribute 154.5M params; 28 decoder layers carry 2,818.7M.

What actually worked

Vergnes spent six days on a single A100 training an 858M model on FineWeb-Edu that scored PIQA 60.45% — worse than GPT-2 124M from 2019. The post-mortem produced five changes that closed the gap:

• Trapezoidal LR schedule. 5% warmup, flat hold, linear cooldown over the last 50% to 5% of peak. The 858M run's cosine-to-zero schedule went flat at 70% of steps; the 3.8B run's eval loss was still descending at the final step.
• Muon for matrix params, AdamW for the rest. Muon is ~25% slower per step in a shallow-accumulation benchmark, but that cost dilutes to ~4% at 7 gradient-accumulation steps.
• ClimbMix instead of FineWeb-Edu. Vergnes calls it a tremendous jump in convergence speed, matching Karpathy's finding.
• FP8 + vocab padding. FP8 via torch._scaled_mm with dynamic tensorwise scaling on all three GEMMs, plus padding vocab from 50,257 to 50,304 (multiple of 64). Together, +33% throughput.
• 1024 context instead of 2048. Halving context roughly doubles batch size at fixed memory; throughput per token barely changes because the run is MLP-dominated.

Steady state was ~480,000 tok/s. The 35.9h wall clock vs 33h of pure training is CORE evals — ten of them, ~15 minutes each, 7% of total. Re-running the same recipe at 2048 context scored 0.3840 vs 0.3384 at 1024; almost all of that gap came from context-dependent tasks.

On the hardware side: 92% SM activity, 40% SM occupancy, ~1,047 TFLOP/s sustained per B200 (~25% MFU against dense FP8 peak, ~50% against bf16). Distributed strategy was plain DistributedDataParallel — at 3.8B on one node, gradient communication was never the constraint.

Throughput work on a single RTX 5090

Before renting a node, Vergnes pushed an 858M baseline from 26,144 tok/s to 37,621 tok/s on one 5090. FP8 added 25%. Vocab padding added 33% cumulative. Fused linear cross-entropy (Liger's FusedLinearCrossEntropyLoss) is 6% slower per step at matched batch size but saves ~8 GB VRAM, letting batch size climb from 6 to 10 and netting a throughput win. Dropping the SwiGLU gate (relu², two matmuls instead of three) went 183,035 → 214,173 tok/s and saved 6 GB — but the caveat is that a SwiGLU intermediate ratio of 2.75 does not transfer to relu²; use 4x for non-gated. bf16 master weights cut VRAM 27% and raised throughput 2.2x on a 1.5B config, at a small quality cost (CORE 0.22 vs 0.23 at 4,000 steps).

What didn't work

Document-boundary masking with flex attention — per-token document IDs and a custom attention mask — was deleted entirely. Best-fit packing replaced it in ~10 lines, and attention went back to an unconditional F.scaled_dot_product_attention(..., is_causal=True). Karpathy found the same: cross-document leakage under BOS-aligned packing doesn't hurt much.