// HACKER NEWS — CYBERSECURITY
Training a 3.8B LLM to 0.384 CORE for $998 – Hugo Vergnes
Somewhere between “nanoGPT toy” and “you need a research lab” there’s a large, under-described region where one person with a few thousand dollars can train a meaningful model.
I wanted to see language and understanding emerge from random weights for myself, and to learn the parts you can only learn by starting from scratch. This project was written in the evenings, debugged on a 5090 and finished on rented B200s. It was heavily inspired by Andrej Karpathy’s nanochat.
The result is a 3.8B-parameter model scoring 0.384 on CORE, trained on 65B tokens in 43 hours for $998.
What follows is what worked, what didn’t, and what I still don’t know.
My model is larger than nanochat d32 and took similar wall-clock time. B200s were better value per unit of work than H100s. But for roughly the same money as nanochat’s $1,000 configuration, this lands meaningfully ahead of it. An encouraging data point about what’s reachable outside a lab or a mega company with millions in compute budget. As the frontier moves, $1,000 takes you further and further.
I’ve built little-lm as a config-driven framework for training small decoder-only LLMs. Every run is fully specified by a YAML file: model, dataset, optimizer, schedule, callbacks. Components self-register into a global registry and get resolved by name, so swapping an optimizer or a dataset is a one-line config change.
Good infrastructure pays for itself almost immediately. Ordinary software engineering discipline (Things like separation of concerns, clean interfaces, components you can swap in) matters a lot in AI work. It cost me a little at the start, and a couple more times afterward to fix bad contracts or suboptimalities. But this time investment pays for itself at the first convergence problem you encounter. I found that a great infra is the infra that almost never requires you to edit code manually. If you can read the config and understand exactly what happens, and there are no hidden mechanics, it means you have done a good job. The following report is the result of being able to express experiments as a three-line YAML diff rather than a branch.
The final model is Llama-style: RMSNorm, RoPE, GQA (24 query heads, 8 KV heads), relu² MLPs, QK-norm, logit softcap, per-layer learnable residual scalars, and ResFormer-style value embeddings.
Worth noting that the value embeddings are 19% of the parameter count. 14 tables of vocab × kv_dim, one on every other layer.
I trained an 858M Llama on FineWeb-Edu for 16.4B tokens, 5.8 days on a single A100. AdamW at 2.5e-4, cosine decay to zero, 5% warmup, batch 256 via gradient accumulation, 2048 context.