// HACKER NEWS — CYBERSECURITY
What happens when an LLM never sees material beyond fifth grade?
The hosted 5B model, live in your browser.
Open in a new tab ↗
if the chat doesn’t load below.
Modern LMs are trained on everything at once, so it is hard to tell whether a new skill
was learned or merely elicited. We constrain the training distribution itself: an 88B-token
corpus filtered to the U.S. elementary-school curriculum, with models trained from scratch on it and
matched unfiltered controls.
An 88B-token corpus distilled from FineWeb-Edu through a five-stage filtering pipeline aligned with Common Core standards (K–5).
Concepts, facts, and vocabulary taught above Grade 5 are explicitly excluded.
Three scales (0.6B / 1.3B / 5B) trained from scratch on LittleCurriculum: chattable models
with an interpretable knowledge boundary. Each ships with a matched Unfiltered control
for clean comparison.
In our experiments, scaling, SFT+GRPO post-training, and in-context learning amplify what the
curriculum taught, but none meaningfully improves out-of-scope performance, indicating that the
pretraining filter sets the effective capability ceiling.
LittleLearner at three scales (0.6B / 1.3B / 5B), each with a matched Unfiltered control
sharing its architecture, tokens, and recipe.
Base: the pretrained model.
GRPO: math specialists post-trained on MathCAMPS; responses may exhibit a tendency toward math-oriented output.
Chatty: variants tuned for general chat behavior.
Can standard interventions push a model past what its pretraining data taught it?
With the boundary under experimental control, we can ask cleanly. In our experiments, each
intervention amplifies in-scope ability; none of them meaningfully improves out-of-scope
performance.
Scaling model size improves performance within the model’s controlled knowledge exposure and
extends modestly to problems along the same learning trajectory, but yields little improvement on
problems requiring more advanced capabilities outside the exposure.
Post-training through GRPO significantly boosts in-scope K–5 capabilities, but fails to
recover out-of-scope beyond-K–5 capabilities, even when training with out-of-scope data.