// HACKER NEWS — CYBERSECURITY
A study of sequence weighting at scale
TL;DR: We study the scaling laws of data weighting across in-house and open-weight LMs, finding non-monotonic behavior across scales.
We vary the weight assigned to sequences during training and measure how strongly the model’s loss reduction on a sequence depends on the sequence’s weight.
Taken together, our results are consistent with a general trend: as models transition from small to medium scale, they transition from learning general patterns independent of data weight to learning data-specific patterns proportional to the data weights.
As models then transition from medium to large scale they are able to learn all patterns present in the data, once again independent of data weight.
When training neural networks, we spend a lot of time thinking about precisely what we want our models to learn.
How much do we care about our models being generally intelligent across all coding tasks versus being good at OCaml specifically?
How useful is data from an older market regime compared to data collected recently?
Generally, how much do we want to upweight high-quality data versus allowing our models to learn from everything?
These questions are ultimately data mixing questions, questions about how and where to spend our models’ representational capacity and our FLOPs.
Further complicating these questions is that in practice, many of the experiments we would like to run are prohibitively expensive.
It’s intractable to run a dense multi-dimensional grid search over hyperparameters even at medium scales, let alone at the largest scales (where we may only be able to train one model for a given task).
The standard approach to this is to fit hyperparameters like data mix weights via a scaling law approach, in which we fit hyperparameters at small scales and then extrapolate those hyperparameters to larger scales (the canonical example of this being Chinchilla scaling).
A core challenge for scaling laws is that for the extrapolation to hold, large-scale models must have behaviors that are either (a) invariant with scale (e.g., MuP-style results in which many hyperparameter optima remain stable across scales) or (b) predictable via extrapolating changes in behavior at small scale (e.g., Kaplan-style results in which loss decreases predictably with scale).
Unfortunately, not all behaviors have these characteristics: some behaviors are emergent at scale and are not predictable via scaling laws.
For the rest of this post, we’ll call such non-predictable scaling behaviors aberrant.
Our internal experiments have shown that data mixing is a setting particularly prone to aberrant scaling behaviors.
This parallels findings in the MAI-Thinking-1 technical report (section 2.5.2), where the relative ordering in quality of training with a code-heavy and a STEM-heavy mix reversed as model size increased.
We therefore set out to precisely characterize the extent to which language models actually learn patterns in the data proportional to the weight accorded to that data in a data mix, and how that behavior changes across scales.
Data mix experiments can conflate the effect of the data’s weight in a corpus with other important but different considerations.
Most notably, results from data mix experiments are heavily affected by variation in data quality across different sources and by differences in the uniqueness of marginal tokens in any given data source.
To isolate the effects of mix weight alone, in this post we specifically analyze data weighting, a variant of data mixing in which each individual sequence in our training dataset receives a different weight in our loss.
We measure the extent to which a model has learned a training sequence with a given weight by first assigning random sequence weights to each sequence in the dataset, training a model on the dataset, then re-evaluating it on the same training dataset.
We then define a metric, the effective sequence weight exponent (denoted by p^*) that measures what power of a sequence’s weight