Data-mix weights extrapolate badly: the effect reverses with model size
Serguey Asael Shinder2 min readAlex Renda and Nitya Mani of Jane Street have published a study of data weighting across in-house and open-weight language models, at https://blog.janestreet.com/a-study-of-sequence-weighting-at-scale/. They vary the weight a sequence carries in the training loss and measure how strongly the model's loss reduction on that sequence depends on it.
The finding is non-monotonic across scale, and the shape is worth stating carefully:
- at small scale, models learn general patterns largely independent of the weights;
- at medium scale, they learn data-specific patterns roughly in proportion to the weights;
- at large scale, they learn everything present in the data and the dependence on weight falls away again.
That matters because of how these decisions are actually made. Running a dense grid search over data-mix hyperparameters is intractable at medium scale and impossible at the largest, where you may train one model for the task at all. The standard workaround is a scaling law: fit the hyperparameter cheaply at small scale and extrapolate — the Chinchilla move.
Extrapolation only holds if the behaviour is either stable across scales or changes predictably. The authors call behaviours that satisfy neither aberrant, and report that data mixing is particularly prone to them. They cite a parallel in the MAI-Thinking-1 technical report (§2.5.2), where the ranking between a code-heavy and a STEM-heavy mix reversed as model size grew.
Methodologically, the reason they study weighting rather than mixing is to isolate one variable: mix experiments confound the weight of a source with its quality and with how much unique information its marginal tokens carry. Per-sequence weights remove both.
What it means
A hyperparameter tuned at small scale is a hypothesis, not a measurement. The whole industry practice of fitting mix weights on small runs assumes the curve is smooth. This says that for data mixing specifically, the curve can turn around — so the small-scale result can be not merely imprecise but backwards.
"The big model learns it anyway" is the practical takeaway for upweighting. If, at large scale, the model absorbs patterns regardless of the weight they carried, then effort spent upweighting a favoured corpus buys less than expected — and the mechanism you actually wanted was probably filtering or dedup, which change what is present rather than how loudly it is repeated.
And the honest version of this result is a warning about your own evaluation. Two teams can run the same mix experiment at different scales and reach opposite conclusions, both correctly. Before quoting a mix ratio as settled, the question is at what scale it was established, and whether anyone checked that the ordering survives the next order of magnitude.