Paper: plain byte-level Transformers outscale subword models with two training tweaks

Victoria Shinder12 October 2026, 08:282 min read

A preprint posted to arXiv on 5 October, Byte Language Models: Scaling, Emergent Abstractions, and Information Allocation, by four authors affiliated with East China Normal University and Fudan University in Shanghai, argues that language models reading raw bytes instead of tokenizer output do not need a special architecture to compete — and that their longer sequences can be an asset rather than a cost.

The question. A typical subword token covers about four bytes, so a byte-level model processes roughly four times as many positions for the same text and loses the ready-made units a tokenizer provides. Recent byte-level designs such as BLT, H-Net and ByteFlow add built-in chunking layers to cope. The authors instead keep a flat Transformer — a mixed-attention model following the Qwen3.5 setup, interleaving Gated DeltaNet and gated softmax attention 3:1 — and compare it with an equivalent subword model using a 49,152-token vocabulary.

Two ingredients. Training bytes with the standard recipe works poorly, the paper says, so the byte models get:

• token-superposition training, which for the first 30% of training averages four consecutive embeddings and predicts the next four-token bag, before reverting to normal next-token prediction;
• hash embeddings, which let each byte carry information about nearby multi-byte patterns (plus a wider convolution in the DeltaNet layers).

Reported results. With those, byte Transformers consistently outperform subword ones as parameter count grows, and keep improving as data grows; at a fixed compute budget the best data-to-parameter ratio is 1.5 to 3 times that of subword models. On an input-copying task the models turned out to read information only at certain segmentation-like positions — not necessarily spaces — and restricting up to 25% of intermediate layers to those positions kept downstream performance without retraining. Per-byte loss is very uneven: many bytes are nearly free to predict, with uncertainty concentrated at the boundaries of these learned units. Using that for speculative decoding gave 3.4× more accepted tokens than in subword models.

Caveats. This is a preprint, not yet peer-reviewed, and the abstract does not state model sizes in absolute terms; the comparisons are within the authors' own training setup, and the byte models receive training techniques the subword baselines do not — the paper says it ablates them separately. Whether the advantage holds at frontier scale is not shown.

Source: arXiv 2610.05978 — https://arxiv.org/abs/2610.05978