Scaling laws are by now well established for large models: the bigger the model, the more capable it is and the better it does on downstream tasks. But how do you know how large a model to train, and how much data it needs to match? The field’s most popular answer — Chinchilla — says spend compute at ~20 tokens per parameter. That recipe, though, assumes an effectively infinite corpus. Our antibody corpus is fixed and small:
U0 = 7.04M sequences ≈ 2.25×109 tokens per epoch. Once compute grows,
we are forced to repeat data, and repeated tokens are worth less than fresh ones. The right
object is therefore a data-constrained ("domain-Chinchilla") law that separates model size N,
unique data U, and repetition R = D/U0:
L(N, U, R) = E + A / Nα + B / Deffβ,
Deff = U0·(1 + Rd*·(1 − e−(R−1)/Rd*))
Six parameters: floor E, capacity term (A, α), data term (B, β), and the
repetition decay constant Rd*. Everything below is aimed at pinning these six from our own runs.
At a glance — return & allocation on the fixed antibody corpus. (Left) repeated data stays near-fresh far longer under masking (Rd*≈151) than under causal LMs (Rd*≈15): 151 epochs are worth ≈96 fresh, with no overfitting turn through the deepest 118.5 epochs. (Right) at one fixed budget the compute-optimal recipe is set by the objective — the two causal-LM recipes (Chinchilla, Muennighoff) pick ≈500M models, while masked modeling (Cheng, ours) goes smaller and longer; our optimum is 80M for 51 epochs.
2 Aim 1 — muP-completeP: tune the LR once, transfer to any size
To trace a loss-vs-N curve at each compute budget we must train models from 20M to 800M.
Re-tuning the learning rate at every size would confound "size effect" with "LR mistuning".
muP-completeP removes that: you tune a single base_lr at the 124M proxy, and the
parameterization fans it out and rescales the rest of the network automatically as a function
of width and depth. Concretely, what completeP modifies:
Tensor group
Quantity
completeP scaling (relative to base 256-wide, 1-deep)
hidden weight matrices
learning rate
base_lr × (1/width_mult) × Lαd−1
hidden weight matrices
init variance
∝ width−1
residual branch
output scale
∝ L−αd (= 1/√L at αd=0.5)
readout / unembedding
init & LR
∝ 1/width ; LR width-invariant
embed / LayerNorm
learning rate
width-invariant (scale 1)
Adam ε, weight decay
—
ε ÷ width_mult (× L−αd on hidden); wd × width_mult
Here width_mult = hidden/256 and depth enters through the exponent αd.
On 124M that gives hidden_lr = base_lr / 12.5 — you set one number, completeP does the rest.
The one fix that mattered: αd = 1.0 → 0.5
At the textbook depth exponent αd=1.0 the depth transfer is broken. Changing the
residual/LR depth exponent to 0.5 (residual ∝ 1/√L) fixes it. Coordinate-check transfer
scores (lower is better; <0.10 excellent, 0.10–0.20 ok, >0.20 broken):
Coord-check axis
αd = 1.0
αd = 0.5
non-muP control
width
0.138
0.091
0.55 (broken)
depth
0.275 (broken)
0.091
0.30 (broken)
joint width+depth+heads (real ladder path)
—
0.075–0.120
1.024 (broken)
Outcome: base_lr locked at 2.5×10−3 (bowl minimum, verified) and reused at every
model size — no per-size LR sweep. An aspect-ratio probe (wide vs deep at fixed N, C) further shows
model shape is a second-order effect (Δloss ≤ 0.001, within noise), so N is the right
coordinate and depth transfer holds in a full training run, not just the 8-step diagnostic.
3 Two experiment families
Family
What varies
Pins
Allocation — iso-FLOP ladder (isoJ)
fixed corpus U0; at each budget ck fix total D, scan N (20M–800M)
α, A, and the N-frontier N*(C)
Return — repetition grid (rdREP)
fresh arm (R=1, vary U) + repetition arm (fix U, crank R→128, 3 seeds)
β, B, E (fresh) and Rd* (repetition)
4 Results
Measured iso-FLOP bowls: eval loss vs model size N (log), one curve per compute budget c1–c5 (★ = compute-optimal N*).
The optimum stays small (≈20–80M) and the empirical frontier climbs only slowly across the 6.5× compute span, far below the free-data Chinchilla prediction. (Repetition-return curves — near-fresh band widening with unique data, sharp turn at R≈64–96 for U0/32 up to >118.5 for the full corpus — are summarized in the overview figure and Section 6.)
Representative bowl (c1, C = 8.3×1018), final eval loss (LR fully decayed):
N
20M
40M
80M
124M
160M
200M
320M
loss
.21967
.22015
.22263
.22694
.23006
.23132
.23438
Headline: the compute-optimal model is ~10× smaller than the free-data Chinchilla prior.
At c1 the prior predicts N*≈228M; we measure N*≈20–40M. Every budget's optimum sits far to the
left, and the optimum stays small (≈20–80M) across all five budgets rather than climbing toward the prediction.
5 The fitted law & compute-optimal frontier
Solving the surface along each iso-FLOP line gives the compute-optimal frontier N*(C), R*(C), L*(C). On a single fixed corpus the six parameters are individually non-identifiable (a "sloppy" model — E, α, β, Rd* are constrained only jointly), so we report the derived frontier rather than parameter point values. The deepest run (20M@c4, ~118 ep) sets the repetition cap at R≤118. The single additive surface carries residual stress at the top two budgets (reduced χ2 ≈ 3.83 on the turn-free R≤118 domain): the fitted N* stays ~30M while the measured bowls turn up at 40–80M. We therefore take the measured bowl minima as the primary N*(C) evidence and use the fit for the two-regime structure and the L* interpolation. The joint fit places the repetition constant at Rd*≈151 (χ2ν≈3.1) — an order of magnitude above the causal-LM ≈15, because dynamic masking makes each repeat only a partial repeat.
C (FLOPs)
fitted N* [95% CI]
R* (ep)
L*
measured N* (bowl)
c1 8.3×1018
25M [10, 31]
24
0.2166
20–40M
c2 1.3×1019
26M [10, 32]
37
0.2131
≤20M
c3 2.1×1019
27M [13, 33]
58
0.2097
20–40M
c4 3.4×1019
29M [21, 35]
88
0.2066
40M
c5 5.4×1019
34M [34, 37]
117
0.2039
80M
Fitted N* uses the joint σ-weighted robust fit (bootstrap CIs, 300 resamples); measured N* is the two-sided bowl minimum on the isoFLOP ladder. They agree at c1–c3 and diverge at c4–c5 — the additive form cannot bend up fast enough, so the measured value is what we report as the optimum.
Fit diagnostics for the data-constrained law (reduced χ2≈3.8 on the turn-free R≤118 cells). (a) eval loss vs N, one curve per budget (points = measured cells, ★ = measured optimum, ◊ = fitted optimum); the held-out c6 measured optimum (80M) lands below its predicted curve. (b) loss vs repetition R along each iso-FLOP line, ★ = fitted compute-optimal R* sliding 24→117. (c) predicted vs measured loss; the held-out c6 cells sit above the diagonal (measured < predicted).
The allocation rule: extra compute does not buy a bigger model first. It buys more epochs of a small (~30M) model (matching Muennighoff’s “smaller models, more epochs”), and grows the model only once epochs saturate at the repetition ceiling (~118 ep, around c4–c5). Under left-censoring a single N*(C) slope is not identifiable, so the frontier is reported as measured optima rather than as a power-law slope. The fresh-arm data exponent is β ≈ 0.44. The turn is flat (~64–96 ep) at small corpora and rises steeply with U, the deeper arms setting lower bounds (U0/8 >160, U0/4 >192, full corpus >118.5). The ceiling is also N-dependent: on U0/32, 124M overfits past 64–96 ep while 20M keeps improving through 128 ep — set by the parameter-to-data ratio N/U, not the corpus alone.
Held-out test fixed before measurement at C = 8.7×1019 FLOPs: predicted N*(c6) ∈ [54, 87]M, L*(c6) = 0.2029 [0.2003, 0.2062]. Measured: the c6 bowl bottoms at 80M (eval loss 0.1922) — inside the predicted size interval ✓ and below the predicted loss, so the fit runs conservative at high C. The size call holds and the loss error falls on the safe side.
6 Headline conclusions
Infrastructure works. muP-completeP (αd=0.5) transfers the LR across width and depth; one base_lr trains the whole 20M–800M ladder, confirmed directly by a 3-size base_lr sweep (optimum stays 2.5–5×10−3 across 8× size).
Compute-optimal models are small — and stay small. Measured N* is ~10× below free-data Chinchilla (~20–40M vs 228M at c1) and climbs only slowly (to ~80M at c5), never toward the free-data prediction.
Spend compute on epochs first, then size. Extra compute buys more epochs of a small (~30M) model until epochs approach the repetition ceiling (~118 ep, around c4–c5), then grows the model — Muennighoff’s “smaller models, more epochs,” made quantitative for antibody MLM.
Repeated data is nearly free — until it isn’t. Near-fresh for a band that widens with unique data (turn 64–96 at U0/32, >160 at U0/8, >192 at U0/4, >118.5 at full corpus), then a sharp overfitting turn — far more repetition-tolerant than the autoregressive ≈15 (fitted Rd*≈151, an order of magnitude higher). The turn is set by the parameter-to-data ratio N/U: a smaller model tolerates deeper repetition (20M still improving at 128 ep where 124M has overfit).
The binding constraint flips from GPUs to unique sequences at the repetition ceiling (~c4–c5): past it the corpus, not compute, caps the loss — reaching a much larger (~3B) model would demand substantially more unique antibody sequences, not more GPUs.
Measured minima define the frontier. On one corpus the six parameters are individually non-identifiable, and the single additive surface stresses at the top two budgets (reduced χ2≈3.83): the fitted N* stays ~30M while the measured bowls turn up at 40–80M. We report the measured minima as the optimum and use the fit for the two-regime structure and the L* interpolation.
Corpus: v2026-05-21 clustered99, 7,044,237 train / 50,000 eval sequences, MLM 15% masking,
max length 320. All losses are held-out eval loss at fully-decayed LR, identical eval set and masking across runs.
