Scaling Laws Revisited: What We Learned Training Bloom 4.6

Every time Bloom Labs sits down to plan a new model, the first fight in the room isn't about architecture — it's about the compute budget spreadsheet. When we started training Bloom 4.6, our balanced mid-tier model, we had enough GPU-hours allocated to go one of two ways: pour it all into parameters and ship a bigger model, or split it between parameters and training data and ship a smarter one. We chose smarter. Here's the math behind that call, and why we think it's the right one for a model that's meant to sit in production answering millions of requests a day, not just win a leaderboard.

The Compute-Optimal Frontier

Classic scaling-law research treats training compute as a fixed budget C, roughly proportional to parameter count N times training tokens D (C ≈ 6ND). For any fixed C, there's a frontier of (N, D) pairs that minimize training loss, and it is emphatically not the pair you get by maximizing parameters alone. Early in the field, a lot of teams over-indexed on parameter count because bigger numbers make better press releases. We did the opposite calculation for Bloom 4.6: we fixed the budget, then swept dozens of candidate (N, D) pairs at smaller scale, measured loss, and extrapolated the curve up to our real training budget before committing a single dollar of compute to the full run.

The result surprised even some of our own engineers: the compute-optimal parameter count for our budget was noticeably smaller than what our first draft plan called for. We had originally spec'd a model nearly 40% larger. The swept curves said that extra weight would have been better spent as additional training tokens.

Why We Cut Parameters and Added Data

• Diminishing returns on width. Past a certain point on our loss curves, adding parameters bought less loss reduction per FLOP than adding tokens did. The crossover for our target budget landed well below where we'd originally planned to build.
• Inference cost compounds. Bloom 4.6 is being engineered to be the workhorse model — the one we expect will eventually handle the majority of day-to-day Bloom Labs traffic once it's released. Every extra billion parameters is a permanent tax on every single inference call, forever. A smaller, better-trained model wins on total cost of ownership even if its raw parameter count looks less impressive on paper.
• Data quality had headroom left. We weren't anywhere near saturating what our curated training corpus could teach a model of this size. Under-training a large model on too few tokens leaves capability on the table that a smaller, longer-trained model can capture instead.
• Serving latency is a product feature. A chunkier model would have meant a slower time-to-first-token in production. For a mid-tier model that needs to feel instant, that tradeoff wasn't close.

So we rebalanced: fewer parameters than our first draft, meaningfully more training tokens, same total compute envelope. The chart below shows how that reallocation played out across our internal loss-efficiency benchmark, comparing the parameter-heavy draft plan against the data-heavy plan we ultimately committed to for Bloom 4.6.

"Bigger Isn't Always Better" — What That Actually Means

We want to be precise about this claim, because it's easy to misread. We are not saying scale doesn't matter — Homan 2.96, our flagship, is proof that pushing scale aggressively still produces the strongest internal results Bloom Labs has ever measured when the budget and use case call for it. What we're saying is narrower: for a fixed compute budget aimed at a mid-tier, high-volume production model, the optimal allocation of that budget is rarely "as many parameters as will fit." It's whatever point on the (N, D) frontier minimizes loss for that specific budget, and for Bloom 4.6, that point called for restraint on size and aggression on data.

This also explains why Bloom 4.6 punches well above its parameter count on many of our internal evals compared to older, larger models from earlier training runs. It isn't magic — it's a longer, better-fed training run on a right-sized architecture.

What's Next

We're now running the same sweep methodology against candidate configurations for our next-generation lineup, and we're extending the analysis to account for inference-time compute (test-time reasoning) as a third axis alongside parameters and training tokens. Early results suggest the frontier shifts in interesting ways once a model is allowed to "think longer" at inference time instead of only getting bigger at training time. We'll have more to share on that in a future post.

Bloom 4.6 has completed internal development and testing, but it hasn't been released publicly yet. Every number in this post — the sweeps, the bench-row results above, the projections about the traffic it's meant to eventually carry — comes from Bloom Labs' internal infrastructure, not from outside usage. Given how capable this model is, we're being deliberate about when it goes out the door: our safety team is running a full risk review, and Bloom 4.6 won't ship publicly until we're confident it can't cause harm. In the meantime, here's a closer look at what this training approach produced.

Comments (9)