Cloudflare cut 100TB of RAM by proving 90% of its hash ring did nothing

A performance engineer filed an ordinary ticket — the Pingora Backend Router, Cloudflare's internal load balancer, was using far more memory than expected, and the memory sat in structures belonging to pingora-ketama, their open-source consistent-hashing library. The investigation that followed ended with more than 100TB of RAM handed back across the fleet, on top of another 100TB the DNS team had recovered a month earlier.

Consistent hashing is the technique that lets a request find the same server every time without reshuffling everything when a machine joins or leaves. Servers and cache keys are both hashed onto the same number line, and a task goes to the first server to its left. To keep the load even, each server is not one point but many, and the number of points is scaled by a weight — at Cloudflare, disk space, because the workload is storage-bound. Their base was 160 hash points per server multiplied by that weight, which for a typical machine worked out at 160 × 625 = 100,000 points.

The number nobody had checked

The interesting part is not the fix, it is what the arithmetic said about the existing configuration. Error in the distribution falls off with the number of hash points, but it falls off logarithmically: going from 10,000 to 100,000 points per server buys a 0.7% reduction in error. Ninety thousand points per server were paying for less than one percent.

Then the model met the implementation. The clean maths assumes a continuous ring; production uses 32-bit hashes, and in a data centre with 2,048 servers the collisions between 10,000 and 100,000 points per server mean contributions get dropped at random — so past a certain point, adding hashes actually makes the distribution less predictable, not more. The team cut the number of points per server by 90% and measured no appreciable change in balance.

Then the Rust part

The second saving came from the shape of a single struct. A hash point is a 32-bit hash plus a 16-bit server index — six bytes of information that the compiler was happy to pad out. Storing it as a raw byte array with accessors instead:

struct Point([u8; 6]);
impl Point {
    fn hash(&self) -> u32 { u32::from_ne_bytes(self.0[0..4].try_into().unwrap()) }
    fn index(&self) -> u16 { u16::from_ne_bytes(self.0[4..6].try_into().unwrap()) }
}

That alone took 25% off the memory used for consistent hashing, and both forms compile to the same code. The rollout ran both rings side by side, comparing routing decisions and origin traffic until the migration hit 100%, and the memory graph shows the drop on the day the old ring was finally decommissioned.

What it means

The expensive setting was a default nobody had re-derived. 160 points per server is a number with a long pedigree — it comes from the original ketama implementation — and it had been multiplied by a weight that made it enormous without anyone asking what the extra points were buying. That is the shape to look for in your own systems: a constant chosen for a small deployment, scaled linearly into a large one, defended by nothing but the fact that it works.

Logarithmic returns are the tell. Whenever accuracy improves with the logarithm of some resource, the last order of magnitude is almost free to give up. Most teams never plot that curve, because the setting is not in the hot path and nobody is being paged about it.

And the theory has a hardware edge. The maths stopped predicting reality at the point where 32-bit hash space started colliding — so the model was right and useless at the same time, and only simulation against the real integer width showed where the curve turned. If you are tuning something with a formula from a paper, check which of its assumptions your types quietly violate.

The changes are shipping in the pingora-ketama crate behind an as-yet unadvertised cargo feature: the v2 ring carries the compacted storage format, a faster sorting method, and the ability to run with far fewer points.