A load generator inside the tested JVM hides half or more of GC tail latency
Victoria Shinder2 min readA benchmark that runs its load generator inside the same JVM as the system it measures reports tail latencies two to three times lower - that is, better-looking - than the same system measured from a separate JVM, according to a study by Jonas Norlinder published on 25 September and featured on inside.java. The experiment used SPECjbb2015 v1.04 on OpenJDK 27, and Norlinder stresses that it is a research exploration of p99 response times, not a compliant benchmark submission.
The mechanism is coordinated omission. A load generator that waits for each response before sending the next request stops generating load whenever the server stalls. SPECjbb2015 corrects for this in the usual way, by measuring from the time a request was scheduled rather than when it was actually sent. But that correction assumes the generator can still schedule requests. If the generator lives in the same JVM and a garbage-collection pause stops the whole process, it cannot, and the requests that real users would have sent during the pause never exist.
Norlinder compared the Composite-Net mode, with everything in one JVM, against Distributed mode on the same machine, giving the distributed setup more memory and splitting CPU cores so neither configuration was starved. Injection rate was fixed at 6,000 requests per second, with ten separate JVM runs per configuration. For the collectors with non-trivial pauses the gap was roughly two to three times; for ZGC, whose pauses are under a millisecond, there was no gap. The absolute numbers are stark: 34 ms in one JVM against 100 ms measured separately in one case, and with Serial GC on a 6 GB heap, 278 ms against 1,750 ms. Timestamps matched against GC logs showed no requests being scheduled in the single-JVM mode during a pause.
What it means
The lesson reaches well beyond one benchmark. Any latency test in which the thing generating load can be paused by the thing being tested - a harness in the same process, a load tool sharing a container's CPU quota, a test client on an overloaded node - will under-report exactly the stalls it is supposed to catch. The result looks precise and is systematically optimistic.
Two practical rules follow. Put the load generator in its own process, ideally on its own cores, whenever tail latency is the question; SPECjbb2015 already offers MultiJVM and Distributed modes for this, and the author recommends only those for latency work. And be suspicious of any comparison of garbage collectors run in-process: the collector with the longest pauses benefits most from the generator pausing too.