Prometheus 3.15 scrapes over Unix sockets and reloads its log level live
Victoria Shinder2 min readPrometheus 3.15.0 was released on 25 September. Two of its new features change how the server is operated. Scrape targets can now be reached over Unix domain sockets, which suits exporters that should not listen on a network port at all. And the log level can now be set through a runtime.log_level configuration setting that takes effect on reload, so the --log.level flag is deprecated in favour of the configuration file.
Memory handling gets attention too. A new --auto-gomemlimit.refresh-interval flag makes Prometheus periodically re-detect the container or system memory limit and update Go's GOMEMLIMIT while running, rather than only at start-up. On the ingestion side, the release implements the OpenMetrics 2.0 scrape format, supports zstd-compressed scrape responses behind the zstd-scrape feature flag, and cuts CPU use when the TSDB head initialises. The XOR2 float chunk encoding is now stable: the xor2-encoding feature flag is deprecated in favour of storage.tsdb.chunk_encoding.floats: xor2, with a warning in the notes to check that other software reading the TSDB directly, such as a Thanos sidecar, supports XOR2 before enabling it.
One fix is worth reading closely. A range query whose end was not aligned to step caused subqueries inside it to evaluate past the parent's last actual step. That inflated peakSamples in query statistics and against the query.max-samples limit, and read samples from storage that never appeared in the result.
What it means
The log-level change is small but operationally welcome. Raising verbosity during an incident used to mean a restart with a different flag, which loses in-memory state and can itself change the behaviour being investigated. Making it a reloadable setting turns it into a configuration change. The memory-limit refresh addresses a similar gap for containers whose limits are changed in place, where a limit read only at start-up would leave the Go runtime tuned for the old value.
The subquery fix matters for anyone who has hit query.max-samples on dashboards with unaligned time ranges: some of those rejections were caused by samples the query did not need. And the XOR2 warning is the line to act on before upgrading storage settings - a format change that the server understands but a sidecar does not is the kind of mismatch that only shows up when data is read back.