Two launches, two kinds of benchmark: one you can rerun today, one you take on trust until month-end

Victoria Shinder06 October 2026, 22:431 min read

Two launches on 6 October came with performance claims, and they show the two ways such claims reach readers.

Polars 2.0 published SQL benchmarks against DuckDB and DataFusion and, in the same post, the conditions that produced them: machine types, engine versions, best of five hot runs, a 60-second timeout, how the data was generated, which queries were excluded and why. It also named a weakness of its own, a constant overhead at 192 threads, and published a repository so anyone can rerun the tests. The numbers can be wrong, but they can be checked.

Mistral Large 4 arrived with impressive figures on coding and security tests, and a public preview API. The weights, the architecture details and the post-training method are promised for later in the month. Until then the numbers are the vendor's, and the explanation for one of the most striking results (that several closed models refuse a vulnerability-reproduction task and so score near zero) is also the vendor's.

What to do with that difference. Neither is a reason to dismiss a release. But they deserve different weight in a decision:

• A claim with a method and code can be tested on your own hardware and data before you change anything. That is what a benchmark is for.
• A claim without them is a statement of intent. It is useful for deciding what to evaluate, not for deciding what to adopt.
• The date matters. Open weights at the end of October turn Mistral's numbers into the first kind. The useful move now is to note which tests matter for your work and run them when the weights are out.

A benchmark you cannot rerun is an advertisement until someone reruns it. Polars invited that on day one; Mistral has promised to.