Cloudflare will score Verified bots on whether they obey robots.txt and back off on 429

Serguey Asael Shinder17 October 2026, 08:322 min read

Cloudflare described on 16 October 2026 a system that keeps checking how Verified bots behave after they have been admitted to the programme. Until now, verification answered the question "are you who you say you are?". The new system is meant to answer the second one Cloudflare sets for every bot, "can we trust you?", and to keep answering it over time.

Identity is already handled. Bots in the programme sign their requests with Web Bot Auth, and BotBase catalogues those identities in an open format. Cloudflare's point is that identity does not guarantee behaviour: a known bot can still ignore a site's preferences, overload an origin or try to get around controls, often because of a misconfiguration or a bug rather than malice. The post distinguishes risk, the chance that a single request is harmful, which changes from moment to moment, from trust, which is built through consistent behaviour. As a precedent it notes that Cloudflare removed Perplexity's bot from the Verified list in 2025 after it tried to reach sites that had opted out of scraping and then attempted to circumvent the blocks.

Two initial signals.

• robots.txt. The system checks whether a bot's observed traffic matches the preferences a site publishes, in line with RFC 9309. Isolated violations are not what it looks for; repeated patterns of ignoring those preferences are. Cloudflare adds that robots.txt sets a minimum: anything it does not forbid is not thereby allowed, and a site can signal that further access is unwelcome in other ways.
• HTTP 429. A bot can stay on allowed paths and still put heavy load on a site. The system checks whether a bot slows down or pauses after "Too Many Requests" responses, or keeps sending at the same rate. Backing off consistently raises trust; ignoring the signal repeatedly lowers it.

Trust bands, not verdicts. The observations are turned into an explainable trust band: a range showing how consistently a bot respects site preferences and which behaviours led to that assessment. Cloudflare stresses that it is not a permanent judgement about a bot or its operator, and that it looks at sustained patterns rather than single events.

What is live and what is planned. The system is in use internally to evaluate Verified bots and build up history. Cloudflare plans to publish the scores on Radar, to add more ways of measuring behaviour, and to make the system available to all customers, who will be able to choose the trust threshold that decides how Verified bot traffic is handled. It sits alongside two existing products: Bot Preference Sync, which aligns published preferences with configured policies, and Bot Management, which decides what happens to the traffic. No dates are given for the public scores or the customer setting.

Source: Cloudflare Blog, "Continuous accountability for Verified bots" — https://blog.cloudflare.com/verified-bots-accountability