Open your robots.txt next week and there may be a block at the top you did not write. Cloudflare announced Bot Preference Sync on 21 August 2026, rolling out within a week to every plan tier including Free: you set a policy per bot category in the dashboard, and Cloudflare keeps the matching rules current in the file your site serves. On by default for new customers. Which is mostly welcome, honestly, because hand maintaining a list of AI crawler tokens is a chore nobody signed up for. Two things in that post deserve more attention than the automation, though. One is a rule in RFC 9309 that quietly changes what the rest of your own file still covers. The other is not about robots.txt at all, and it is the reason this announcement matters more than it reads.
The short answer
Cloudflare will keep your robots.txt in step with the AI bot policy you set
in the dashboard, on every plan including Free, on by default for new
customers. Useful. But it prepends its block above your own rules, and under
RFC 9309 a bot named up there stops reading the wildcard group below. Check
what you actually serve after you turn it on.
Image: Cloudflare
What it writes
Three categories, and they are not the same shape. Search covers collecting and indexing your pages so a system can answer questions about them later. Agent covers automation acting in real time for a person who asked for something. Training covers crawlers gathering material to train or fine tune a model.
Search and Agent each get three settings: allow, block on ad supported pages, or block everywhere. Training is binary. Allow or Disallow.
Whatever you pick lands in your file as a block wrapped in # BEGIN Cloudflare Bot Preference Sync and a matching # END. It gets prepended, so everything you already serve stays put underneath. The user agent list behind it refreshes from Cloudflare’s BotBase tracker, which is the actual selling point: a crawler that appears next month and lands in a category you blocked is covered without you touching anything.
If you have no robots.txt at all, Cloudflare generates one.
The bit that will surprise people
Here it is, and it has nothing to do with Cloudflare being careless. It is how the standard works.
RFC 9309 section 2.2.1: a crawler finds the group matching its own product token and obeys that group’s rules. It uses the group with a * user agent only if no matching group exists. Groups do not inherit. There is no cascade.
So the moment a bot is named in that block at the top of your file, your User-agent: * group below stops applying to it. Every path disallow you had in there. Gone, for that bot. It reads four lines and stops.
Most of the time this is harmless, because the managed block usually says Disallow: / and there is nothing left to inherit. It stops being harmless when you allow a category. Allow Search for a mixed purpose crawler, and that crawler now reads a group written by Cloudflare rather than the one you wrote, and your staging path or your faceted search trap is not in it.
Two minutes of work, once: curl https://yourdomain.com/robots.txt and read the whole thing top to bottom. Not the dashboard. The file. Our HTTP Headers Checker will tell you what the edge is actually returning if you want to confirm you are not looking at a cached origin copy.
The four conditions, which are the real news
Buried under the automation story is something with far more weight. Cloudflare has written down what a mixed purpose crawler, one doing Search and Training at once, has to do to keep access to a site that disallows training.
Four conditions. It must respect a no training preference in robots.txt by any mechanism. It must give site owners a way to opt out of AI summaries. It must provide URL level visibility into which pages were made available for training, plus metrics on search results. And it must show publicly that disallowing training does not hurt your traditional search results.
Miss any of them and you are blocked wherever training is disallowed. Cloudflare’s wording: crawlers that don’t provide transparency will not get the benefit of the doubt.
Read that list again. The third and fourth conditions are the ones with teeth, because almost nobody publishes per URL training visibility and almost nobody has publicly demonstrated that a training opt out leaves search rankings untouched. That is a high bar, set unilaterally, by a company sitting in front of a large slice of the web. No working group, no RFC, no published appeal process.
I might be wrong about how this lands, but it looks less like a robots.txt feature and more like a CDN writing the terms of the AI crawling debate while the standards bodies are still drafting.
Does any of it work
Partly. That’s the honest answer.
Cloudflare’s own documentation says it: robots.txt compliance is voluntary, and it does not technically prevent access. Enforcement is a different product, AI Crawl Control, and the docs say so directly.
TollBit put a number on the gap. Around 15 percent of identified AI page fetchers in Europe reached URLs that sites had marked disallowed. ChatGPT-User, Bytespider and Youbot each hit disallowed pages on close to half the European sites naming them. The ChatGPT-User case is the one worth understanding rather than being angry about, because OpenAI documents it: robots.txt may not apply when a human initiated the fetch. That is a definitional disagreement about what a user agent is, not a bug, and no amount of dashboard automation resolves it.
So the sync makes your preference legible and current. It doesn’t make it binding.
Before you disallow everything
One number that deserves to travel with this story. Zhao at Rutgers and Berman at Wharton studied news publishers who blocked LLM crawlers through robots.txt and found roughly a 7 percent drop in weekly visits within six weeks, visible in Comscore’s human browsing panel rather than only in bot counts. Real people, not stripped out bot traffic.
The caveat matters as much as the finding. Their window runs November 2022 to May 2024, stopping just before AI Overviews shipped. The referral economics have moved since. We’d treat 7 percent as evidence that the tradeoff is real and measurable, not as a forecast for your site in 2026.
If you run an ad supported site, that tradeoff is exactly what Cloudflare’s ad supported onboarding default assumes you have already made. Worth deciding on purpose. The same argument played out on Twitch this month, where the AI training toggle went on by default and covered chat, and the people affected mostly found out afterwards.
What we’d do
Turn it on. It solves a real maintenance problem and the category model is better than chasing user agent strings.
Then fetch the file. Read it as a crawler would, group by group, and check that anything you had relied on in your wildcard group is either duplicated where it needs to be or genuinely irrelevant. If you allow Search or Agent for anything, that check is not optional.
And keep the expectations calibrated. This is a signal, published well and kept current. The enforcement question is somewhere else entirely.
Sources
Cloudflare’s announcement, Say it once: introducing Bot Preference Sync, published 21 August 2026, and the managed robots.txt developer documentation. Independent coverage and the transparency criteria breakdown from PPC Land, plus its report on the TollBit measurement of AI fetchers reaching disallowed URLs. Group selection rules quoted from RFC 9309. The traffic study is Strategic Response of News Publishers to Generative AI by Hangcheng Zhao and Ron Berman.
Frequently asked questions
What is Cloudflare Bot Preference Sync?
A feature announced on 21 August 2026 that writes AI crawler rules into your robots.txt from the settings you pick in the Cloudflare dashboard. You choose a policy for three categories, Search, Agent and Training, and Cloudflare maintains the matching block in the file your site serves, refreshing the user agent list from its own BotBase tracker as new crawlers appear. It is available on every plan tier including Free, and it can be switched off at any time.
Is Bot Preference Sync on by default?
For new customers, yes, the sync itself defaults to on. That is not the same as being blocked by default. A non-publisher customer gets no blocks applied at onboarding, so the managed block starts empty in practice. Ad supported publishers can pick an onboarding option that sets Training to Disallow. Existing customers on the older managed robots.txt feature get prompted to review and confirm their preferences during the transition rather than being switched silently.
Does Bot Preference Sync actually stop AI crawlers?
No, and Cloudflare does not claim it does. Its own documentation states plainly that robots.txt compliance is voluntary and points at AI Crawl Control for enforcement at the network layer. The sync publishes a preference in a machine readable place. TollBit measured roughly 15 percent of identified AI page fetchers in Europe reaching URLs that sites had marked disallowed, so treat the file as a statement of intent rather than a control.
Will adding a named bot to robots.txt break my existing rules?
It can change which rules apply to that bot, yes. RFC 9309 section 2.2.1 says a crawler finds the group matching its own product token and obeys that group, falling back to the group with a star user agent only if no matching group exists. Groups do not cascade. So once a crawler is named in the managed block at the top of the file, your wildcard group underneath no longer applies to it, including any path disallows you had relied on. Fetch your own robots.txt and read it top to bottom after enabling the sync.
Does blocking AI training crawlers cost traffic?
There is measured evidence that it can. A paper by Hangcheng Zhao of Rutgers Business School and Ron Berman of The Wharton School found news publishers who blocked LLM crawlers via robots.txt lost roughly 7 percent of weekly visits within six weeks, and the drop showed up in Comscore human browsing panel data rather than only in bot counts. Read the window before you act on it: the study covers November 2022 to May 2024, which ends just before Google rolled out AI Overviews, so the referral landscape it measured is not today.