← Back to feed

Cloudflare published a new control today that resolves a structural problem it identified three months ago: the mixed-use crawler tradeoff. Until now, a publisher blocking AI training from a crawler like Googlebot — which indexes for search and trains on content via the same crawl — also blocked search indexing. Block one, lose both.

The new setting, called Disallow AI Training, changes that. It publishes a no-training preference via a mechanism Cloudflare calls Bot Preference Sync, which writes the applicable directive to the site’s robots.txt. Mixed-use crawlers from Accountable operators — Apple, Google, and Microsoft — honour the training opt-out while continuing to index for search. Amazon, Anthropic, Meta, and OpenAI operate separate training-only crawlers; those are blocked outright under Disallow AI Training, which carries no search penalty because these companies’ training bots do not index for search.

The Scale of the Problem

Cloudflare’s own data puts the stakes into focus. Less than 1% of Cloudflare-managed sites block search bots. Seventeen percent choose to activate some mechanism to block AI training. The demand for training-specific controls exists at scale; the tooling to express it cleanly has not.

A robots.txt directive alone cannot enforce the preference. Anyone can publish one, but robots.txt cannot identify who is crawling, determine why, or stop a crawler that ignores it. Cloudflare’s position as network intermediary for a large fraction of web traffic gives it both the visibility and the enforcement layer that individual site operators lack.

Accountable Designation

Since July, Cloudflare has been negotiating directly with major bot operators. The result is the Accountable designation, which recognises operators that meet or commit to meet four requirements:

1. A mechanism for site owners to opt out of AI training, via robots.txt or equivalent.
2. A mechanism to opt out of AI summaries — directly with the operator now, through Cloudflare by early next year.
3. URL-level visibility into which pages were made available for training and how content appeared in search.
4. A guarantee that opting out of AI training will not affect traditional search results.

Apple, Google, and Microsoft qualify on the mixed-use crawler dimension. Cloudflare also categorises Amazon, Anthropic, Meta, and OpenAI as Accountable because they maintain separate training-only crawlers, distinct from any search indexing infrastructure — meaning opting out of their training bots does not touch search discoverability.

Three Crawler Behaviors, Four Settings

Cloudflare classifies bots by behavior: Search (building a search index), Training (feeding model pre-training or fine-tuning), and Agent (user-directed task automation). A mixed-use crawler exhibits both Search and Training.

The four available settings, applied domain-wide:

• Allow — all crawlers permitted unless blocked elsewhere.
• Disallow AI Training — Bot Preference Sync writes the robots.txt preference; Accountable mixed-use crawlers remain allowed for search; all other training crawlers are blocked, including training-only crawlers from Amazon, Anthropic, Meta, and OpenAI.
• Block on pages with ads — all crawlers including mixed-use blocked only on ad-serving pages.
• Block — all crawlers blocked.

The “Block” setting’s scope has also expanded. Previously it did not apply to mixed-use crawlers because blocking them risked losing search. Now that Disallow AI Training exists as a separate, more surgical option, Block applies to all training crawlers including mixed-use ones.

What Comes Next

AI summaries are the next control surface Cloudflare intends to address. A domain-level opt-out is too blunt — how much of a site’s content appears in an AI summary matters as much as whether it appears at all. Cloudflare says it will deliver granular AI summary controls set once on Cloudflare rather than negotiated separately with each operator, with a target of early next year.

Agents — automated processes visiting pages on behalf of users — are not included in the Disallow setting for now. Cloudflare cited the lack of an established robots.txt-equivalent directive for agents and said it will revisit the approach as standards such as the IETF’s ai-prefs working group mature.

Key Numbers

• 17% of Cloudflare sites block some form of AI training; less than 1% block search
• Accountable operators: Apple, Google, Microsoft
• Training-only crawlers blocked by Disallow AI Training: Amazon, Anthropic, Meta, OpenAI
• AI summary granular controls: targeted for early 2027
• Published: Cloudflare Blog, September 15, 2026