Cloudflare Splits AI Crawlers Into Search, Agents, and Training
Cloudflare's new AI traffic controls let site owners manage search crawlers, user-directed agents, and training crawlers separately, with new defaults coming in September.
The web's bot problem used to be easier to name. A crawler indexed pages, a site received search traffic, and both sides could argue that the exchange was roughly fair.
AI broke that simplicity. The same automated visit can now mean very different things: a search indexer discovering content, a user-directed agent fetching a page to complete a task, or a training crawler absorbing text into a model. Treating all of those as one category called "AI bots" is no longer precise enough.
On July 1, 2026, Cloudflare announced new AI traffic controls for all customers, including Free tier users.1 Instead of a single broad switch, site owners can now manage three AI-related use cases separately: Search, Agent, and Training crawlers.1
Cloudflare defines Search as behavior that collects or indexes content so it can answer questions later. Agent traffic is automated behavior acting in real time on a person's behalf, such as a chat fetch bot or a browser-use agent. Training is crawling that takes content to train or fine-tune a model.1
That distinction matters because the incentives are not the same. Search can still send visitors back. A user-directed agent may be doing exactly what a human asked it to do. Training crawlers, by contrast, may create long-term model value without returning attention, attribution, or payment to the source site.
The default is changing
The most concrete change is scheduled for September 15, 2026. For new domains onboarding to Cloudflare, Training and Agent categories will be blocked by default on pages that display ads, while Search will remain allowed by default.1
Cloudflare says multi-purpose crawlers will be evaluated according to all of their behaviors. If a crawler combines Search with Training, the most restrictive applicable rule wins.1 The company names Googlebot, Applebot, and BingBot as examples of multi-purpose crawlers that could be blocked for customers who choose to block Training.1
That is a meaningful shift away from the older model where "verified bot" often implied default access. Cloudflare says the Verified label now makes a bot allowable within its relevant category rather than automatically allowed everywhere.1
For site owners, the practical point is not that every AI request should be blocked. It is that policy can now be closer to intent. A publisher might allow search indexing, block training, and still permit certain real-time agents. A SaaS app might want user-directed agents to complete workflows, but not want competitive data collection or model training.
Content use becomes part of the policy language
Cloudflare is also testing a new use signal for robots.txt through its Content Signals work.1 The new values express how a bot may use content after access:
immediate: interact, but store and reuse nothingreference: index, excerpt, and link backfull: summarize and reproduce
Cloudflare says customers using managed robots.txt will now get use=reference added to the existing preference that search is allowed and AI training is not.1 Like other robots.txt entries, this is a preference signal rather than an enforcement mechanism by itself. Enforcement still depends on Cloudflare's traffic controls and bot classification.
The larger pattern is clear. The web is gaining machine-readable policy layers for automated readers. Robots.txt answered "may you fetch this path?" These new signals try to answer harder questions: why are you here, who are you acting for, and what will you do with the content afterward?
Why developers should care
This is not only a publisher story. Any developer building on Cloudflare, Nuxt, Vue, or another edge-heavy stack now has to think about agent traffic as a first-class production concern.
Public documentation, marketing pages, pricing pages, changelogs, APIs, and MCP endpoints may all be read by agents. Some of that is valuable. Some of it is risky. A blunt block can make a product invisible in AI-driven discovery. A fully open policy can invite scraping, training, or automated abuse.
Cloudflare's July update does not solve the entire trust problem. It depends on classification quality, operator honesty, and the still-emerging norms around content signals. But it is an important step because it replaces a vague "AI bot" switch with a vocabulary that matches how the web is actually being used in 2026.
The agentic web is not just about agents becoming more capable. It is also about sites becoming more explicit about the terms under which agents, crawlers, and search systems may participate.