Cloudflare's New AI Traffic Taxonomy: Search, Agent, Training — and New Defaults

Cloudflare is updating its AI bot controls with a more nuanced taxonomy that separates crawlers into three behavioral categories: Search, Agent, and Training. The move comes a year after the company first gave website owners a one-click "Block AI Bots" option and a pay-per-crawl marketplace. The new system, announced today, applies to all Cloudflare customers, including Free tier users.

Why the change?

The old all-or-nothing approach to AI bots no longer fits. Website owners want to block training crawlers that permanently absorb content into models, but still allow search bots that drive referral traffic. They also need to handle real-time agents (like ChatGPT-User or browser-use agents) differently from batch crawlers. Cloudflare's new classification aims to give site owners that granularity.

The three categories

Search: bots that index content to answer queries later. Site owners should expect referral traffic or compensation in return.
Agent: real-time automated behavior acting on a person's behalf — chat fetch bots, browser-use agents. Often a human is waiting on the other end.
Training: crawlers that take content to train or fine-tune models. Data is permanently absorbed into the AI's architecture.

Cloudflare strongly encourages bot operators to separate their crawlers by purpose, so site owners can manage access transparently. Multi-purpose crawlers (e.g., Googlebot, Applebot, BingBot) will be classified under all applicable categories.

New defaults starting September 15, 2026

For all new domains onboarding to Cloudflare, Training and Agent bots will be blocked by default on pages that display ads. Search bots remain allowed by default. The rationale: ad-supported pages signal that human attention is the monetization goal, and Training/Agent bots interfere with that. Search, however, funnels visitors back to the site.

Multi-purpose crawlers that combine Search with Training will be subject to the most restrictive rule — so if a site blocks Training, Googlebot (which does both) will be blocked entirely. Existing customers can opt out of these defaults via Security settings before the deadline.

BotBase: enterprise visibility

Enterprise Bot Management customers get a new dashboard feature called BotBase — a searchable database of all known verified bots and agents, with behavioral classifications. It allows filtering by bot, copying detection IDs for use in Security rules, and will later become a control center for managing automated traffic.

The taxonomy also includes additional behaviors like Transact, Data Collection, Security Testing, SEO, Ads Verification, Social/Link Preview, Feed Fetching, and Monitoring & Operations — but only the three AI categories (Search, Agent, Training) are configurable for all customers today.

"We want a classification system that is scalable and representative of the world of automated traffic as it evolves."

The update reflects a broader industry shift: AI is no longer a single threat vector, and site owners need surgical controls, not just a block-all switch.