Cloudflare AI Search Gets a Managed Mode: Index, Search, and Serve Without the Plumbing

Cloudflare is folding its AI Search primitives into a managed product. Instead of stitching together Workers AI, AI Gateway, Vectorize, R2, and Browser Run, developers can now create an AI Search instance that handles crawling, ingestion, embedding, and retrieval automatically. The company also previewed pricing and added features like sitemap-less discovery, public endpoints, and custom domains.

What's new

The headline feature is the ability to index a collection of data—files or websites—into a namespace. For websites, you no longer need a sitemap; the new 'Discover' parsing option follows links to find pages, powered by Browser Run's /crawl. Each namespace can expose public /search and /mcp endpoints that query across all instances without authentication, and you can put a custom domain on top (e.g., search.example.com/mcp). Cloudflare Access can be layered on for private search.

Cloudflare also released an AI Search plugin for EmDash, its open-source CMS, so sites built on EmDash get semantic search out of the box. The company is using AI Search on its own properties: the Blog, Developer Docs, and Cloudflare.com all run on it now, using hybrid search (semantic + keyword).

Dev Stack MCP: a concrete use case

The new Cloudflare Dev Stack MCP server is built on AI Search. It indexes 10 Cloudflare-owned surfaces—Docs, Blog, API Docs, Community, Astro, Vite, Vitest, Hono, Replicate, OpenNext—into separate instances, then fans out a single query across all of them from a Worker-bound namespace. The result is cited, current answers for coding agents, replacing the slow, token-heavy fallback of web search plus page fetching.

You can also skip the Worker entirely: enable public URLs on the namespace and get /search and /mcp endpoints that query every instance with zero code.

Pricing preview

AI Search is free during beta, but Cloudflare shared preview pricing: ingestion at $0.75 per 1M tokens (plus $0.50 for image processing), storage at $2.00 per GB-month, semantic queries at $0.75 per 1k, full-text at $0.10 per 1k. Embedding and reranking are free when using select Workers AI models—a deliberate move to make costs predictable.

Bot policy compliance

AI Search identifies itself with the user agent 'Cloudflare-AI-Search', follows robots.txt, and respects bot controls—a nod to the growing tension between AI crawlers and site owners.