Cloudflare AI Search: give your agents a search engine for your data (opens in new tab)
Cloudflare AI Search now automates the components previously needed to build a search system, including crawling, ingestion, embeddings, vector storage, and retrieval. The update adds sitemap-free website discovery, public and custom-domain endpoints, MCP support, and integrations such as EmDash. Cloudflare is also previewing predictable pricing by including embedding and reranking costs when using select default Workers AI models. ## Easier Data Indexing - AI Search can index structured and unstructured data, including files and websites. - Website sources currently must be owned or verified through a Cloudflare zone. - The new **Discover** parsing mode crawls sites without requiring a sitemap by following links, powered by Browser Run’s `/crawl`. - A single AI Search instance can ingest, embed, and retrieve content from a website or collection of sites. ## Unified Search Across Multiple Sources - Multiple AI Search instances can be grouped into a namespace and queried together. - Users can enable public URLs to receive: - `/search` for search requests - `/mcp` for Model Context Protocol integrations - These endpoints can search across multiple websites or instances without requiring authentication. - A Worker can also bind to a namespace and perform a single multi-instance search, with results tagged and cited by source. ## Custom Domains and Access Control - Public endpoints can use custom domains such as `search.example.com/mcp`. - Cloudflare Access can be placed in front of these domains to make search private. - This supports both easily shareable public search and authenticated search for authorized users or agents. ## Pricing Model - Cloudflare is previewing a predictable, scalable pricing model for AI Search. - Embedding and reranking are included at no additional cost when using select default models from the Workers AI catalog. - Customers therefore do not need to estimate token usage for those operations. ## Cloudflare Dev Stack MCP Cloudflare uses AI Search to power its Dev Stack MCP server, which provides coding agents with current, cited documentation. - Separate instances index Cloudflare Docs, Blog, API Docs, Community, Astro, Vite, Vitest, Hono, Replicate, and OpenNext. - These sources can be combined because Cloudflare owns the underlying website data. - A Worker-based MCP server searches all relevant instances in one call. - Search results include citations and identify the originating instance. - Users can alternatively enable public namespace endpoints without writing or deploying a Worker. - The MCP server can be added to an agent with a configuration such as: ```json { "mcpServers": { "dev-stack": { "url": "https://stack.mcp.cloudflare.com/mcp" } } } ``` ## Additional Integrations - Cloudflare’s Blog, Developer Docs, and Cloudflare.com use AI Search internally. - The open-source EmDash CMS has an AI Search plugin that adds semantic search to site content. - AI Search is also used in Cloudflare’s own MCP and documentation experiences. For an existing application or MCP server, the Worker binding offers the most flexibility. For a quick, shareable search service, public `/search` and `/mcp` endpoints provide a simpler deployment path, with custom domains and Cloudflare Access available for branding and security.
