Introducing the Agent Readiness score. Check to see if your site is agent-ready (opens in new tab)
Cloudflare argues that websites must evolve beyond browser and search-engine compatibility to become usable by AI agents. Its new Agent Readiness score evaluates whether sites support standards for discovery, content access, bot control, and agent capabilities. Early data shows adoption is extremely low, creating both a challenge and an opportunity for sites that adopt these standards early. ## Agent readiness across the web - Cloudflare analyzed the 200,000 most visited domains, excluding categories unlikely to need agent interaction. - The resulting Cloudflare Radar dataset tracks adoption of AI-agent standards and will be updated weekly. - robots.txt exists on 78% of sites, but most files target traditional search crawlers rather than AI agents. - Only 4% of sites declare AI usage preferences through Content Signals. - Just 3.9% support Markdown content negotiation via `Accept: text/markdown`. - MCP Server Cards and API Catalogs based on RFC 9727 appear on fewer than 15 sites, showing how early these standards remain. ## The Agent Readiness score Site owners can test their websites at **isitagentready.com**. Cloudflare scans the site and scores it across four dimensions: - **Discoverability:** robots.txt, sitemap.xml, and Link Headers under RFC 8288. - **Content:** Markdown for Agents. - **Bot Access Control:** Content Signals, AI-specific robots.txt rules, and Web Bot Auth. - **Capabilities:** Agent Skills, API Catalogs, OAuth discovery standards, MCP Server Cards, and WebMCP. - The tool also checks commerce standards such as x402, Universal Commerce Protocol, and Agentic Commerce Protocol, though these do not yet affect the score. - Each failed check includes a prompt that can be handed to a coding agent for implementation. The service itself supports agents through a stateless MCP server with a `scan_site` tool and publishes Agent Skills documents explaining how to implement each supported standard. ## Discoverability for AI agents - robots.txt helps agents understand crawl permissions and locate sitemaps. - Sitemaps provide a structured list of site paths, reducing the need to discover content by following every HTML link. - HTTP Link headers, defined by RFC 8288, expose important resources directly in responses without requiring agents to parse page markup. - Sites can use headers such as `rel="api-catalog"` to point agents toward machine-readable capabilities. ## Making content easier to read - `llms.txt` provides an LLM-oriented reading list at the site root, describing the site and linking to important content in a format designed for model context windows. - Markdown content negotiation lets agents request a clean Markdown version of a page with `Accept: text/markdown`. - Cloudflare measured token reductions of up to 80% compared with HTML, improving speed, cost, and the likelihood that agents can consume an entire document within their context limits. Cloudflare’s recommendation is to evaluate sites with the Agent Readiness tool and adopt the relevant standards incrementally. With current adoption so low, early support can make a site significantly easier for AI agents to discover, understand, authenticate with, and use.