techlist.io
KREN
cloudflare

Redirects for AI Training enforces canonical content (opens in new tab)

cloudflareai-trainingrobots.txthttp-301noindexwrangler-clicanonical-tagsbots

Cloudflare argues that AI training crawlers often ignore deprecation notices, noindex, and canonical tags, causing outdated documentation to enter training data. Its new Redirects for AI Training feature converts qualifying canonical tags into HTTP 301 redirects for verified AI training bots, sending them to current content instead. The feature is intended to enforce content freshness without affecting human visitors, search crawlers, or AI agents.

The Problem with Deprecated Content

  • Cloudflare’s older Wrangler documentation includes deprecation banners, noindex, and canonical tags.
  • AI training crawlers consumed deprecated and current documentation at roughly the same rate.
  • Unlike humans, crawlers may ingest the entire page, treating warnings as ordinary text.
  • Blocking crawlers with robots.txt creates a dead end and does not tell them where the replacement content is.
  • Outdated content can persist in trained models even after the original page has been updated.

Canonical Tags as Redirect Instructions

  • The HTML <link rel="canonical"> tag identifies the authoritative version of a page.
  • Canonical tags appear on an estimated 65–69% of websites and are often generated automatically by CMS platforms.
  • Cloudflare’s feature uses this existing content hierarchy rather than requiring separate redirect rules.
  • For verified AI training crawlers, a non-self-referencing canonical becomes an HTTP 301 Moved Permanently redirect.

How Redirects for AI Training Works

  • Cloudflare identifies eligible requests using cf.verified_bot_category.
  • The AI Crawler category includes bots such as GPTBot, ClaudeBot, and Bytespider.
  • Cloudflare inspects the requested page’s HTML:
    • If it has a canonical URL on the same domain, the crawler is redirected there.
    • Self-referencing canonicals do not trigger redirects.
    • Cross-origin canonicals are excluded.
  • Human traffic, search crawlers, and AI Assistant or AI Search bots are unaffected.

Limitations and Alternatives

  • The feature cannot remove outdated material already ingested into training datasets.
  • It does not cover unverified crawlers.
  • AI Agents visiting deprecated pages are not redirected.
  • Traditional redirect rules can work for a small number of paths, but they require manual maintenance, user-agent tracking, and rule capacity.
  • Canonical-based enforcement stays aligned with content changes automatically.

Cloudflare’s Documentation Test

  • In March 2026, legacy Workers documentation received thousands of crawls from OpenAI, Anthropic, and Meta.
  • An AI assistant provided the deprecated Wrangler syntax kv:key put instead of the current wrangler kv key put.
  • After enabling the feature, 100% of AI training crawler requests for pages with non-self-referencing canonicals were redirected during the first seven days.
  • Cloudflare expects this to improve future AI answers, though the impact depends on training pipelines and recrawl timing.

Enabling the Feature

  • In the Cloudflare dashboard, go to AI Crawl Control → Quick Actions → Redirects for AI training and enable the toggle.
  • Path-specific controls are available through Configuration Rules and Cloudflare for SaaS.
  • Cloudflare also added AI crawler response-status analysis to Radar’s AI Insights page, covering 2xx, 3xx, 4xx, and 5xx responses.

Website owners with canonical tags can use the feature to automatically steer verified AI training crawlers toward current content. It is most useful for sites with frequently changing documentation and many deprecated URLs, but it should complement—not replace—ordinary redirects, access controls, and content maintenance.