Cloudflare’s AI Platform: an inference layer designed for agents (opens in new tab)

Cloudflare is positioning AI Gateway as a unified inference layer for agentic applications. It provides one API for models from multiple providers, centralized cost and reliability controls, and low-latency access through Cloudflare’s global network. The platform is also expanding toward customer-hosted models and multimodal inference, reducing dependence on any single AI vendor.

A Unified Model Catalog and Endpoint

• Developers can call Cloudflare-hosted and third-party models through the same env.AI.run() interface.
• Switching models, such as from Workers AI to Anthropic or OpenAI, requires only a one-line change.
• Cloudflare plans to add REST API support for applications that do not run on Workers.
• The catalog includes more than 70 models from over 12 providers, spanning:
  • Text and reasoning models
  • Image, video, and speech models
  • Open-source models hosted on Workers AI
  • Proprietary models from major providers
• Users pay through one credit system rather than managing separate provider accounts.

Centralized Cost and Usage Management

• AI Gateway gives companies a consolidated view of spending across providers.
• Request metadata can categorize costs by:
  • Team
  • User
  • Customer
  • Workflow
  • Free versus paid usage
• This is particularly useful for agents, which may make many model calls for a single user request.

Bringing Custom Models to Workers AI

• Cloudflare is working on allowing customers to deploy fine-tuned or specialized models through Workers AI.
• The deployment workflow uses Replicate’s Cog technology to package models and manage:
  • Python dependencies
  • CUDA requirements
  • Model weights
  • Runtime configuration
• Developers define the environment in cog.yaml and implement model setup and inference in predict.py.
• After building the Cog container, Cloudflare can deploy and serve it through existing Workers AI APIs.
• Planned improvements include customer-facing APIs, Wrangler commands, and GPU snapshotting to reduce cold-start times.

Optimizing Agent Responsiveness

• For live agents, time to first token can matter more than total inference time.
• A small reduction in startup latency can make an agent feel significantly more responsive, even if the full answer still takes several seconds.
• Cloudflare’s presence in 330 cities places gateways close to users and inference endpoints, reducing network delay before streaming begins.
• Workers AI is adding agent-focused open-source models, including Kimi K2.5 and real-time voice models.

Cloudflare’s direction is to make model selection, deployment, monitoring, and failover easier through a single infrastructure layer. For teams building agents, the platform offers a way to mix providers and custom models while controlling cost, latency, and operational complexity.