Cloudflare’s AI Platform: an inference layer designed for agents (opens in new tab)
Cloudflare is positioning AI Gateway as a unified inference layer for agentic applications. It provides one API for models from multiple providers, centralized cost and reliability controls, and low-latency access through Cloudflare’s global network. The platform is also expanding toward customer-hosted models and multimodal inference, reducing dependence on any single AI vendor. ### A Unified Model Catalog and Endpoint - Developers can call Cloudflare-hosted and third-party models through the same `env.AI.run()` interface. - Switching models, such as from Workers AI to Anthropic or OpenAI, requires only a one-line change. - Cloudflare plans to add REST API support for applications that do not run on Workers. - The catalog includes more than 70 models from over 12 providers, spanning: - Text and reasoning models - Image, video, and speech models - Open-source models hosted on Workers AI - Proprietary models from major providers - Users pay through one credit system rather than managing separate provider accounts. ### Centralized Cost and Usage Management - AI Gateway gives companies a consolidated view of spending across providers. - Request metadata can categorize costs by: - Team - User - Customer - Workflow - Free versus paid usage - This is particularly useful for agents, which may make many model calls for a single user request. ### Bringing Custom Models to Workers AI - Cloudflare is working on allowing customers to deploy fine-tuned or specialized models through Workers AI. - The deployment workflow uses Replicate’s Cog technology to package models and manage: - Python dependencies - CUDA requirements - Model weights - Runtime configuration - Developers define the environment in `cog.yaml` and implement model setup and inference in `predict.py`. - After building the Cog container, Cloudflare can deploy and serve it through existing Workers AI APIs. - Planned improvements include customer-facing APIs, Wrangler commands, and GPU snapshotting to reduce cold-start times. ### Optimizing Agent Responsiveness - For live agents, time to first token can matter more than total inference time. - A small reduction in startup latency can make an agent feel significantly more responsive, even if the full answer still takes several seconds. - Cloudflare’s presence in 330 cities places gateways close to users and inference endpoints, reducing network delay before streaming begins. - Workers AI is adding agent-focused open-source models, including Kimi K2.5 and real-time voice models. Cloudflare’s direction is to make model selection, deployment, monitoring, and failover easier through a single infrastructure layer. For teams building agents, the platform offers a way to mix providers and custom models while controlling cost, latency, and operational complexity.