Ai Gateway | Techlist.io

cloudflareAug 7, 20263 min readCurated summary

Unifying Workers AI and AI Gateway into a single AI control plane

AI Gateway and Workers AI are converging into a unified control plane for accessing models across Cloudflare and external providers. A single Workers binding or REST API can now provide inference, observability, logging, security, and billing without requiring users to choose a product upfront. Cloudflare’s longer-term goal is model-first routing, where applications request capabilities or models while the gateway handles provider selection, failover, and load balancing. ## Unified Bindings and API - The Workers AI binding and AI Gateway now share the same entrypoint. - Requests can use the built-in `default` gateway or a named gateway for separate applications and customized policies. - The unified REST API routes requests through `/ai/` endpoints, using the `cf-aig-gateway-id` header. - This removes the need to decide between Workers AI and AI Gateway before building an application. ## Automatic Observability for Workers AI - Passing `default` as the gateway ID automatically creates an AI Gateway on the first authenticated request. - Requests receive built-in: - Full request and response logging - Token tracking by model - Cost attribution - Latency and error metrics - Developers can begin with the default gateway and later switch to a named gateway for features such as custom caching or application-specific traffic separation. - The AI Gateway dashboard provides detailed visibility into prompts, responses, latency, token usage, and failures. ## Unified Billing with AI Gateway Credits - AI Gateway credits can now pay for Workers AI usage in addition to providers such as OpenAI and Anthropic. - Users can maintain one prepaid credit balance across supported providers. - Workers AI users who use unified billing receive elevated rate limits, subject to current Cloudflare policies and documentation. ## Model-First Routing - Cloudflare plans to route requests based on the desired model rather than requiring users to select a specific provider. - The gateway could handle: - Provider selection - Failover - Load balancing - Capacity management - For example, a request for a model such as Kimi K2.7 Code could be served by Workers AI, the model’s original provider, or another vetted provider hosting the same weights. - Applications could remain available if one provider is overloaded or unavailable. - Users will still be able to restrict traffic to a single provider when necessary. - Routing is intended to preserve requirements such as Zero Data Retention and maintain model quality. Cloudflare recommends using the unified binding or REST API with the default gateway to gain observability and centralized billing immediately. As model-first routing develops, applications can rely less on provider-specific infrastructure and gain greater resilience through automatic provider management.

Read original

Ai Gateway Observability Rest Api Load Balancing+3

cloudflareAug 7, 20263 min readCurated summary

Introducing Radar Researcher: An AI tool for exploring Internet data in plain language

Cloudflare Radar Researcher is an AI-powered assistant that lets users explore Radar’s Internet data through natural-language questions. It replaces manual API queries, filters, and chart hunting with interactive visualizations, explanations, follow-up questions, and auditable analysis. The beta tool is designed for both casual users and technical researchers, while grounding responses in real data from Radar’s API. ## Why Cloudflare Built Radar Researcher - Cloudflare Radar publishes global Internet data covering: - DNS queries from the 1.1.1.1 resolver - HTTP traffic across Cloudflare’s network - Internet quality measurements from Cloudflare Speed Test - Network outages and other datasets - Radar serves a wide audience, from journalists and curious users to network operators and academics. - AI lowers the barrier to using complex datasets by allowing users to ask questions without knowing a dataset’s structure, terminology, or API. - The tool is especially useful for people working under time pressure, such as journalists investigating current Internet disruptions. ## Natural-Language Data Exploration - Radar Researcher is available from every Radar page in a docked panel. - Users can: - Ask questions in plain language. - Receive short answers or more detailed reports. - View real, interactive Radar charts alongside explanations. - Continue with suggested follow-up questions. - Use text, voice input, or Radar’s search bar. - Conversations are saved in searchable history, can be pinned, and can be shared through links that expire after 30 days. - Users can inspect how the assistant interpreted their question, which datasets it queried, and how it derived its answer. ## Explaining Existing Charts - An “Explain with AI” action lets users start a conversation from any Radar visualization. - The assistant receives: - A screenshot of the chart for visual context. - The exact underlying data from Radar’s API. - The current location, date range, and applied filters. - This combination lets the model explain the exact chart being viewed while preserving numerical accuracy and context. ## Example: Internet Quality in Portugal - A user can ask about home Internet quality in Portugal without manually selecting API parameters or searching Radar pages. - Researcher queries the Internet quality API and presents the results through familiar interactive charts. - Users can ask follow-up questions, such as comparing Portugal with Spain or another neighboring country. - The assistant can also suggest related investigations, including common Internet outages. ## Example: Investigating an Internet Shutdown - Researcher can support more open-ended investigations for engineers, researchers, journalists, and network operators. - For Iran’s 2026 government-directed shutdowns, it: - Retrieved recorded outage events. - Collected corresponding HTTP traffic data. - Explained the timeline in natural language. - The analysis described traffic falling from approximately 0.58 on January 7 to nearly zero by January 9, partially recovering around January 17, and approaching normal levels by January 27. - It displayed the findings as an annotated traffic chart and a table of outages, including another shutdown beginning February 28. - Suggested follow-ups included comparing Iran’s traffic with neighboring countries. ## Cloudflare’s Technical Implementation - The application runs entirely on Cloudflare’s developer platform. - A Cloudflare Worker uses the Cloudflare Agents SDK. - Each conversation is stored in a stateful Durable Object with its own SQLite database. - Chat history, titles, and streaming responses persist even if users leave the page during generation. - Workers AI runs open models, including Kimi K2.7. - An ordered fallback chain across three model families helps maintain availability if one model provider reaches capacity. - Requests are routed through AI Gateway. Radar Researcher makes Cloudflare’s extensive public datasets easier to use without sacrificing precision or transparency. It is particularly valuable when users need fast, context-specific analysis backed by interactive charts and verifiable source data.

Read original

Ai Gateway Large Language Models Cloudflare Workers Durable Objects+3

cloudflareAug 6, 20264 min readCurated summary

From ranking to recommended: get your site ready to thrive in the age of AI agents

AI assistants are becoming a major channel through which customers discover, compare, and purchase from businesses. As agents replace traditional search journeys, discoverability increasingly depends on whether a site is easy for machines to access, understand, trust, and recommend. Cloudflare’s Agent Readiness and Answer Engine Optimization (AEO) tools measure both technical accessibility and visibility in AI-generated recommendations. ## The Rise of the Agentic Audience - Fewer than half of HTML page requests reportedly come from humans, with the remainder including crawlers, bots, and increasingly capable agents. - Customers may ask AI assistants for: - Solutions to specific problems - Recommendations tailored to their needs - Comparisons between products or services - Actions performed on their behalf - Traditional metrics such as clicks and page views do not show whether AI systems can use a site or recommend it. - Agent-focused discoverability requires being found, read, and confidently cited by answer engines, shopping assistants, and research tools. ## Diagnostics: Measuring Agent Readiness Cloudflare’s Diagnostics feature evaluates a site from an agent’s perspective rather than simply loading its homepage. - It checks whether agents: - Are permitted to access the site - Can discover its content - Can retrieve clean, machine-readable content - Can identify callable interfaces and authentication methods - Diagnostic checks examine: - `robots.txt` - XML sitemaps - Response headers - Markdown content - Published metadata - APIs and agent-facing tools - Results are grouped into readiness levels from “Not Ready” to fully agent-native. - Each check returns: - Pass, fail, or neutral status - An explanation of its significance - Evidence showing the exact request and response ### Diagnostic Improvement Areas - **Quick wins:** Crawler-readable `robots.txt`, XML sitemaps, AI-crawler rules, and clean Markdown. - **Technical groundwork:** Content Signals, API catalogs, link headers, and agent login instructions. - **Advanced integration:** OAuth discovery, MCP, A2A agent cards, skills indexes, Web Bot Auth, and WebMCP. - **Commerce:** Emerging standards such as x402, ACP, UCP, and AP2. These are currently informational and do not affect the readiness score. - Recommended fixes either link directly to Cloudflare settings or generate a coding prompt that can be given to an agent. ## AEO: Measuring AI Recommendations Agent Readiness shows whether agents can access a site; AEO measures whether assistants actually recommend it. - Cloudflare infers a site’s industry and category, then tests likely customer prompts against assistants such as Claude and GPT. - Prompts cover recommendations, product comparisons, and general category advice without naming the customer’s brand. - AEO reports several visibility metrics: - **Citation Rate:** How often the site is cited as a source. - **Prominence:** How early and substantially the site appears in an answer. - **Mention Rate:** How often the brand is named, even without a source citation. - **Share of Voice:** The site’s share of citations compared with competitors. - Comparing mention and citation rates distinguishes brand awareness from authoritative attribution. ## Category Benchmarks and Industry Fit Cloudflare builds a benchmark for each industry and category before scoring individual sites. - AI assistants are queried with representative prompts to identify: - Which sites are cited - Where citations appear - How prominently each brand is represented - The benchmark is reused across accounts in the same category rather than regenerated for every scan. - This approach provides: - Instant result loading - Lower AI-compute costs - An **Industry Fit** score showing whether the site appears alongside its real competitors ## Multimodel Evaluation Because AI responses vary, Cloudflare queries assistants multiple times across different models using AI Gateway. - The system analyzes the actual answer text and cited sources customers would see. - Workers AI evaluates citation and mention patterns on Cloudflare’s infrastructure. - Exact text analysis is used alongside model-based judgment, rather than asking a model to grade its own response. - The process converts many variable responses into consistent, actionable metrics without requiring site owners to build their own evaluation framework. ## Operator Activity Cloudflare also reports real crawl and referral activity from AI operators. - Activity is shown by operator, including OpenAI and Google. - Site owners can see: - Which operators read their content - Which operators send visitors back - Errors encountered during crawling, such as `403` blocks and `404` links The broader goal is to connect controlled AEO benchmarks with real-world agent traffic, helping businesses understand both how AI systems perceive them and whether those systems can successfully access and use their sites. Businesses should treat agents as a core audience: first make the site technically accessible and machine-readable, then measure whether AI assistants cite and recommend it. Acting early may provide an advantage because most websites are not yet optimized for agent discovery.

Read original

Ai Gateway Model Context Protocol Cloudflare Oauth+4

cloudflareAug 5, 20264 min readCurated summary

Catching rogue AI behavior with identity-aware analytics

AI usage is difficult to govern without knowing both who made each request and what normal usage looks like for that person or agent. Cloudflare’s new Identity-aware AI Gateway and User Insights address this by attaching verified identities to requests and detecting behavior that significantly deviates from historical patterns. Together, they provide centralized visibility, per-user cost controls, and anomaly detection without requiring additional setup for traffic already routed through AI Gateway. ## AI Gateway as a Central Control Plane - AI Gateway routes requests from applications, developer tools, and agent harnesses—including Claude Code, Codex, and GitHub Copilot—through one platform. - It provides centralized observability, security, governance, and spend management across providers such as OpenAI, Anthropic, Google, and Workers AI. - This centralization makes it possible to analyze usage consistently across both human users and automated agents. ## Identity-Aware Requests with Cloudflare Access - The Cloudflare Access integration places a custom domain, such as `ai.example.com`, in front of the gateway. - Organizations can: - Authenticate users through SAML-compatible providers such as Okta or Microsoft Entra. - Apply access policies to specific users. - Avoid distributing Cloudflare API keys. - Each authenticated request includes the Access user ID as `cf.user_id`. - Administrators can filter logs, analytics, and spending by the actual requester rather than by a shared API key. - Per-user spend limits can assign each person a separate budget and either block requests or route them to cheaper models after the limit is reached. - Planned improvements will use identity-provider groups to control model access and spending—for example, granting frontier-model access to machine learning teams while limiting support teams. ## User Insights and Behavioral Baselines - User Insights is available to all AI Gateway customers at no extra cost. - It analyzes existing gateway traffic without requiring additional configuration. - The feature builds behavioral profiles for every account, including both people and agents. - It tracks cost inefficiencies such as poor cache-hit rates and oversized context windows, but focuses primarily on whether usage is normal for that particular account. - Human users and automated agents are evaluated according to their own patterns: - Agents may have regular, predictable sessions. - Humans typically have more irregular prompts, timing, and session lengths. ## Session-Based Anomaly Detection - User Insights evaluates sessions rather than individual requests, reducing noise from isolated events. - Each session is compared with the account’s rolling 95th-percentile session cost over the previous 30 days. - A session becomes a strong anomaly candidate when it exceeds twice that personal p95 baseline. - This relative comparison avoids misleading fixed thresholds: - A $500 session may be normal for a consistently heavy user. - A $50 session may be highly unusual for an agent that normally spends $5. - Baselines adjust over time as an account’s usage changes. ## Combining Personal and Organization-Wide Thresholds - User Insights also applies an organization-wide p99 cost ceiling. - In the example analysis: - Most sessions cost less than $10. - The organizational p95 is $20. - The p99 is $200, meaning only 1% of sessions reach that amount. - Alerts are triggered only when a session is both: - More than twice the account’s personal p95. - Above the organization’s p99 ceiling. - This prevents alerts for: - Small-dollar spikes that are statistically unusual but not worth investigating. - Expensive sessions that are routine for a particular user. - A dollar floor also prevents tiny accounts from triggering alerts because of insignificant percentage increases. ## Filtering for Rogue Behavior - The resulting interface presents a feed of accounts that have broken their established usage patterns. - This focuses administrators on potentially meaningful incidents instead of showing every unusual request. - The approach is designed to detect trusted users or agents that suddenly perform more of an already-authorized activity—behavior that traditional controls may not block because no new tool or forbidden action is involved. Cloudflare’s recommendation is to route AI traffic through AI Gateway, authenticate it with Cloudflare Access, and use identity-based budgets alongside behavioral baselines. This combination helps organizations connect spending and activity to specific people or agents while concentrating investigations on statistically significant, high-impact deviations.

Read original

Ai Gateway Ai Security Anomaly Detection Cloudflare Access+4

gitlabJun 30, 20262 min readCurated summary

Claude Sonnet 5 on GitLab: More reliable, more efficient

Claude Sonnet 5 is now available on GitLab Duo Agent Platform through GitLab’s AI Gateway across all tiers and deployment models. GitLab reports that it is the first model to complete every task in its evaluation suite, compared with 93.8% for Sonnet 4.6, while resolving 8.8% more issues. The result is intended to make software-engineering agents more reliable, efficient, and suitable for production-scale workflows. ## More Reliable Agent Runs - Sonnet 5 is designed for multi-step development tasks, code generation, and automated workflows. - Completing every benchmark task reduces failures that require diagnosis, reprompting, and verification. - GitLab highlights applications including: - Multi-file refactoring - Test generation - Security investigations across repository history - Pipeline-failure analysis - Higher completion rates allow developers to review agent output instead of repeatedly restarting incomplete runs. ## Lower Cost Through Efficiency - Reliability and resource efficiency reduce the effective cost of completed agent tasks. - Models on GitLab Duo consume GitLab Credits at different rates. - Teams can control spending by using a model whose cost and performance fit routine development work. - GitLab directs users to its Credits documentation for model-specific consumption rates. ## Selecting the Right Model - Sonnet 5 is positioned as a dependable default for everyday software-development tasks. - Sonnet-class models aim to balance quality, speed, and cost. - Claude Opus 4.8 remains available for complex, long-running tasks requiring greater reasoning depth. - Teams can select models per task through model selection in their GitLab instance. ## Availability - Claude Sonnet 5 is available immediately on GitLab Duo Agent Platform through the AI Gateway. - It runs on GitLab Credits and is offered across all tiers and deployment models. - New users can start through a Duo Agent Platform trial or GitLab Free, while Premium and Ultimate subscribers can use included credits. Overall, GitLab presents Claude Sonnet 5 as a more reliable and economical choice for routine agent-assisted development, with more powerful models available when tasks require deeper reasoning.

Read original

Ai Gateway AI Agents DevSecOps Gitlab Duo Agent Platform+2

gitlabJun 10, 20262 min readCurated summary

GitLab on Google Cloud: Fully managed, compliant, and AI-ready

GitLab is introducing a fully managed deployment on Google Cloud through certified managed service providers such as Beyond and Digital Future. The offering combines data residency and compliance controls with access to Google’s Gemini and Gemma models through GitLab Duo Agent Platform. Organizations can also purchase the platform through Google Cloud Marketplace, applying existing cloud commitments to GitLab, AI inference, and infrastructure costs. ## Fully Managed GitLab on Google Cloud - Certified MSPs operate GitLab on Google Cloud under service-level agreements, removing infrastructure-management responsibilities from customer teams. - Organizations retain control over where code, pipelines, and security data are stored, supporting sovereignty and data-residency requirements. - GitLab’s audit and policy controls provide visibility into agent actions, merge requests, and security findings. ## AI Model Choice for Different Workloads - Gemini models, including Gemini 3.5 Flash, are available in Duo Agent Platform through Google’s Gemini Enterprise Agent Platform. - GitLab’s participation in Google’s early-access program is intended to bring new Gemini models to Duo as they become available. - Regulated or self-hosted teams can use Gemma 4 with GitLab Duo Self-Hosted. - With self-hosted models, the AI Gateway and all requests and responses remain within an organization’s on-premises or private-cloud environment. ## Using Existing Google Cloud Commitments - GitLab and Duo Agent Platform can be purchased through Google Cloud Marketplace. - Existing Google Cloud commitments can fund GitLab subscriptions, model inference, and related infrastructure without starting a new procurement cycle. - Consolidated Google Cloud billing reduces reconciliation across vendors. - GitLab retains its own cost-management features, including usage dashboards, model policies, and GitLab Credits for more predictable AI spending. ## One Governed DevSecOps Platform - GitLab Duo Agent Platform provides software-delivery context that standalone coding assistants lack, including merge requests, pipelines, and deployment targets. - This context helps agents perform multi-step work and supports code review at monorepo scale. - Combining GitLab’s governance and lifecycle data with Google’s models keeps deployment, model selection, compliance, and spending aligned in one platform rather than fragmented across multiple tools. Organizations can start with a Duo Agent Platform trial, enable it through the free GitLab tier, or use included GitLab Credits with Premium and Ultimate subscriptions. Overall, the offering is aimed at teams that want managed GitLab operations, flexible AI model access, and strong control over data location and costs on Google Cloud.

Read original

Ai Gateway Gitlab DevSecOps Gemini+4

cloudflareApr 20, 20264 min readCurated summary

The AI engineering stack we built internally — on the platform we ship

Cloudflare built an internal AI engineering stack that now supports AI coding tools for 93% of its R&D organization. The system combines centralized authentication and model routing with internal knowledge, automated review, and sandboxed agent execution. Cloudflare argues that building these capabilities on its own platform improved security, visibility, cost control, and developer velocity, while also serving as a proving ground for products it ships publicly. ## Adoption and Impact - Over the previous 30 days: - 3,683 employees used AI coding tools, representing 60% of the company and 93% of R&D. - AI tools generated 47.95 million requests. - 295 teams used agentic AI tools or coding assistants. - AI Gateway handled 20.18 million requests and routed 241.37 billion tokens. - Workers AI processed 51.83 billion tokens. - The four-week rolling average of merge requests increased from roughly 5,600 per week to more than 8,700. - The week of March 23 reached 10,952 merge requests, nearly twice the Q4 baseline. - Cloudflare’s initial focus was MCP servers, but the effort expanded to standards, code review, onboarding, and propagating changes across thousands of repositories. ## Architecture at a Glance Cloudflare organized the stack into three layers: - **Platform layer:** Authentication, model routing, inference, MCP access, and code execution. - **Knowledge layer:** System context and repository guidance through Backstage and `AGENTS.md`. - **Enforcement layer:** Automated quality controls using AI Code Reviewer and the Engineering Codex. The stack uses Cloudflare products including: - **Cloudflare Access** for zero-trust authentication. - **AI Gateway** for centralized LLM routing, cost tracking, bring-your-own-key support, and zero-data-retention controls. - **Workers AI** for open-weight model inference. - **Workers and Access** for an MCP Server Portal with single OAuth. - **Dynamic Workers** for sandboxed agent-generated code execution. - **Agents SDK and Durable Objects** for stateful, long-running agent sessions. - **Sandbox SDK** for isolated cloning, building, and testing environments. - **Workflows** for durable, multi-step processes. - **Backstage** for a 16,000-plus-entity knowledge graph. ## Centralized Authentication and AI Routing - Cloudflare Access authenticates users and enforces zero-trust policies. - Every LLM request passes through AI Gateway, providing a single control point for: - Provider credentials - Usage and cost attribution - Model selection - Data-retention policies - Provider permissions - In the past month, frontier providers handled 91.16% of requests, while Workers AI handled 8.84%. - Cloudflare routes requests through a proxy Worker rather than connecting clients directly to AI Gateway. - The proxy enables later additions such as per-user attribution, model catalogs, permission enforcement, and support for new coding tools without changing client configurations. ## Workers AI and Open-Weight Models - Workers AI runs open-source models on GPUs distributed across Cloudflare’s global network. - Keeping inference on the same network as Workers, Durable Objects, and storage reduces latency, network failures, and cross-cloud configuration. - Kimi K2.5, with a 256,000-token context window, tool calling, and structured outputs, processes more than 7 billion tokens per day for a Cloudflare security agent. - Cloudflare estimates that running this workload on Workers AI is 77% cheaper than using a mid-tier proprietary model. - Workers AI is also used for: - Documentation review in CI - Generating `AGENTS.md` files - Lightweight inference where latency matters more than maximum model capability - Cloudflare expects open-source models to handle an increasing proportion of its internal workloads. ## One-Command Client Configuration - Engineers begin setup with: ```bash opencode auth login https://opencode.internal.domain ``` - The command uses an OpenCode discovery endpoint at: ```text https://opencode.internal.domain/.well-known/opencode ``` - The Worker-hosted endpoint provides authentication and configuration information. - This mechanism is designed to configure providers, models, MCP servers, agents, commands, and permissions without requiring engineers to edit configuration files manually. ## Overall Recommendation Cloudflare’s experience suggests that organizations adopting AI coding tools should build a centralized control plane early: authenticate users consistently, route model traffic through one managed gateway, maintain shared system knowledge, and enforce quality through automated review and isolated execution. Using the same production platform for internal tooling can also expose product gaps and accelerate improvements to the platform itself.

Read original

Ai Gateway Database Design CI/CD Open Source+4

cloudflareApr 16, 20263 min readCurated summary

Cloudflare’s AI Platform: an inference layer designed for agents

Cloudflare is positioning AI Gateway as a unified inference layer for agentic applications. It provides one API for models from multiple providers, centralized cost and reliability controls, and low-latency access through Cloudflare’s global network. The platform is also expanding toward customer-hosted models and multimodal inference, reducing dependence on any single AI vendor. ### A Unified Model Catalog and Endpoint - Developers can call Cloudflare-hosted and third-party models through the same `env.AI.run()` interface. - Switching models, such as from Workers AI to Anthropic or OpenAI, requires only a one-line change. - Cloudflare plans to add REST API support for applications that do not run on Workers. - The catalog includes more than 70 models from over 12 providers, spanning: - Text and reasoning models - Image, video, and speech models - Open-source models hosted on Workers AI - Proprietary models from major providers - Users pay through one credit system rather than managing separate provider accounts. ### Centralized Cost and Usage Management - AI Gateway gives companies a consolidated view of spending across providers. - Request metadata can categorize costs by: - Team - User - Customer - Workflow - Free versus paid usage - This is particularly useful for agents, which may make many model calls for a single user request. ### Bringing Custom Models to Workers AI - Cloudflare is working on allowing customers to deploy fine-tuned or specialized models through Workers AI. - The deployment workflow uses Replicate’s Cog technology to package models and manage: - Python dependencies - CUDA requirements - Model weights - Runtime configuration - Developers define the environment in `cog.yaml` and implement model setup and inference in `predict.py`. - After building the Cog container, Cloudflare can deploy and serve it through existing Workers AI APIs. - Planned improvements include customer-facing APIs, Wrangler commands, and GPU snapshotting to reduce cold-start times. ### Optimizing Agent Responsiveness - For live agents, time to first token can matter more than total inference time. - A small reduction in startup latency can make an agent feel significantly more responsive, even if the full answer still takes several seconds. - Cloudflare’s presence in 330 cities places gateways close to users and inference endpoints, reducing network delay before streaming begins. - Workers AI is adding agent-focused open-source models, including Kimi K2.5 and real-time voice models. Cloudflare’s direction is to make model selection, deployment, monitoring, and failover easier through a single infrastructure layer. For teams building agents, the platform offers a way to mix providers and custom models while controlling cost, latency, and operational complexity.

Read original

Ai Gateway Cloudflare Python Api Integration+4

daangnDec 22, 2025Original article

Daangn's GenAI Platform (opens in new tab)

Daangn has scaled its Generative AI capabilities from a few initial experiments to hundreds of diverse use cases by building a robust, centralized internal infrastructure. By abstracting model complexity and empowering non-technical stakeholders, the company has optimized API management, cost tracking, and rapid product iteration. The resulting platform ecosystem allows the organization to focus on delivering product value while minimizing the operational overhead of managing fragmented AI services. ### Centralized API Management via LLM Router Initially, Daangn faced challenges with fragmented API keys, inconsistent rate limits across teams, and the inability to track total costs across multiple providers like OpenAI, Anthropic, and Google. The LLM Router was developed as an "AI Gateway" to consolidate these resources into a single point of access. * **Unified Authentication:** Service teams no longer manage individual API keys; they use a unique Service ID to access models through the router. * **Standardized Interface:** The router uses the OpenAI SDK as a standard interface, allowing developers to switch between models (e.g., from Claude to GPT) by simply changing the model name in the code without rewriting implementation logic. * **Observability and Cost Control:** Every request is tracked by service ID, enabling the infrastructure team to monitor usage limits and integrate costs directly into the company’s internal billing platform. ### Empowering Non-Engineers with Prompt Studio To remove the bottleneck of needing an engineer for every prompt adjustment, Daangn built Prompt Studio, a web-based platform for prompt engineering and testing. This tool enables PMs and other non-developers to iterate on AI features independently. * **No-Code Experimentation:** Users can write prompts, select models (including internally served vLLM models), and compare outputs side-by-side in a browser-based UI. * **Batch Evaluation:** The platform includes an Evaluation feature that allows users to upload thousands of test cases to quantitatively measure how prompt changes impact output quality across different scenarios. * **Direct Deployment:** Once a prompt is finalized, it can be deployed via API with a single click. Engineers only need to integrate the Prompt Studio API once, after which non-engineers can update the prompt or model version without further code changes. ### Ensuring Service Reliability and Stability Because third-party AI APIs can be unstable or subject to regional outages, the platform incorporates several safety mechanisms to ensure that user-facing features remain functional even during provider downtime. * **Automated Retries:** The system automatically identifies retry-able errors and re-executes requests to mitigate temporary API failures. * **Region Fallback:** To bypass localized outages or rate limits, the platform can automatically route requests to different geographic regions or alternative providers to maintain service continuity. ### Recommendation For organizations scaling AI adoption, the Daangn model suggests that investing early in a centralized gateway and a no-code prompt management environment is essential. This approach not only secures API management and controls costs but also democratizes AI development, allowing product teams to experiment at a pace that is impossible when tied to traditional software release cycles.

Ai Gateway AI Large Language Models Generative AI+3