Orchestrating AI Code Review at scale (opens in new tab)
Cloudflare built a CI-native AI code review system to reduce review bottlenecks without overwhelming engineers with noisy or generic model feedback. Instead of using one large prompt, it orchestrates up to seven specialized agents for areas such as security, performance, compliance, and documentation, then uses a coordinator to deduplicate and assess findings. The system now reviews tens of thousands of merge requests, approving clean changes and blocking serious bugs or vulnerabilities. ## Why Naive AI Review Wasn’t Enough - Traditional code review can leave merge requests waiting for hours and creates repeated context switching. - Commercial AI review tools provided useful functionality but lacked the flexibility and customization required across Cloudflare’s organization. - A basic “send the Git diff to an LLM” approach produced: - Vague recommendations - Hallucinated syntax errors - Repetitive advice such as adding error handling where it already existed - Complex codebases required specialized analysis rather than generic summarization. ## Specialized Agents and Coordination - The system launches up to seven focused reviewers covering: - Security - Performance - Code quality - Documentation - Release management - Internal Engineering Codex compliance - A coordinator agent: - Deduplicates overlapping findings - Evaluates the actual severity of issues - Produces one structured review comment - The system can actively block merges when it detects serious defects or security vulnerabilities. ## Plugin-Based Architecture - The platform uses composable plugins so it can support different: - Version-control systems - AI providers - Internal standards - Repository-specific requirements - Each plugin implements a `ReviewPlugin` interface with three lifecycle phases: - `bootstrap`: Runs concurrently and is non-fatal. - `configure`: Runs sequentially and is fatal if essential configuration fails. - `postConfigure`: Handles asynchronous work after configuration assembly. - Through `ConfigureContext`, plugins can: - Register agents and AI providers - Set environment variables - Inject prompt sections - Configure agent permissions - Plugins contribute through the context API rather than accessing the final configuration directly. - The core assembler combines these contributions into `opencode.json`. - This separation prevents unrelated components from becoming tightly coupled; for example, GitLab logic does not need to understand Cloudflare AI Gateway settings. ## Plugin Responsibilities - `@opencode-reviewer/gitlab` - Provides GitLab merge request data and a comment server. - `@opencode-reviewer/cloudflare` - Configures AI Gateway model tiers and fallback chains. - `@opencode-reviewer/codex` - Checks compliance with internal engineering RFCs. - `@opencode-reviewer/braintrust` - Adds distributed tracing and observability. - `@opencode-reviewer/agents-md` - Verifies that repository `AGENTS.md` instructions are current. - `@opencode-reviewer/reviewer-config` - Retrieves remote model overrides for individual reviewers. - `@opencode-reviewer/telemetry` - Tracks reviews asynchronously. ## Why OpenCode - Cloudflare already used OpenCode extensively and understood its behavior. - Its open-source implementation allows engineers to: - Investigate problems directly - Contribute fixes upstream - Extend the system through its SDK - Cloudflare engineers had contributed more than 45 upstream pull requests at the time of writing. - Its server-first design was especially important: - Review sessions can be created programmatically. - Prompts can be sent through an SDK. - Multiple concurrent sessions can be managed without scraping or wrapping a CLI interface. ## Coordinator Process - The coordinator runs OpenCode as a child process using `Bun.spawn`. - Its prompt is passed through `stdin` rather than a command-line argument. - This avoids Linux’s `ARG_MAX` limit, which previously caused `E2BIG` failures for unusually large merge requests containing extensive descriptions or logs. - OpenCode runs with `--format json`, emitting JSONL events through standard output. - This event-based interface allows the orchestration layer to collect and process results from concurrent reviewer sessions. A practical takeaway is to treat AI review as an orchestrated CI system rather than a single LLM prompt. Specialized agents, strict plugin boundaries, structured outputs, and observability are essential for making automated review reliable enough to influence merge decisions at organizational scale.