gitlab·May 18, 2026·3 min readCurated summary

Fix bugs with Codex and GitLab

Codex accelerates coding in the terminal, but producing a fix is only one part of shipping software. GitLab supplies the surrounding lifecycle: issues, merge requests, CI/CD, security scanning, code review, and human approval. The tutorial demonstrates this progression through a Rust WebSocket bug, first with local Codex, then with GitLab MCP for issue context, and finally with Codex as an external agent in GitLab Duo Agent Platform. ## Prerequisites and Project Setup - Configure Codex in the terminal, Rust/Cargo, and access to a GitLab project. - Import and clone the Tanuki IoT Platform project, then launch Codex from its repository root. - The tutorial focuses on `backend/`, where: - Sensors submit readings through a REST API. - Dashboards receive live readings through WebSocket streams. - `AGENTS.md` provides Codex with repository structure, toolchain instructions, build commands, and quality expectations. ## Reproducing the WebSocket Filtering Bug - Start the Rust metrics backend on port `9090`: ```bash PORT=9090 cargo run --manifest-path backend/rust-metrics-store/Cargo.toml ``` - Connect to a filtered WebSocket stream: ```bash websocat 'ws://localhost:9090/ws?sensor=arduino-iot-collector&metric=temperature_celsius' ``` - Submit both temperature and humidity readings for the same sensor through the REST API. - The stream incorrectly returns both metrics instead of only `temperature_celsius`, proving that the WebSocket handler does not apply the metric filter. ## Fixing the Bug with Codex - Give Codex a focused request to add metric filtering to `/ws`. - Codex examines the Rust source and identifies that the endpoint already supports `sensor` filtering but lacks an optional `metric` condition. - It updates the handler, adds tests, and keeps documentation aligned with the implementation. - Codex runs formatting, tests, and builds before creating a branch, committing, and pushing the change. - Once the merge request is created, GitLab handles: - CI/CD pipelines - Security scanning - GitLab Duo Code Review - A follow-up WebSocket test confirms that supplying both sensor and metric now returns only the requested metric. ## Adding GitLab Context with MCP - Local Codex can inspect repository files, but it cannot automatically see GitLab issues, requirements, implementation notes, merge-request discussions, or pipeline status. - The GitLab MCP server connects Codex to that development lifecycle context. - Codex can retrieve the existing issue directly instead of requiring the developer to copy its contents into the prompt. - The issue acts as the shared source of truth and includes: - The bug description - Functional behavior requirements - Non-functional requirements - Required tests - Updates to `README.md` and `AGENTS.md` - Implementation notes - This helps Codex produce a fix that satisfies the agreed requirements rather than merely addressing the symptom visible in the local code. ## Using Codex as an External GitLab Agent - The tutorial’s third workflow uses Codex inside GitLab Duo Agent Platform as an external agent. - This allows the agent to participate after the merge request is open, particularly when addressing review feedback. - GitLab remains the system coordinating issues, merge requests, pipelines, reviews, and deployment, while Codex contributes its terminal-oriented coding capabilities. - The overall workflow moves from bug report to implementation, automated validation, review feedback, revisions, and an informed human decision to ship. ## Practical Conclusion Use Codex for fast, repository-local implementation, but connect it to GitLab through MCP or Duo Agent Platform when requirements and review context matter. The strongest workflow combines Codex’s coding speed with GitLab’s issue-aware, automated, and human-governed delivery lifecycle.

Read original(opens in new tab)

CI/CDAI AgentsGitlabRust+3

gitlab·May 13, 2026·2 min readCurated summary

Harden your pipeline perimeter for the era of AI-assisted coding

AI-assisted coding brings humans, agents, and third-party code together in the software pipeline, increasing the speed at which vulnerabilities can be introduced. The post argues that security tools must become part of the development workflow rather than separate portals. GitLab Ultimate presents this as a three-part control plane: see every risk, automatically enforce policy, and help developers fix vulnerabilities in context. ## See Every Project, Risk, and Action - The Group Security Dashboard consolidates SAST, SCA, secret detection, container, IaC, DAST, and fuzz-testing results across projects. - Security Inventory identifies projects that have never been scanned, exposing governance gaps hidden by project-level dashboards. - Credentials Inventory tracks tokens, owners, scopes, expiration, and revocation status, enabling rapid response to compromised credentials. - Token Lifetime Enforcement applies mandatory maximum lifetimes to tokens. - Audit Event Streaming sends events such as token creation, permission changes, MR approvals, and role updates to SIEM systems in real time. - Group-wide SBOM search helps identify open-source dependency exposure across the project portfolio. ## Enforce Policies Automatically - Scan Execution Policies inject mandatory SAST, SCA, and secret-detection jobs into production pipelines and prevent removal or bypass through configuration such as `[skip ci]`. - Pipeline Execution Policies enforce platform-owned CI templates, closing gaps created by ungoverned or “shadow” pipelines. - MR Approval Policies automate requirements for protected branches, approvers, and code owners. - The Compliance Center maps controls to SOC 2, ISO 27001, NIST, and PCI DSS while providing live dashboards and audit trails. - Secret Push Protection rejects credentials before they enter Git history and records bypass attempts. ## Fix Vulnerabilities in the Development Workflow - The MR security widget displays SAST, SCA, container, IaC, and secret findings directly alongside code changes before they reach the default branch. - Advanced SAST uses cross-file taint analysis to trace untrusted input from source to sink. - GitLab Duo helps identify likely false positives and explain the reasoning behind its assessment. - The Duo Security Analyst Agent prioritizes issues using exploitability, exposure, and business context in addition to CVSS. - Agentic Vulnerability Resolution can create fix merge requests for high-impact SAST findings, allowing developers to review and merge changes through the normal approval process. The practical recommendation is to make security controls native to the pipeline. For organizations adopting AI-assisted development, GitLab Ultimate’s integrated visibility, automated enforcement, and AI-supported remediation are presented as a way to maintain security without sacrificing delivery speed.

Read original(opens in new tab)

CI/CDGitlabDevSecOpsSast+4

gitlab·May 11, 2026·4 min readCurated summary

GitLab Act 2

GitLab is restructuring its organization and strategy to prepare for an agent-driven software industry. It expects AI agents to dramatically increase software production, making scalable infrastructure, orchestration, context, and governance more important than traditional developer tooling. The company is reducing geographic footprint and management layers while reorganizing R&D around smaller, autonomous teams, reaffirming its FY27 guidance pending final restructuring costs. ## Organizational Restructuring - GitLab is conducting the process openly, including a voluntary separation window. - The new organizational shape is expected to be finalized by June 1 where possible; local legal processes may extend timelines. - Planned operational changes include: - Reducing the number of countries with small GitLab teams by up to 30%, while relying on partners in affected markets. - Removing up to three management layers in some functions. - Reorganizing R&D into approximately 60 smaller teams with end-to-end ownership. - Automating internal reviews, approvals, and handoffs with AI agents, then adjusting roles accordingly. - The restructuring and strategic shift are related but independently justified. - GitLab will disclose the restructuring’s final scope and financial impact during its June 2 earnings call. ## Software Development in the Agentic Era - Software will increasingly be produced by machines under human direction. - Agents will plan, code, review, deploy, and repair software. - Engineers will remain responsible for architecture, customer understanding, judgment, and difficult tradeoffs. - Lower software-production costs are expected to expand demand for software and increase the value of developer platforms. - Deep engineering skills—such as system design, distributed systems, failure analysis, and integrating new capabilities safely—will become more important and scarce. - GitLab points to its Duo Agent Platform, released in January, as an initial investment in this future. ## Infrastructure for Machine-Scale Development - Agents can create merge requests, trigger pipelines, and push commits at volumes far beyond human teams. - Git and existing development platforms were not designed for this level of activity. - GitLab plans to: - Reengineer Git for machine-scale workloads. - Replace parts of its monolithic architecture with API-first, composable services. - Provide agent-specific APIs so agents can interact as first-class platform users. - The company argues that reliability, performance, and scalability at this level will become a major source of platform value. ## Orchestration Across the Software Lifecycle - Enterprises need more than individual agents that generate code or open merge requests; they need software that reaches production and delivers business value. - GitLab’s orchestration layer is intended to coordinate agents across the lifecycle by: - Assigning work and managing state. - Passing context between tasks. - Resolving conflicts. - Enforcing policies and guardrails. - Keeping humans involved where judgment is required. - CI/CD is being reconsidered as part of this shift, with orchestration serving as the runtime for validating and safely deploying machine-rate changes. ## Context as a Competitive Advantage - Code generation capabilities are increasingly similar across developer-tool vendors. - GitLab believes its advantage lies in the connected context accumulated across planning, code, review, security, deployment, and operations. - It plans to make this data model a first-class, API-accessible service. - More contextual information should allow agents to use fewer tokens and produce better results. ## Governance Built Into the Platform - As agents perform more work, enterprises need strong control over identity, permissions, policies, auditing, and data location. - GitLab intends to make governance core infrastructure rather than an add-on product. - Every agent, pipeline, and merge request should operate through platform services that can: - Control who or what may perform an action. - Record what happened and why. - Protect sensitive code and data. - Support flexible deployment models. ## One Platform, Three Modes - GitLab notes that most business software cannot realistically be rewritten for the agentic era. - Its platform strategy is therefore intended to support existing codebases alongside newer development models. - The provided text ends before explaining the three modes in detail. GitLab’s overall recommendation to itself is to reshape both its organization and platform around machine-scale software development, while preserving human control over architecture, judgment, and governance.

Read original(opens in new tab)

CI/CDAI AgentsGitlabDistributed Systems+4

github·May 7, 2026·4 min readCurated summary

Agent pull requests are everywhere. Here’s how to review them.

Agent-generated pull requests are increasing rapidly, while human review capacity remains limited. Although these changes often look clean and pass CI, research suggests they can introduce more redundancy and technical debt—and reviewers may be more likely to approve them. The solution is not to review more slowly, but to focus human judgment on risks agents are least equipped to recognize. ## The Scale of Agent-Generated Pull Requests - GitHub Copilot code review has processed more than 60 million reviews and grown tenfold in under a year. - More than 20% of GitHub code reviews now involve an agent. - Developers can launch many agent sessions simultaneously, causing pull-request volume to grow faster than human review capacity. - Reviewers therefore need a deliberate method for identifying high-impact issues. ## Understanding the Agent’s Limitations - Coding agents are productive and literal, but lack: - Incident history - Team-specific edge-case knowledge - Operational constraints not documented in the repository - Agents can produce code that appears complete while quietly embedding incorrect assumptions. - Human reviewers provide the context and judgment that automated tools cannot fully replicate. ## CI Gaming Agents may weaken CI when their changes fail, for example by removing tests, skipping linting, or adding commands such as `|| true`. Reviewers should verify: - Coverage thresholds were not reduced. - Tests were not removed, renamed, or skipped. - Workflows still run for forks and pull requests. - CI steps were not placed behind new restrictive conditions. Any such change requires explicit justification before approval. ## Blindness to Existing Code Reuse Agents may copy patterns from nearby code without discovering equivalent utilities elsewhere in the repository. Warning signs include: - Duplicate helper or utility functions - Reimplemented validation logic - New middleware duplicating shared modules - “Almost identical” helpers with different names Reviewers should search for existing implementations and require consolidation rather than merely commenting on duplication. For larger agent pull requests, requiring justification for new utilities can prevent redundant code from becoming future “prior art.” ## Hallucinated Correctness The most dangerous agent errors are not obvious API or syntax failures. They are changes that compile, pass tests, and still behave incorrectly under conditions such as: - Pagination boundaries - Missing permission checks - Validation edge cases - Race conditions at scale Reviewers should trace a critical path from input to output, checking empty, zero, and maximum values, external input validation, permissions on every branch, and unusual conditionals. A claimed bug fix should include a test that fails before the change; otherwise, the fix or the agent’s understanding may be incomplete. ## Agentic Ghosting and Oversized Pull Requests Large, poorly structured agent pull requests are more likely to become abandoned or misaligned. Before conducting an in-depth review, check: - Whether the agent has responded usefully in earlier review rounds - Whether the pull request includes a clear implementation plan - Whether the changes can be divided into smaller, scoped units If no plan exists, request a breakdown or a clear explanation of each component before spending time on detailed comments. ## Untrusted Input in Agent Workflows Workflows that send pull-request bodies, issue content, or commit messages to an LLM can create prompt-injection risks—especially when model output is later executed with `GITHUB_TOKEN` permissions. Reviewers should block workflows that: - Interpolate untrusted content into prompts without sanitization - Grant write access when read-only permissions are sufficient - Execute model output as shell commands without validation - Expose secrets to agent steps or logs Safer designs should use least-privilege permissions such as `permissions: read-all`, sanitize and quote untrusted content, separate analysis from execution, and require human approval before actions affecting production. Agent pull requests should not automatically receive either extra trust or blanket suspicion. Reviewers should focus on CI integrity, reuse, behavior under edge cases, reviewability, and workflow security—the areas where contextual human judgment adds the most value.

Read original(opens in new tab)

CI/CDLarge Language ModelsGithub CopilotCode Review+3

gitlab·May 7, 2026·3 min readCurated summary

Automate deployment processes with GitLab Duo Agent Platform

GitLab Duo Agent Platform can automate the complex, repetitive work of onboarding a microservice into an established GitOps workflow. By analyzing an application’s repositories and configuration, a custom agent can generate manifests, update pipelines, configure image automation, and follow organization-specific conventions. The approach combines AI-driven speed with GitLab-managed versioning, governance, and enterprise security. ## TanukiBank’s GitOps Use Case - The fictional TanukiBank application needs a new `intra-account-transfers` microservice for its Quick Transfer feature. - Its deployment architecture includes: - Individual service projects with container registries and build pipelines. - **Tanuki Bank - Delivery**, which stores deployment manifests and delivery pipelines. - **Flux Config**, which contains Flux manifests for Kubernetes. - Flux Image Automation watches service registries and updates corresponding delivery manifests. - A delivery pipeline then builds and signs the image, while Flux CD synchronizes it to the Kubernetes cluster. - Adding a service manually requires coordinated changes across all these components. ## Generating the Custom Agent’s System Prompt - GitLab Duo Agentic Chat examines the TanukiBank group, subgroups, source files, Dockerfiles, manifests, configuration, and dependencies. - It generates a detailed system prompt describing: - The existing GitOps workflow. - Required operating rules. - Reporting instructions. - Recommended tools. - The prompt is specific to the workflow at the time it is generated. - If the application’s GitOps process changes, the prompt should be regenerated. ## Creating and Configuring the Agent - A new `application-agents` project manages custom agents, their administrators, and where they can run. - A managed agent named **TanukiBank Microservice Onboarder** is created with: - A description. - The generated system prompt. - Tools recommended by GitLab Duo. - The agent is enabled in both **Tanuki Bank - Delivery** and **Flux Config**. - Its presence in each project’s Agentic Chat agent selector confirms that it is available. ## Creating the Microservice - A new `services/intra-account-transfers` project is created. - GitLab Duo’s **Developer** foundational flow implements the service from an issue specification. - The flow: - Reads the requirements. - Writes the implementation. - Creates a branch and merge request. - Links the merge request to the issue. - After local verification with `curl`, the merge request is merged and the project pipeline publishes container images. - At this stage, the service exists, but the GitOps system has not been updated: - `manifests/dev` has no service manifests. - The delivery pipeline does not reference the service. - `Flux Config` lacks an `image-update-automation.yaml` entry. ## Using the Custom Onboarding Agent - The custom agent is enabled in the new service project. - From **Tanuki Bank - Delivery**, the user selects **TanukiBank Microservice Onboarder** in Agentic Chat and provides the service name and hostname. - The agent begins onboarding by: - Finding and reading the service’s Dockerfile. - Determining the application port. - Generating the required Kubernetes manifests. - Updating the relevant delivery pipelines. - This automates the coordinated repository changes normally required for a new microservice. ## Practical Takeaway A custom GitLab Duo agent is most valuable when it is grounded in an organization’s real repositories and deployment conventions. Generate its prompt from the current system, keep the agent centrally governed, and regenerate the prompt whenever the GitOps workflow changes.

Read original(opens in new tab)

CI/CDKubernetesDocker & ContainersMicroservices+3

github·May 6, 2026·3 min readCurated summary

Validating agentic behavior when “correct” isn’t deterministic

Agentic systems such as GitHub Copilot cloud agent can complete tasks through multiple valid action sequences, making traditional deterministic tests unreliable. Timing changes, loading screens, and UI differences often produce false negatives even when the agent achieves the correct result. The post proposes an independent “Trust Layer” that validates essential outcomes and convergent behavior rather than rigid step-by-step execution paths. ## Challenges of Agent-Driven Validation - An agent may adapt to network delays or changing UI conditions and still complete its task successfully. - Conventional CI tests can fail when execution no longer matches a recorded script or expected assertion timing. - This creates a trust gap: - **False negatives:** successful tasks are reported as failures. - **Fragile infrastructure:** rendering, timing, and environment noise affect test results. - **Compliance trap:** valid behavioral variation is mistaken for regression. - Agent correctness should focus on reliably achieving essential outcomes, not reproducing an identical sequence of actions. ## Why Traditional Testing Breaks Down - **Assertion-based tests** require manually specifying every expected check and often omit valid alternative paths. - **Record-and-replay tools** are highly sensitive to timing and rendering differences. - **Visual regression tests** compare screenshots without understanding semantic meaning or the broader workflow. - **ML-based oracles** need large training datasets and generally provide little explanation for their decisions. - All four approaches assume correctness means following a stable sequence of observable states, which does not fit autonomous agents. ## Essential, Optional, and Convergent Behavior The proposed approach distinguishes between behavior that determines success and behavior that merely reflects environmental variation: - **Essential states:** Required milestones, such as reaching a VS Code “Search Results” screen. - **Optional variations:** Incidental states, including loading spinners or decorative UI changes. - **Convergent paths:** Different action sequences—such as using a keyboard shortcut or a menu—that eventually reach the same result. - A loading screen may appear in one run and not another, but the appearance of search results is what establishes success. ## Dominator Analysis The post connects this model to **dominator relationships** from compiler theory: - In a control-flow graph, node A dominates node B when every path to B must pass through A. - Applying dominator analysis to agent execution traces can identify: - Mandatory states - Optional states - Points where different execution paths converge - This produces a minimal and explainable definition of correctness instead of relying on every recorded step. ## Graph-Based Execution Modeling - Agent behavior should be represented as a graph rather than a linear script. - Graphs capture branching paths, optional states, and convergence points. - This structure provides a foundation for lightweight, explainable validation in GitHub Actions and other CI environments. A reliable validation system for agents should test whether essential outcomes occurred and whether critical invariants held, while ignoring harmless differences in timing, rendering, and execution order. This outcome-oriented Trust Layer can reduce false failures and make agentic workflows more dependable in production CI pipelines.

Read original(opens in new tab)

CI/CDAI AgentsGithub CopilotGitHub Actions+3

cloudflare·May 1, 2026·3 min readCurated summary

Introducing Dynamic Workflows: durable execution that follows the tenant

Dynamic Workflows extends Cloudflare’s durable execution system to multi-tenant and dynamically generated applications. While Dynamic Workers provide isolated runtime compute, Durable Object Facets provide tenant-specific storage, and Artifacts provide versioned source control, Dynamic Workflows lets each tenant supply its own long-running workflow code. The result is durable execution that can resume the correct tenant’s workflow after failures, hibernation, or delays of days. ## The Gap Between Durable and Dynamic Execution - Cloudflare Workflows turns a `run(event, step)` function into a durable program. - Workflow steps can: - Survive isolate recycling and failures - Sleep for hours or days - Wait for external events - Resume from the exact point where execution stopped - Workflows V2 supports up to 50,000 concurrent instances and 300 new instances per second per account. - Traditional Workflows assume the workflow class is included in the deployment and statically configured in `wrangler.jsonc`. - That model breaks for: - Multi-tenant SaaS platforms - AI-generated tenant applications - Repository-specific CI/CD pipelines - Agents that create their own durable plans - In these systems, workflow code varies by tenant, agent, repository, or request, so a single statically bound class is insufficient. ## Dynamic Workflows - `@cloudflare/dynamic-workflows` is a roughly 300-line TypeScript library. - It introduces a Worker Loader that: - Loads each tenant’s code dynamically - Routes workflow creation to the appropriate tenant - Ensures later workflow execution returns to that tenant’s code - The Loader creates a dynamic Worker with: - A tenant-specific module - A `TenantWorkflow` entrypoint - A wrapped `WORKFLOWS` binding - The dynamic entrypoint is registered as the workflow class in `wrangler.jsonc`. - Tenant code remains ordinary Cloudflare Workflows code and does not need to know it is being dynamically dispatched. ## Tenant Workflow Behavior - Tenants can use the normal Workflow APIs, including: - `env.WORKFLOWS.create(...)` - Workflow IDs and `.status()` - `.pause()` - Retries and durable steps - `step.sleep('24 hours')` - `step.waitForEvent()` - A tenant can define a standard `WorkflowEntrypoint` with a `run(event, step)` method. - The library’s primary responsibility is preserving the association between a workflow instance and the tenant implementation when the workflow resumes later. ## Three-Layer Execution Model - Dynamic Workflows consists of three layers: - The Cloudflare Workflows engine - The platform’s Worker Loader - The tenant’s dynamically loaded Worker code - A request first enters the Loader, which identifies the tenant and routes execution to its dynamic code. - The workflow engine then persists the workflow state and later invokes `run(event, step)`. - The Loader resolves the correct tenant implementation when execution resumes, even after delays or failures. Dynamic Workflows provides the missing durable-execution counterpart to Cloudflare’s dynamic compute, storage, and source-control primitives. It is particularly suited to platforms where customers or agents generate workflow code at runtime while still requiring standard durable guarantees.

Read original(opens in new tab)

CI/CDDatabase DesignTypeScriptCloudflare Workers+4

gitlab·Apr 28, 2026·3 min readCurated summary

How to build CI/CD observability at scale

CI/CD observability is essential for improving pipeline performance at enterprise scale, particularly in self-managed GitLab environments. The post presents a containerized solution built with `gitlab-ci-pipelines-exporter`, Prometheus, Grafana, and Node Exporter to turn pipeline and infrastructure data into actionable insights. Its conclusion is that centralized dashboards help teams identify bottlenecks, plan runner capacity, and measure delivery performance. ## Defining CI/CD Performance - Teams should first determine: - Which metrics matter, such as pipeline duration, job success rates, queue times, and runner utilization. - Who needs access, including developers, DevOps engineers, platform teams, and leadership. - Which decisions the data will support, such as infrastructure investment, bottleneck remediation, and capacity planning. ## Observability Architecture - The solution uses two exporters: - **Pipeline Exporter:** Collects pipeline duration, job status, and deployment metrics through the GitLab API. - **Node Exporter:** Collects host CPU, memory, and disk metrics for infrastructure correlation. - Prometheus gathers and stores the metrics. - Grafana provides real-time and historical dashboards. - Dashboards are provisioned automatically through Grafana’s file-based provisioning and can be filtered by project, branch, or time range. ## Grafana Dashboards - **Pipeline Overview:** Displays pipeline volume, success and failure rates, cancelled runs, and average duration trends. - **Job Performance:** Shows job-duration histograms, the ten slowest jobs, and failure heatmaps by project and stage. - **Runner & Infrastructure:** Correlates runner queue times with CPU, memory, and disk usage to support capacity planning. - **Deployment Frequency:** Tracks deployment counts and durations by environment, supporting DORA-style delivery analysis and detection of environment drift. ## Kubernetes Deployment - The recommended enterprise deployment runs each component as a separate workload in a dedicated `gitlab-observability` namespace. - A Kubernetes secret stores the GitLab personal access token, which requires the `read_api` scope. - The Pipeline Exporter runs as a Deployment with a service on port `8080`. - Node Exporter runs as a DaemonSet so each node can expose host metrics on port `9100`. - Prometheus and Grafana are deployed alongside the exporters and configured to scrape and visualize their metrics. - Kubernetes deployment supports existing cluster infrastructure, secrets managers, network policies, and scalable operations. ## Prerequisites - GitLab Self-Managed 18.1 or later. - Kubernetes for enterprise deployments, or Docker/Podman for smaller environments and proof-of-concept testing. - A GitLab personal access token with `read_api` permissions. - Secure secret-management practices, preferably using external secret operators in production. The practical recommendation is to begin with clearly defined performance questions, then deploy the exporter–Prometheus–Grafana stack in a controlled namespace. Combining pipeline data with host metrics provides the context needed to distinguish inefficient jobs from infrastructure capacity problems.

Read original(opens in new tab)

CI/CDKubernetesObservabilityGitlab+4

gitlab·Apr 22, 2026·3 min readCurated summary

GitLab AI Hackathon 2026: Meet the winners

Nearly 7,000 developers participated in GitLab’s 2026 AI Hackathon, creating more than 600 agents and workflows for the GitLab Duo Agent Platform. The winning projects focused on practical software delivery challenges—including organizational knowledge loss, security, compliance, migrations, and sustainability—rather than simple chatbot interactions. The results suggest that agentic AI is becoming most valuable when integrated directly into development workflows and given richer project context. ## Hackathon Scope and Evaluation - The hackathon ran from February 9 to March 25, 2026, on Devpost. - Google Cloud and Anthropic co-sponsored the event, contributing judges, prizes, and cloud resources. - Nineteen judges evaluated projects on: - Technical execution - Design - Potential impact - Quality of the idea - Total prizes reached $65,000. ## Grand Prize: LORE - LORE, or Living Organizational Record Engine, addresses the loss of institutional knowledge when engineers leave. - It combines: - Eight specialized agents - A router that directs questions to the appropriate agent - Protections against circular loops in its knowledge graph - A visual dashboard - Carbon tracking - Its command-line tool includes 43 tests, leading judges to describe it as a polished product rather than a typical hackathon prototype. ## Google Cloud and Anthropic Winners - **Gitdefender**, the Google Cloud Grand Prize winner, detects security issues during code review, writes fixes, and opens the review automatically. - **Aegis**, the Google Cloud Runner Up, explains the reasoning behind its AI decisions and is deployed on Google Cloud. - **GraphDev**, the Anthropic Grand Prize winner, maps code relationships and shows how systems evolve, helping developers understand the impact of changes. - **DocSync**, the Anthropic Runner Up, uses Detector, Writer, and Reviewer agents to update documentation. It opens a review when confident and creates an issue for human review when uncertain. ## Category Winners - **Time-Traveler**, winner for technical achievement, creates a safe copy of a production environment and runs database migrations against it using five connected agents, PostgreSQL, real data, and Google Cloud deployment. - **RedAgent**, the most impactful project, verifies AI-generated security findings before developers act on them, addressing distrust in automated reports. - **Launch Control**, recognized for ease of use, combined polished user experience, strong infrastructure, and sustainability considerations. ## Sustainability-Focused Projects - Five projects received sustainability prizes or bonuses as the organizers highlighted the growing energy cost of CI/CD systems and large language models. - **GreenPipe** analyzes CI/CD pipelines and generates carbon-footprint reports. - Sustainable Design bonuses recognized projects including: - **BugFlow**, which generated 10 fixes from one bug report in 20 minutes - **DELTA Cyber Reasoning**, an automated fuzz-testing tool - **CarbonLint**, which applies code analysis to energy consumption - **TFGuardian**, which includes carbon-footprint analysis - One project reduced monthly costs from $556 to $18, representing a reported 96% carbon reduction. ## Honorable Mentions - **SecurityMonkey** tests security scanners by injecting known vulnerabilities. - **stregent** enables CI/CD investigation and fixes through WhatsApp. - **Compliance Sentinel** evaluates merge requests for compliance risk and blocks critical violations. - **Carbon Tracker** measures the carbon footprint of individual pipeline jobs and suggests improvements. - **RepoWarden** captures the rationale behind code, not only its behavior. - **MR Compliance Auditor** maps merge-request evidence to SOC 2 controls and displays compliance scores in real time. ## What Comes Next The projects operated within a single GitLab project, but many teams supplemented their agents with local knowledge graphs to understand code relationships and dependencies. GitLab plans to build on this approach in future hackathons by providing agents with richer context. GitLab’s hackathon demonstrates that the strongest AI agents are workflow-integrated tools that can investigate, make decisions, execute changes, and involve humans when needed. Developers can explore the 600-plus projects in the gallery or build their own agents on the GitLab Duo Agent Platform.

Read original(opens in new tab)

CI/CDDatabase DesignMachine LearningGitlab+4

cloudflare·Apr 20, 2026·4 min readCurated summary

The AI engineering stack we built internally — on the platform we ship

Cloudflare built an internal AI engineering stack that now supports AI coding tools for 93% of its R&D organization. The system combines centralized authentication and model routing with internal knowledge, automated review, and sandboxed agent execution. Cloudflare argues that building these capabilities on its own platform improved security, visibility, cost control, and developer velocity, while also serving as a proving ground for products it ships publicly. ## Adoption and Impact - Over the previous 30 days: - 3,683 employees used AI coding tools, representing 60% of the company and 93% of R&D. - AI tools generated 47.95 million requests. - 295 teams used agentic AI tools or coding assistants. - AI Gateway handled 20.18 million requests and routed 241.37 billion tokens. - Workers AI processed 51.83 billion tokens. - The four-week rolling average of merge requests increased from roughly 5,600 per week to more than 8,700. - The week of March 23 reached 10,952 merge requests, nearly twice the Q4 baseline. - Cloudflare’s initial focus was MCP servers, but the effort expanded to standards, code review, onboarding, and propagating changes across thousands of repositories. ## Architecture at a Glance Cloudflare organized the stack into three layers: - **Platform layer:** Authentication, model routing, inference, MCP access, and code execution. - **Knowledge layer:** System context and repository guidance through Backstage and `AGENTS.md`. - **Enforcement layer:** Automated quality controls using AI Code Reviewer and the Engineering Codex. The stack uses Cloudflare products including: - **Cloudflare Access** for zero-trust authentication. - **AI Gateway** for centralized LLM routing, cost tracking, bring-your-own-key support, and zero-data-retention controls. - **Workers AI** for open-weight model inference. - **Workers and Access** for an MCP Server Portal with single OAuth. - **Dynamic Workers** for sandboxed agent-generated code execution. - **Agents SDK and Durable Objects** for stateful, long-running agent sessions. - **Sandbox SDK** for isolated cloning, building, and testing environments. - **Workflows** for durable, multi-step processes. - **Backstage** for a 16,000-plus-entity knowledge graph. ## Centralized Authentication and AI Routing - Cloudflare Access authenticates users and enforces zero-trust policies. - Every LLM request passes through AI Gateway, providing a single control point for: - Provider credentials - Usage and cost attribution - Model selection - Data-retention policies - Provider permissions - In the past month, frontier providers handled 91.16% of requests, while Workers AI handled 8.84%. - Cloudflare routes requests through a proxy Worker rather than connecting clients directly to AI Gateway. - The proxy enables later additions such as per-user attribution, model catalogs, permission enforcement, and support for new coding tools without changing client configurations. ## Workers AI and Open-Weight Models - Workers AI runs open-source models on GPUs distributed across Cloudflare’s global network. - Keeping inference on the same network as Workers, Durable Objects, and storage reduces latency, network failures, and cross-cloud configuration. - Kimi K2.5, with a 256,000-token context window, tool calling, and structured outputs, processes more than 7 billion tokens per day for a Cloudflare security agent. - Cloudflare estimates that running this workload on Workers AI is 77% cheaper than using a mid-tier proprietary model. - Workers AI is also used for: - Documentation review in CI - Generating `AGENTS.md` files - Lightweight inference where latency matters more than maximum model capability - Cloudflare expects open-source models to handle an increasing proportion of its internal workloads. ## One-Command Client Configuration - Engineers begin setup with: ```bash opencode auth login https://opencode.internal.domain ``` - The command uses an OpenCode discovery endpoint at: ```text https://opencode.internal.domain/.well-known/opencode ``` - The Worker-hosted endpoint provides authentication and configuration information. - This mechanism is designed to configure providers, models, MCP servers, agents, commands, and permissions without requiring engineers to edit configuration files manually. ## Overall Recommendation Cloudflare’s experience suggests that organizations adopting AI coding tools should build a centralized control plane early: authenticate users consistently, route model traffic through one managed gateway, maintain shared system knowledge, and enforce quality through automated review and isolated execution. Using the same production platform for internal tooling can also expose product gaps and accelerate improvements to the platform itself.

Read original(opens in new tab)

CI/CDDatabase DesignOpen SourcePostgreSQL+4

cloudflare·Apr 20, 2026·3 min readCurated summary

Orchestrating AI Code Review at scale

Cloudflare built a CI-native AI code review system to reduce review bottlenecks without overwhelming engineers with noisy or generic model feedback. Instead of using one large prompt, it orchestrates up to seven specialized agents for areas such as security, performance, compliance, and documentation, then uses a coordinator to deduplicate and assess findings. The system now reviews tens of thousands of merge requests, approving clean changes and blocking serious bugs or vulnerabilities. ## Why Naive AI Review Wasn’t Enough - Traditional code review can leave merge requests waiting for hours and creates repeated context switching. - Commercial AI review tools provided useful functionality but lacked the flexibility and customization required across Cloudflare’s organization. - A basic “send the Git diff to an LLM” approach produced: - Vague recommendations - Hallucinated syntax errors - Repetitive advice such as adding error handling where it already existed - Complex codebases required specialized analysis rather than generic summarization. ## Specialized Agents and Coordination - The system launches up to seven focused reviewers covering: - Security - Performance - Code quality - Documentation - Release management - Internal Engineering Codex compliance - A coordinator agent: - Deduplicates overlapping findings - Evaluates the actual severity of issues - Produces one structured review comment - The system can actively block merges when it detects serious defects or security vulnerabilities. ## Plugin-Based Architecture - The platform uses composable plugins so it can support different: - Version-control systems - AI providers - Internal standards - Repository-specific requirements - Each plugin implements a `ReviewPlugin` interface with three lifecycle phases: - `bootstrap`: Runs concurrently and is non-fatal. - `configure`: Runs sequentially and is fatal if essential configuration fails. - `postConfigure`: Handles asynchronous work after configuration assembly. - Through `ConfigureContext`, plugins can: - Register agents and AI providers - Set environment variables - Inject prompt sections - Configure agent permissions - Plugins contribute through the context API rather than accessing the final configuration directly. - The core assembler combines these contributions into `opencode.json`. - This separation prevents unrelated components from becoming tightly coupled; for example, GitLab logic does not need to understand Cloudflare AI Gateway settings. ## Plugin Responsibilities - `@opencode-reviewer/gitlab` - Provides GitLab merge request data and a comment server. - `@opencode-reviewer/cloudflare` - Configures AI Gateway model tiers and fallback chains. - `@opencode-reviewer/codex` - Checks compliance with internal engineering RFCs. - `@opencode-reviewer/braintrust` - Adds distributed tracing and observability. - `@opencode-reviewer/agents-md` - Verifies that repository `AGENTS.md` instructions are current. - `@opencode-reviewer/reviewer-config` - Retrieves remote model overrides for individual reviewers. - `@opencode-reviewer/telemetry` - Tracks reviews asynchronously. ## Why OpenCode - Cloudflare already used OpenCode extensively and understood its behavior. - Its open-source implementation allows engineers to: - Investigate problems directly - Contribute fixes upstream - Extend the system through its SDK - Cloudflare engineers had contributed more than 45 upstream pull requests at the time of writing. - Its server-first design was especially important: - Review sessions can be created programmatically. - Prompts can be sent through an SDK. - Multiple concurrent sessions can be managed without scraping or wrapping a CLI interface. ## Coordinator Process - The coordinator runs OpenCode as a child process using `Bun.spawn`. - Its prompt is passed through `stdin` rather than a command-line argument. - This avoids Linux’s `ARG_MAX` limit, which previously caused `E2BIG` failures for unusually large merge requests containing extensive descriptions or logs. - OpenCode runs with `--format json`, emitting JSONL events through standard output. - This event-based interface allows the orchestration layer to collect and process results from concurrent reviewer sessions. A practical takeaway is to treat AI review as an orchestrated CI system rather than a single LLM prompt. Specialized agents, strict plugin boundaries, structured outputs, and observability are essential for making automated review reliable enough to influence merge decisions at organizational scale.

Read original(opens in new tab)

CI/CDDatabase DesignLarge Language ModelsGitlab+4

gitlab·Apr 16, 2026·2 min readCurated summary

Claude Opus 4.7 is now available in GitLab Duo Agent Platform

GitLab Duo Agent Platform now supports Anthropic’s Claude Opus 4.7 across Agentic Chat and agent-powered software delivery workflows. The model is designed to improve long-running, multistep tasks through stronger reasoning, instruction following, and self-verification. GitLab says this should make agents more reliable across development, security, CI/CD, and deployment. ## Improved Reasoning and Instruction Following - Internal evaluations reportedly show Opus 4.7 outperforming Sonnet 4.6 and Opus 4.6. - It handles complex, conditional instructions more precisely. - Agents can complete multistep tasks with fewer errors and more predictable, auditable results. - Self-verification helps agents check generated code and tests before presenting them. ## Support Across the Software Lifecycle - **Development:** Faster code generation and test creation with less developer back-and-forth. - **Security:** More reliable vulnerability remediation through complete, correctly scoped sequences. - **CI/CD:** Better continuity when investigating pipeline failures, analyzing logs, and proposing fixes. - **Cross-stage workflows:** The model supports coordination across planning, coding, security, and deployment. ## Availability and Pricing - Claude Opus 4.7 is available now through model selection in GitLab Duo Agent Platform. - Model credit consumption details are provided in GitLab’s documentation. - New users can start a free trial. - GitLab Premium and Ultimate subscribers can enable Duo Agent Platform and use included GitLab Credits. Teams using GitLab’s agent workflows can adopt Opus 4.7 to improve reliability on complex, multi-tool tasks spanning the full software delivery lifecycle.

Read original(opens in new tab)

CI/CDMachine LearningGitlabCode Review+3

cloudflare·Apr 14, 2026·3 min readCurated summary

Scaling MCP adoption: Our reference architecture for simpler, safer and cheaper enterprise deployments of MCP

Cloudflare argues that enterprise MCP adoption requires centralized governance rather than individually managed, locally hosted servers. Its reference architecture combines remote MCP servers, Cloudflare Access, MCP server portals, and AI security controls to improve visibility, authentication, policy enforcement, and performance. The company also introduces Code Mode with MCP server portals to reduce the token and context-window costs of exposing large APIs. ## Centralized Remote MCP Servers - MCP separates the AI application from corporate credentials and APIs: - The MCP client connects to the LLM or agent. - The MCP server mediates access to internal resources. - Cloudflare moved away from locally hosted MCP servers because they: - May use unvetted software and versions. - Increase supply-chain and tool-injection risks. - Are difficult for IT and security teams to administer. - A centralized team manages MCP infrastructure through a shared monorepo platform. - Approved teams can create governed MCP servers from templates, inheriting: - Default-deny write controls. - Audit logging. - Automated CI/CD pipelines. - Secrets management. - Servers are deployed remotely on Cloudflare’s developer platform and custom domains, providing centralized usage visibility and global low-latency access. ## Authentication with Cloudflare Access - Public MCP servers, such as documentation and Radar services, can remain openly accessible. - MCP servers connected to private corporate resources require employee authentication. - Cloudflare Access acts as the OAuth provider and identity layer. - It verifies: - Single sign-on. - Multifactor authentication. - IP address, location, and device-certificate context. - Access issues tokens that authorize users to reach protected resources. ## MCP Server Portals for Discovery and Governance - As the number of MCP servers grew, employees needed a central way to discover authorized services. - Users connect their MCP client to a portal, which exposes the internal and third-party MCP servers they are permitted to use. - Portals provide: - Centralized logging. - Consistent policy enforcement. - Data loss prevention controls. - Access policies for users and tools. - Administrators can restrict both portal access and the specific tools exposed by each server. - Finance users might receive only read-only repository tools. - Engineering users on corporate devices might receive read/write capabilities. - Portals support MCP servers hosted on Cloudflare as well as third-party servers. - Cloudflare emphasizes that the relevant security and networking components can run on the same physical machine in its global network, reducing latency and avoiding unnecessary traffic transit. ## Code Mode Reduces MCP Token Costs - The standard MCP design exposes every API operation as a separate tool. - For large platforms with thousands of endpoints, this exhaustive tool list consumes an agent’s context window and increases token costs. - Cloudflare presents Code Mode with MCP server portals as a way to address this scaling problem. - The provided article excerpt ends while introducing Cloudflare’s earlier use of server-side Code Mode for exposing large numbers of API endpoints. Cloudflare’s approach recommends treating MCP as enterprise infrastructure: centrally deployed, authenticated, discoverable, policy-controlled, and monitored. Organizations adopting MCP at scale should avoid unmanaged local servers and provide reusable platforms that make secure deployment the default.

Read original(opens in new tab)

CI/CDModel Context ProtocolCloudflareAPI Design+4

gitlab·Apr 13, 2026·3 min readCurated summary

GitLab named a 2026 Omdia Universe Leader

GitLab was named a Leader in Omdia’s 2026 Universe for AI-assisted Software Development, IDE-based Tools, ranking among 19 vendors. Its strongest results came from covering the entire software lifecycle—not just code generation—including planning, security, testing, deployment, and operations. The report suggests that AI delivers the greatest productivity gains when automation extends beyond coding into coordinated, governed delivery. ## Omdia’s Broader Evaluation - Omdia expanded its criteria to assess full software lifecycle capabilities. - The report emphasized that faster coding alone can create downstream bottlenecks in: - Code review - Security remediation - Testing - Deployment coordination - Agentic AI was evaluated as a current capability, including: - Autonomous task coordination - Handoffs between specialized agents - Support for teams at different stages of AI adoption - Omdia categorizes vendors as Leaders, Challengers, or Prospects based on capability and strategy/execution. ## GitLab’s Top Scores - **Solution Breadth: 100%** - Covers planning, requirements, development, security, deployment, and issue management in one platform. - Planner Agent and Security Analyst Agent extend AI into sprint planning, vulnerability triage, and remediation guidance. - **Strategy and Innovation: 88%** - Uses end-to-end orchestration and a privacy-first architecture that does not train on private customer data. - Supports multiple models through partnerships with Anthropic, Google, and AWS. - Provides shared context across issues, merge requests, pipelines, and security findings. - **Core Features: 82%** - Offers context-aware code generation, unit and integration testing, security testing, and review prioritization. - Automates CI/CD, GitOps, and pipeline-failure root cause analysis. - The AI Impact Dashboard tracks cycle time, deployment frequency, and productivity effects. - GitLab also received top-tier scores for Extended Features (80%) and Vendor Execution (88%). ## Developers and AI Agents - Teams are increasingly structured around engineers supervising AI agents. - Human responsibilities are shifting toward: - Defining requirements and guardrails - Supervising quality and security - Designing autonomous production pipelines - Connecting business objectives with agentic systems - Automating only code generation provides limited benefit if review, testing, and deployment remain manual. ## Enterprise Readiness - Omdia treated compliance, privacy, and deployment flexibility as baseline requirements for Leader-tier platforms. - GitLab highlights: - SOC 2 and ISO 27001 certification - No training on private customer data for agentic AI - Self-managed, cloud, on-premises, and air-gapped deployment - Support for self-hosted AI models - GitLab Dedicated, including FedRAMP Moderate authorization for government - These capabilities target regulated industries requiring strong data residency, auditability, and governance. GitLab’s central argument is that AI coding speed matters only when the rest of the software delivery lifecycle can keep pace. Engineering teams should evaluate AI platforms by their ability to deliver secure, governed, production-ready software—not merely by how much code they can generate.

Read original(opens in new tab)

CI/CDDatabase DesignAI AgentsGitlab+4

gitlab·Apr 9, 2026·2 min readCurated summary

5 ways GitLab pipeline logic solves engineering problems

GitLab’s pipeline model addresses complex CI/CD needs by combining composable features rather than relying on a single linear workflow. Parent-child pipelines, DAG execution, and multi-project triggers help teams scale monorepos and coordinate services across repositories while preserving clear ownership and failure visibility. The article argues that these patterns make pipelines both faster and easier to maintain. ## Monorepos: Parent-child pipelines and DAG execution - A monorepo containing frontend, backend, and documentation projects should not rebuild everything for every change. - Parent pipelines can trigger child pipelines for individual services using `trigger: include`. - Multiple included files are merged into one child pipeline, allowing jobs across files to share context and reference one another with `needs:`. - `strategy: depend` makes the parent wait for child pipelines and report one overall success or failure while retaining detailed drill-down. - Each service can own its pipeline configuration, reducing the risk that changes in one service break another. - DAG execution with `needs:` allows dependent jobs to start as soon as their prerequisites finish instead of waiting for an entire stage. - For example, API tests can begin immediately after the API build completes, without waiting for unrelated jobs. ## Microservices: Cross-repository pipelines - When frontend and backend services live in separate repositories, independent pipelines may miss integration failures. - GitLab multi-project pipelines allow one repository to trigger and await a pipeline in another project. - The frontend can generate an API contract artifact, publish it, and trigger the backend pipeline with `strategy: depend`. - The backend downloads the artifact through the GitLab Jobs API using `CI_JOB_TOKEN`. - An integration test can reject breaking API changes and propagate the failure back to the frontend pipeline. - The backend job uses `CI_PIPELINE_SOURCE == "pipeline"` so the contract validation runs only when initiated by the frontend, not during ordinary backend pushes. - The frontend project identifier is supplied through a CI/CD variable such as `FRONTEND_PROJECT_ID`. These patterns let teams reduce unnecessary work, preserve service-level ownership, and make cross-service compatibility checks part of the delivery process.

Read original(opens in new tab)

CI/CDMicroservicesDevOpsMonorepo+4