How to build CI/CD observability at scale
CI/CD observability is essential for improving pipeline performance at enterprise scale, particularly in self-managed GitLab environments. The post presents a containerized solution built with `gitlab-ci-pipelines-exporter`, Prometheus, Grafana, and Node Exporter to turn pipeline and infrastructure data into actionable insights. Its conclusion is that centralized dashboards help teams identify bottlenecks, plan runner capacity, and measure delivery performance. ## Defining CI/CD Performance - Teams should first determine: - Which metrics matter, such as pipeline duration, job success rates, queue times, and runner utilization. - Who needs access, including developers, DevOps engineers, platform teams, and leadership. - Which decisions the data will support, such as infrastructure investment, bottleneck remediation, and capacity planning. ## Observability Architecture - The solution uses two exporters: - **Pipeline Exporter:** Collects pipeline duration, job status, and deployment metrics through the GitLab API. - **Node Exporter:** Collects host CPU, memory, and disk metrics for infrastructure correlation. - Prometheus gathers and stores the metrics. - Grafana provides real-time and historical dashboards. - Dashboards are provisioned automatically through Grafana’s file-based provisioning and can be filtered by project, branch, or time range. ## Grafana Dashboards - **Pipeline Overview:** Displays pipeline volume, success and failure rates, cancelled runs, and average duration trends. - **Job Performance:** Shows job-duration histograms, the ten slowest jobs, and failure heatmaps by project and stage. - **Runner & Infrastructure:** Correlates runner queue times with CPU, memory, and disk usage to support capacity planning. - **Deployment Frequency:** Tracks deployment counts and durations by environment, supporting DORA-style delivery analysis and detection of environment drift. ## Kubernetes Deployment - The recommended enterprise deployment runs each component as a separate workload in a dedicated `gitlab-observability` namespace. - A Kubernetes secret stores the GitLab personal access token, which requires the `read_api` scope. - The Pipeline Exporter runs as a Deployment with a service on port `8080`. - Node Exporter runs as a DaemonSet so each node can expose host metrics on port `9100`. - Prometheus and Grafana are deployed alongside the exporters and configured to scrape and visualize their metrics. - Kubernetes deployment supports existing cluster infrastructure, secrets managers, network policies, and scalable operations. ## Prerequisites - GitLab Self-Managed 18.1 or later. - Kubernetes for enterprise deployments, or Docker/Podman for smaller environments and proof-of-concept testing. - A GitLab personal access token with `read_api` permissions. - Secure secret-management practices, preferably using external secret operators in production. The practical recommendation is to begin with clearly defined performance questions, then deploy the exporter–Prometheus–Grafana stack in a controlled namespace. Combining pipeline data with host metrics provides the context needed to distinguish inefficient jobs from infrastructure capacity problems.
Read original(opens in new tab)