DevOps

21 posts

gitlab3 min readCurated summary

Green DevOps: Why carbon measurement belongs in your CI/CD pipeline

CI/CD pipelines consume energy and generate emissions that standard pipeline metrics do not reveal. The post argues that measuring this impact is the first step toward reducing it, recommending Eco CI for job-level measurements and Carmen for infrastructure and application-level analysis. Both tools integrate with GitLab pipelines with minimal effort and can improve sustainability, performance, and cloud costs. ## Why CI/CD Carbon Measurement Matters - Modern teams run hundreds of jobs daily, including increasingly resource-intensive AI-assisted testing and automation. - Each job has an energy and carbon cost that is usually absent from pipeline logs and architecture diagrams. - Green DevOps involves measuring emissions per pipeline, service, and Kubernetes workload to guide engineering decisions. ## Measuring Emissions at Different Layers ### Pipeline-level measurement with Eco CI - Eco CI measures energy consumption and carbon emissions for individual CI/CD jobs. - It runs as lightweight Bash scripts without additional servers or databases. - Teams can identify expensive jobs, track emissions over time, and add a carbon badge to a project README. - Because it works within existing pipelines, it is a practical starting point. ### Infrastructure and application measurement with Carmen - Carmen, the Carbon Measurement Engine, uses the Green Software Foundation Impact Framework. - It measures emissions from virtual machines, Kubernetes pods, and application workloads. - Reports are provided per component in CSV format and distinguish: - Operational carbon from energy consumption - Embodied carbon from hardware manufacturing and disposal - Outputs such as `EnergykWh` and `TotalCarbonGramsCO2eq` can be imported into Grafana, FinOps dashboards, or custom tools. - Carmen helps teams compare emissions across services, such as an API gateway versus a data-processing layer. ## Adding Carbon Reports to GitLab - Both tools can be integrated through `.gitlab-ci.yml`. - A Carmen job can: - Use a Python 3.12 image - Install Node.js, npm, Git, and the Impact Framework toolchain - Clone and install the Carmen project - Run `carbon-daemon` - Store generated reports as GitLab artifacts - Carmen can run as a separate, non-blocking job, keeping carbon measurement outside the critical delivery path. ## Turning Visibility into Engineering Improvements - Eco CI may reveal that integration tests produce unusually high emissions because dependencies are reinstalled on every run. - Adding dependency caching can reduce runtime, emissions, and CI costs without architectural changes. - Carmen can expose idle infrastructure, such as deprecated services still running in a staging Kubernetes cluster. - Decommissioning unused workloads eliminates both their operational and embodied carbon costs. ## Business and Regulatory Benefits - Emissions data creates a baseline for future reporting requirements, even when a company’s absolute footprint is relatively small. - Carbon-efficient systems are often faster and cheaper because they use fewer compute resources. - Caching, right-sizing runners, and removing idle services support both sustainability and FinOps goals. - Regulations such as the EU’s CSRD, along with enterprise procurement requirements, are increasing pressure for supply-chain and cloud-emissions reporting. Teams can start with Eco CI on one GitLab pipeline, then add Carmen for infrastructure-level visibility. Measuring carbon alongside runtime and cost makes sustainability a practical part of everyday engineering rather than a separate initiative.

Read original(opens in new tab)
aws3 min readCurated summary

Top announcements of the AWS Summit in New York, 2026 | Amazon Web Services

AWS Summit New York 2026 focused on making AI agents more capable, secure, autonomous, and easier to operate in production. Announcements spanned Amazon Bedrock AgentCore, security automation, developer tooling, workplace agents, and S3 data management. The overall direction is toward agents that can access governed knowledge, act independently, and continuously improve while remaining subject to enterprise controls. ## Building More Capable Agents - **Amazon Bedrock Managed Knowledge Base** simplifies enterprise RAG with native data connectors, Smart Parsing for multi-format data, and an Agentic Retriever for complex, multi-step queries. - **Web Search for Bedrock AgentCore** provides managed, current, cited web knowledge while keeping data within the customer’s secured AWS environment. - **Bedrock AgentCore harness** is now generally available, allowing developers to define an agent’s model, tools, skills, and instructions through configuration rather than custom orchestration loops. - **AWS Context**, coming soon, will map relationships across organizational data into a knowledge graph. Agents will be able to use governed relationships, business rules, and domain knowledge at runtime. - **AWS WAF Bot Control** lets publishers and content owners price, meter, and collect payment from AI bots accessing content and APIs, with scoped access enforced at the edge. ## Securing Agents and Applications - **AWS Continuum**, available in gated preview, aggregates vulnerability findings, ranks them by business impact, verifies exploitability, and routes fixes through existing development processes. - **AWS Security Agent**, now part of Continuum, adds: - Threat modeling based on the STRIDE framework - Pull-request code scanning and remediation across major Git platforms - IDE integrations through Kiro, Claude Code, and MCP - These tools are intended to let developers perform security reviews and address vulnerabilities without leaving their normal workflows. ## AI-Assisted Software Development - **Kiro for iOS**, in gated preview, lets developers start, monitor, steer, and approve Kiro sessions from a phone, including reviewing diffs and approving changes without keeping a laptop running. - **AWS DevOps Agent** adds release readiness reviews and autonomous release testing. It evaluates changes against natural-language standards and tests them in production-like environments. - **AWS Transform continuous modernization**, in preview, scans repositories against configurable technical-debt baselines and can autonomously generate remediation pull requests. ## Autonomous Workplace Agents - New **Amazon Quick autonomous agents** can work in the background with defined expertise, tone, permissions, and tools. - Example use cases include: - A finance agent processing incoming orders - A sales agent analyzing CRM, email, and Slack activity - Agents drafting follow-ups, identifying risks, and recommending next steps - A new **activity feed** combines email, messaging, calendars, and tasks into a prioritized view that adapts to the user’s communication and work patterns. ## Richer Metadata for Amazon S3 - **S3 annotations** allow up to 1 GB of mutable, queryable context to be attached directly to an object. - The feature targets AI agents and autonomous workflows that need to discover and interpret data without maintaining separate metadata systems. AWS’s announcements point toward an agent ecosystem that combines managed knowledge retrieval, web access, security automation, autonomous development workflows, and persistent workplace assistance. Organizations adopting these capabilities should pair autonomy with strong governance, scoped permissions, and continuous validation in production.

Read original(opens in new tab)
gitlab2 min readCurated summary

Introducing the 2026 EMEA GitLab Partner Award winners

GitLab announced its 2026 EMEA Partner Award winners, recognizing organizations that drove customer success, technical innovation, certification, business growth, and joint marketing. The awards highlight partners helping enterprises adopt DevSecOps, cloud-native platforms, managed services, and AI-enabled software development across the region. ## Regional Partners of the Year - **Central Europe: cc cloud GmbH** — Combines infrastructure and DevOps expertise to manage cloud applications, platforms, and IT operations. - **Northern Europe: Eficode** — Supports more than 1,600 customers through consulting, managed services, toolchain implementation, and AI-augmented development. - **Southern Europe: Kiratech** — Helps enterprises modernize infrastructure using cloud-native, DevOps, and PlatformOps practices. - **Eastern Europe and Israel: Bynet** — An established systems integrator supporting enterprise IT, cloud, cybersecurity, modernization, DevSecOps, and AI adoption. ## Technical and Enablement Awards - **Best Technical Solution/Project: Capgemini | Sogeti** — Recognized for impactful, complex technical solutions using AI-driven quality engineering, data, and cloud capabilities. - **Most Certified and Enabled Partner: Devoteam** — Awarded for having the largest number of GitLab-certified professionals. - **Rookie of the Year: ITDOTCOM** — A Uzbekistan-based technology distributor that achieved rapid success supporting software, infrastructure, cybersecurity, and business automation across Central Asia. ## Growth and Collaboration Awards - **First Order Master: Linux Polska** — Recognized for winning new customers and business through open-source consulting, DevOps, automation, containerization, and data analytics. - **Co-marketing Partner of the Year: Conoa, a PROACT Company** — Honored for joint marketing efforts and expertise in Kubernetes, cloud-native technologies, container platforms, and managed operations. The awards demonstrate the breadth of GitLab’s EMEA partner ecosystem, from regional systems integrators and cloud specialists to technical consultants and Kubernetes providers. Together, these partners are helping customers modernize delivery practices and adopt DevSecOps and AI capabilities.

Read original(opens in new tab)
aws3 min readCurated summary

AWS Weekly Roundup: Claude Opus 4.7 in Amazon Bedrock, AWS Interconnect GA, and more (April 20, 2026) | Amazon Web Services

The roundup highlights major AWS advances in AI, networking, developer tooling, and security. Claude Opus 4.7 is now available through Amazon Bedrock with stronger agentic coding and research capabilities, while AWS Interconnect simplifies private connectivity across clouds and remote locations. Additional launches improve container supply-chain security, application modernization, database access, cost attribution, and quantum-resistant encryption. ## Anthropic Claude Opus 4.7 in Amazon Bedrock - Anthropic’s latest Opus model improves: - Agentic coding and long-running tasks - Complex code reasoning - Document creation, financial analysis, and multi-step research - It scores: - 64.3% on SWE-bench Pro - 87.6% on SWE-bench Verified - Bedrock features include: - Dynamic capacity allocation - Adaptive thinking and request-specific token budgets - A 1-million-token context window - High-resolution image support for charts, documents, and screen interfaces - The model launched in US East, Tokyo, Ireland, and Stockholm, supporting up to 10,000 requests per minute per account and Region. ## AWS Interconnect Reaches General Availability - **AWS Interconnect – Multicloud** provides Layer 3 private connectivity between AWS VPCs and other clouds. - Google Cloud is supported initially; Azure and OCI are planned. - Traffic uses private networks and the AWS global backbone rather than the public internet. - Includes MACsec encryption, multi-facility resilience, and CloudWatch monitoring. - The underlying specification is open source under Apache 2.0. - **AWS Interconnect – Last Mile** connects branches, data centers, and remote sites to AWS through network providers. - Automatically provisions four redundant connections across two physical locations. - Configures BGP, MACsec, and Jumbo Frames. - Supports adjustable bandwidth from 1 to 100 Gbps. - Launches in US East with Lumen. ## Developer, Database, and Modernization Updates - Amazon ECR pull-through cache now discovers and synchronizes OCI referrers such as signatures, SBOMs, and attestations. - AWS Transform is available directly in Kiro and VS Code for migrations such as language-version upgrades and AWS SDK updates. - Aurora DSQL’s PHP connector supports IAM authentication, SSL, connection pooling, and optional optimistic-concurrency retries. - AWS Transform Custom can modernize VB6 applications into C# ASP.NET Core applications, including COM, ADO, and UI migration challenges. ## Security, Access Control, and Cost Management - Amazon Q for Google Drive now enforces document-level permissions using indexed ACLs and real-time access checks. - AWS Secrets Manager supports hybrid post-quantum TLS using ML-KEM through updated agents, Lambda extensions, and CSI drivers. - Amazon Bedrock can attribute inference costs to individual IAM principals, with reporting through CUR 2.0 and aggregation by teams, projects, or cost centers. ## Compute, Kubernetes, and Storage - EC2 C8in and C8ib instances use sixth-generation Intel Xeon processors and AWS Nitro cards. - C8in offers up to 600 Gbps networking. - C8ib provides up to 300 Gbps EBS bandwidth. - Both scale to 384 vCPUs. - EKS Auto Mode automates networking components such as VPC CNI, load balancers, and DNS while retaining enterprise security controls. - EBS Volume Clones provide immediately usable point-in-time copies for development, disaster recovery testing, and CI/CD workflows. ## Additional AWS Guidance - CloudFront Functions and CloudFront KeyValueStore can support zero-downtime API decomposition using user-aware routing and the Strangler Fig pattern. - The roundup also points readers to AWS events, weekly Power Hour training, and Community.aws meetups. The most significant developments are Bedrock’s expanded AI capabilities and Interconnect’s managed private networking. Teams should evaluate Claude Opus 4.7 for complex AI workflows, use Interconnect where multicloud or resilient connectivity is required, and consider the new security and cost-attribution features for stronger governance.

Read original(opens in new tab)
gitlab2 min readCurated summary

CI Expert and Data Analyst AI agents target development gaps

GitLab 18.11 introduces two Duo Agent Platform agents aimed at development gaps left by faster AI-generated coding. CI Expert Agent helps teams create working pipelines, while Data Analyst Agent answers software delivery questions using GitLab data. Both benefit from being embedded in GitLab, where they can use current repository, pipeline, issue, and merge request context. ## CI Expert Agent: Faster Pipeline Setup - Now available in beta. - Inspects a repository to identify its language, framework, and testing setup. - Generates runnable build and test configurations without requiring developers to write YAML manually. - Explains each pipeline step in plain language through Agentic Chat. - Uses native GitLab CI semantics. - Addresses the common problem of delaying CI because `.gitlab-ci.yml` is difficult to create or validate. - Helps reduce regressions, oversized changes, and dependence on undocumented team knowledge. - Available on GitLab.com, Self-Managed, and Dedicated across Free, Premium, and Ultimate editions with Duo Agent Platform enabled. ## Data Analyst Agent: Natural-Language SDLC Queries - Generally available in GitLab 18.11. - Lets users ask questions about development performance in plain language and receive visualizations in Agentic Chat. - Covers merge requests, issues, projects, pipelines, and jobs. - Supports questions about: - MR cycle time and review bottlenecks - Project throughput and contribution patterns - Flaky tests and pipeline performance - Runner utilization and deployment frequency - Cross-portfolio lead times and project health - Eliminates the need to learn GitLab Query Language, request custom dashboards, or rely on separate analytics tools. - Generated GLQL queries can be copied into GitLab Flavored Markdown. - Exporting results to work items and dashboards is planned. - Available across GitLab.com, Self-Managed, and Dedicated Free, Premium, and Ultimate editions with Duo Agent Platform enabled. ## The Advantage of Platform-Native Context - Both agents operate within GitLab and can access existing code, pipelines, issues, and merge requests. - Their recommendations and answers are based on current operational data rather than generic examples or disconnected tools. - The agents are designed to support the full lifecycle: understanding code, configuring CI, shipping changes, and evaluating delivery performance. Together, these agents make GitLab Duo more useful beyond code generation. Teams should consider trying CI Expert Agent for faster initial pipeline setup and Data Analyst Agent for immediate, self-service insight into delivery performance.

Read original(opens in new tab)
gitlab1 min readCurated summary

GitLab 18.11 release notes | GitLab Docs

Agentic SAST Vulnerability Resolution is now generally available in GitLab 18.11 through the GitLab Duo Agent Platform. It autonomously analyzes critical and high-severity SAST findings, proposes code fixes, and creates ready-to-review merge requests. The feature is designed to accelerate remediation while giving reviewers confidence assessments and the option to apply fixes directly from vulnerability details. ## Availability and Integration - Available with GitLab Ultimate. - Supported on GitLab.com, GitLab Self-Managed, and GitLab Dedicated. - Runs: - As part of a SAST scan - After SAST false-positive detection - Manually for individual SAST vulnerabilities ## Automated Vulnerability Analysis - Examines each finding and reasons through the surrounding code context. - Generates proposed fixes for critical and high-severity SAST vulnerabilities. - Automatically opens a merge request containing the remediation for review. ## Review and Remediation Workflow - Provides quality assessments to help reviewers evaluate the likely reliability of each proposed fix. - Lets users apply resolutions directly from vulnerability details pages. - Feedback is welcomed through GitLab issue 585626. ## Practical Recommendation Teams using GitLab Ultimate can enable the feature to streamline SAST remediation, while still reviewing agent-generated merge requests before merging changes.

Read original(opens in new tab)
gitlab3 min readCurated summary

A guide to the breaking changes in GitLab 19.0

GitLab 19.0 is expected to introduce 15 breaking changes, primarily by removing deprecated components and outdated platform support. The most significant effects involve Helm chart networking and bundled services, OAuth authentication, PostgreSQL, Redis, and supported operating systems. Administrators should audit their deployments and complete migrations before upgrading. ## Release and Deployment Windows - **GitLab.com:** Primary breaking-change window is May 4–6, 2026, with a fallback window on May 11–13. - **GitLab Self-Managed:** GitLab 19.0 becomes available May 21, 2026. - **GitLab Dedicated:** Upgrades occur during assigned maintenance windows, with GitLab 19.0 scheduled for the week of June 22, 2026. - Additional changes may roll out outside these windows in exceptional circumstances. ## High-Impact Changes ### NGINX Ingress Replaced by Gateway API - The GitLab Helm chart will use **Gateway API with Envoy Gateway** as its default networking configuration. - Bundled NGINX Ingress reached end-of-life in March 2026. - Existing deployments can explicitly continue using bundled NGINX Ingress until its planned removal in GitLab 20.0. - The change does not affect: - NGINX used by the Linux package. - Deployments using externally managed Ingress or Gateway API controllers. - Administrators should plan migration to Envoy Gateway or another externally managed controller. ### Bundled PostgreSQL, Redis, and MinIO Removed - The GitLab Helm chart and GitLab Operator will no longer bundle Bitnami PostgreSQL, Bitnami Redis, or the forked MinIO chart. - These components were intended for proof-of-concept and test environments, not production. - Deployments using them must migrate to external services before upgrading. - PostgreSQL and Redis bundled with the Linux package are unaffected. ### OAuth ROPC Grant Removed - The Resource Owner Password Credentials OAuth flow will be removed across GitLab.com, Self-Managed, and Dedicated. - ROPC is being eliminated because of security limitations and its removal from OAuth 2.1. - Applications using ROPC must migrate to a supported flow, such as Authorization Code. - After upgrading, ROPC will not work even when client credentials are provided. ### PostgreSQL 17 Becomes Required - PostgreSQL 16 will no longer be supported; PostgreSQL 17 becomes the minimum version. - Single PostgreSQL instances installed through the Linux package may be upgraded automatically during GitLab 18.11. - Cluster deployments and installations that opt out of automatic upgrades require a manual migration. - Administrators should verify sufficient disk space and complete the upgrade before GitLab 19.0. ## Medium-Impact Changes ### Ubuntu 20.04 Packages Discontinued - GitLab will stop publishing Linux packages for Ubuntu 20.04. - GitLab 18.11 is the final release supporting that distribution. - Affected installations must upgrade to Ubuntu 22.04 or another supported operating system first. ### Redis 6 Support Removed - External Redis 6 deployments must migrate to Redis 7.2 or Valkey 7.2. - The Linux package’s bundled Redis is unaffected because it has used Redis 7 since GitLab 16.2. - Migration options vary by provider: - AWS ElastiCache and GCP Memorystore: Redis 7.2 or Valkey 7.2. - Azure: self-host Redis or Valkey on VMs or AKS until managed support is available. - Self-hosted installations: upgrade directly to Redis 7.2 or Valkey 7.2. ### Auto DevOps Builder Image Updated - The CNB builder image used by Auto DevOps changes from `heroku/builder:22` to `heroku/builder:24`. - Pipelines relying on the older image may need testing or configuration updates. GitLab administrators should review the deprecations and upgrade documentation, identify whether their deployment uses any affected components, and complete required migrations before GitLab 19.0.

Read original(opens in new tab)
gitlab2 min readCurated summary

5 ways GitLab pipeline logic solves engineering problems

GitLab’s pipeline model addresses complex CI/CD needs by combining composable features rather than relying on a single linear workflow. Parent-child pipelines, DAG execution, and multi-project triggers help teams scale monorepos and coordinate services across repositories while preserving clear ownership and failure visibility. The article argues that these patterns make pipelines both faster and easier to maintain. ## Monorepos: Parent-child pipelines and DAG execution - A monorepo containing frontend, backend, and documentation projects should not rebuild everything for every change. - Parent pipelines can trigger child pipelines for individual services using `trigger: include`. - Multiple included files are merged into one child pipeline, allowing jobs across files to share context and reference one another with `needs:`. - `strategy: depend` makes the parent wait for child pipelines and report one overall success or failure while retaining detailed drill-down. - Each service can own its pipeline configuration, reducing the risk that changes in one service break another. - DAG execution with `needs:` allows dependent jobs to start as soon as their prerequisites finish instead of waiting for an entire stage. - For example, API tests can begin immediately after the API build completes, without waiting for unrelated jobs. ## Microservices: Cross-repository pipelines - When frontend and backend services live in separate repositories, independent pipelines may miss integration failures. - GitLab multi-project pipelines allow one repository to trigger and await a pipeline in another project. - The frontend can generate an API contract artifact, publish it, and trigger the backend pipeline with `strategy: depend`. - The backend downloads the artifact through the GitLab Jobs API using `CI_JOB_TOKEN`. - An integration test can reject breaking API changes and propagate the failure back to the frontend pipeline. - The backend job uses `CI_PIPELINE_SOURCE == "pipeline"` so the contract validation runs only when initiated by the frontend, not during ordinary backend pushes. - The frontend project identifier is supplied through a CI/CD variable such as `FRONTEND_PROJECT_ID`. These patterns let teams reduce unnecessary work, preserve service-level ownership, and make cross-service compatibility checks part of the delivery process.

Read original(opens in new tab)
aws3 min readCurated summary

AWS Weekly Roundup: AWS DevOps Agent & Security Agent GA, Product Lifecycle updates, and more (April 6, 2026) | Amazon Web Services

The April 6, 2026 AWS Weekly Roundup highlights the general availability of AWS DevOps Agent and AWS Security Agent, autonomous “frontier agents” designed to handle complex operational and security tasks. It also reviews AWS service lifecycle changes and summarizes notable product launches and technical updates from the previous week. The overall message is that AWS is expanding agentic automation while helping customers manage service transitions and adopt new capabilities. ## AWS DevOps Agent and Security Agent Reach GA - **AWS DevOps Agent** - Investigates incidents, accelerates resolution, and helps prevent recurring problems. - Works continuously across multiple steps until an operational goal is complete. - Customers report up to **75% lower mean time to resolution (MTTR)** and **3–5 times faster incident resolution**. - Western Governors University reduced resolution times from hours to minutes. - **AWS Security Agent** - Provides continuous, context-aware penetration testing during the software development lifecycle. - Operates similarly to a human penetration tester. - LG CNS reported testing that was more than **50% faster**, approximately **30% less expensive**, and produced fewer false positives. - **Deployment flexibility** - Both agents support AWS, multicloud, and on-premises environments. - They are intended to automate repetitive investigative and testing work while allowing teams to focus on higher-value activities. ## AWS Service Lifecycle Changes AWS updated its Product Lifecycle Changes guidance on March 31, 2026, including migration recommendations and alternative services. - Services with availability changes or maintenance guidance include: - AWS App Runner - AWS Audit Manager - AWS CloudTrail Lake - AWS Glue Ray jobs - AWS IoT FleetWise - Amazon Application Recovery Controller Readiness Check - Amazon Comprehend features such as Topic Modeling and Prompt Safety Classification - Amazon Rekognition streaming and batch moderation features - Amazon SNS Message Data Protection - Services listed as entering sunset include: - AWS Service Management Connector - Amazon RDS Custom for Oracle - Amazon WorkMail - Amazon WorkSpaces Thin Client - **Amazon Chime SDK Proxy Sessions** is reaching sunset. AWS recommends reviewing the relevant service documentation or contacting Support to reduce operational disruption. ## Notable AWS Launches - Amazon ECS introduced **Managed Daemons for ECS Managed Instances**. - The AWS Sustainability console now consolidates **Scope 1–3 emissions reporting**. - **Amazon Bedrock AgentCore Evaluations** became generally available. - AWS Transform added generally available automated codebase analysis. - CloudWatch introduced OpenTelemetry Container Insights for Amazon EKS in preview. - Amazon Lightsail added compute-optimized bundles with up to **72 vCPUs**. - Amazon CloudFront added **SHA-256 support** for signed URLs and signed cookies. ## Additional AWS Resources The roundup also points readers to material on: - Architecting agentic AI applications on AWS. - Reducing data-transfer costs with Network Load Balancers. - Preventing hallucinations in production AI agents. - The AWS World Sports Innovation Cup. - Exploring AWS communities through an interactive 3D globe. AWS also encourages readers to participate in Builder Center discussions, community events, AWS Summits, and developer-focused programs. AWS teams should review the lifecycle notices for services they depend on, while developers and operations groups may benefit from evaluating the new agents and launches for automation, security testing, and observability improvements.

Read original(opens in new tab)
line3 min readCurated summary

Applying Spark on Kubernetes to process large-scale advertising data for LINE services

LINE Ads processes tens of billions of advertising events daily and nearly one hundred billion internal data records. As growing numbers of features increased computational demands, its Spark-on-YARN environment suffered from resource contention, inefficient scaling, and Hadoop dependencies. The team migrated to Spark on Kubernetes to achieve infrastructure independence, containerized execution, flexible scaling, and easier operational automation. ## Large-Scale LINE Ads Data Pipelines - The data pipeline supports: - Real-time advertising-event processing - Abuse and validity checks - Machine-learning systems and model training - Analytics and system integration - Advertiser reporting - The platform must handle hundreds of billions of events per day and hundreds of thousands per second. - It must provide low latency, elastic capacity, minimal service impact during failures, and rapid recovery. - The most heavily used table grew to approximately 2.91 times its December 2022 size by December 2025 as more features were added. ## Limitations of Spark on YARN - Hadoop’s storage and compute resources were colocated, causing Spark workloads to compete with HDFS and other Hadoop components. - Scaling compute required adding Hadoop nodes, even when additional storage was unnecessary, increasing cost and wasting capacity. - JVM and Spark versions were difficult to manage independently, limiting access to newer Spark features. - Applications became tightly coupled to the Hadoop infrastructure. ## How Spark on Kubernetes Works - Kubernetes replaces YARN as the cluster manager. - Spark drivers and executors run as separate Kubernetes pods. - In cluster mode: - `spark-submit` requests a driver pod. - Kubernetes schedules the driver on an appropriate node. - The driver creates a `SparkContext`, builds the DAG, and requests executors. - Executors run as independent pods with individually allocated CPU and memory. - The driver divides the DAG into stages and distributes tasks to executors. - Shuffle data is normally tied to executor-pod lifecycles unless an external shuffle service is configured. ## Advantages over YARN - **Containerized execution:** Docker images package application dependencies, improving reproducibility and CI/CD integration. - **Infrastructure independence:** Spark can use HDFS, S3, GCS, or other storage systems without requiring a Hadoop cluster. - **Simpler autoscaling:** Kubernetes can scale pods and integrate with cloud VM autoscalers. - **Unified platform:** Spark, Airflow, machine-learning workloads, and API servers can share a Kubernetes cluster. - **Governance and isolation:** Namespaces, `ResourceQuota`, and RBAC provide flexible team-level controls. - **Operational automation:** Helm, ArgoCD, GitOps, and rolling updates enable more automated application management. ## LINE Ads’ Kubernetes-Based System The platform is organized into four layers: - **Deployment layer** - GitHub Actions runs CI workflows based on repository events. - ArgoCD monitors desired and deployed states and supports easier rollback and synchronization. - **Compute layer** - Kubeflow’s Spark Operator deploys applications through the `SparkApplication` Kubernetes custom resource. - Apache YuniKorn schedules batch jobs and supports resource coordination and gang scheduling. - LogSender forwards pod logs to OpenSearch. - ClusterMonitoring sends Prometheus metrics to the company’s monitoring system. - **Storage layer** - Kafka provides high-throughput, low-latency storage for real-time advertising actions. - Hadoop remains available for large-scale, long-term analysis. - **Monitoring layer** - Kubernetes workers and Spark applications are monitored through exposed Prometheus metrics and centralized logging. The migration to Spark on Kubernetes is recommended for organizations whose Spark workloads are outgrowing tightly coupled Hadoop environments. It separates compute from storage, improves deployment flexibility, and allows data applications to be managed as cloud-native workloads.

Read original(opens in new tab)
aws2 min readCurated summary

Our First 2026 Heroes Cohort Is Here! | Amazon Web Services

AWS has announced its first 2026 Heroes cohort, recognizing Maurizio, Ray Goh, and Sheyla Leacock for combining technical expertise with community leadership. Their work spans cloud architecture, generative AI, machine learning, and cybersecurity, while emphasizing mentorship, education, and meaningful human connections. Together, they demonstrate how technology leaders can expand access to skills and strengthen communities globally. ## Maurizio – Pignola, Italy - CTO and organizer of the AWS User Group Basilicata. - Has spent more than a decade developing cloud communities and technology ecosystems in areas where they previously did not exist. - Founded an international technology conference in a small mountain village, connecting global experts with local developers. - Covers topics including cloud architecture, DevOps, and web scaling, alongside creative networking opportunities. - Mentors children, university students, and professionals transitioning into cloud careers. - Combines technical leadership with inclusive, cross-generational community building. ## Ray Goh – Singapore - AI and machine learning community leader involved in AWS programs since 2018. - Founded The Gen-C in 2024, offering public library workshops on generative AI, LLM fine-tuning, and AWS AI agents. - Has spoken at major AWS events and contributed to the AWS Machine Learning Blog. - Led DBS Bank’s AWS DeepRacer initiative, which trained more than 3,100 employees. - Trained over 1,300 ASEAN students in LLM techniques in 2025. - Supports skills-based programs teaching AI and machine learning to women, children, and young people. ## Sheyla Leacock – Panama City, Panama - IT security professional, mentor, technical writer, and international speaker. - Leads the AWS User Group in Panama and participates in AWS Community Days and regional meetups. - Has spoken at AWS Summits, AWS re:Invent PeerTalk sessions, and more than 20 international conferences. - Publishes educational content focused on AWS cloud computing and cybersecurity. - Works with universities as a guest lecturer to help develop future technology and security professionals. - Strengthens the cloud and cybersecurity ecosystem through education, knowledge sharing, and community leadership. The new cohort highlights the broader impact of community-driven technology leadership. Readers can visit the AWS Heroes webpage to learn more about the program or connect with a Hero.

Read original(opens in new tab)
gitlab2 min readCurated summary

10 AI prompts to speed your team’s software delivery

AI-assisted coding can accelerate code production without accelerating delivery, because review, security, documentation, and planning often become the new bottlenecks. The post recommends applying AI across the full software lifecycle, using targeted prompts to reduce routine work and let teams focus on architecture, risk, and business decisions. ## Code Review as an Accelerator - AI can review merge requests (MRs) for: - Logical errors, edge cases, and potential bugs. - API changes, altered return types, schema modifications, and configuration changes that may break consumers. - Catching these issues before human review reduces repeated review cycles and helps prevent deployment-time rollbacks. ## Shifting Security Left - Security scan analysis can use AI to: - Distinguish real vulnerabilities from false positives. - Explain risks and recommend remediation. - Prioritize findings by severity and exploitability. - AI-assisted code reviews can identify injection flaws, authorization problems, data exposure, insecure dependencies, and cryptographic weaknesses before an MR is created. - This reduces security-team backlogs and limits late-stage developer/security rework. ## Keeping Documentation Current - AI can generate release notes from merged MRs, organizing changes into features, fixes, performance improvements, breaking changes, and deprecations. - It can also identify which README files, API references, architecture diagrams, and onboarding guides need updates after code changes. - Automating these checks helps prevent documentation drift without creating a separate manual task. ## Breaking Down Complex Planning - An AI planning prompt can decompose an epic into implementable issues by considering: - Technical dependencies. - Appropriate issue sizes. - Acceptance criteria. - Implementation order. - The goal is to replace lengthy planning meetings with an initial AI-generated breakdown followed by team review. The practical recommendation is to treat AI as a team workflow accelerator, not merely a code generator. Applying focused prompts to review, security, documentation, and planning can help prevent increased coding speed from creating larger downstream bottlenecks.

Read original(opens in new tab)
gitlab2 min readCurated summary

AI can detect vulnerabilities, but who governs risk?

AI can increasingly detect vulnerabilities and suggest fixes, but detection alone does not make software secure. The post argues that enterprises also need governance, context, continuous assurance, and supply-chain oversight to determine which risks are acceptable and what can ship. GitLab presents its platform as the orchestration layer for enforcing these controls across AI-assisted development. ## Trust Requires Governance - AI analysis is not the same as accountability. - Humans must define acceptable risk, policies, guardrails, separation of duties, and audit requirements. - As autonomous agents gain more control over development, stronger governance becomes essential rather than optional. - Governance enables organizations to trust AI at scale without relying on unchecked autonomy. ## Context Matters Beyond Code Scanning - LLMs typically assess code in isolation, while enterprise platforms can evaluate its broader context. - Important factors include: - Who authored the change - The application’s business criticality - Its dependencies and infrastructure interactions - Whether vulnerable code is reachable in production - Whether the vulnerability is exploitable in the actual runtime environment - Context reduces noisy alerts and supports faster, more effective risk triage. ## Risk Changes Continuously - Dependencies, environments, and system interactions evolve after an initial scan. - A clean static scan does not guarantee that software remains safe at release time. - Organizations need continuous assurance embedded throughout development, testing, and deployment. - Detection identifies risk, while ongoing governance determines how that risk is managed. ## Governing AI-Generated Software - Modern software combines AI-generated code, open-source libraries, and third-party dependencies across many projects. - Governing this entire supply chain is more difficult than detecting flaws in individual code changes. - The post argues that developer-side AI tools alone are not designed to provide organization-wide enforcement and auditability. - GitLab Ultimate is positioned as a platform combining policy enforcement, security scanning, governance, and auditing within software delivery workflows. Organizations adopting AI most successfully will pair capable coding assistants with strong, continuous governance. The practical recommendation is to treat AI security as a platform and lifecycle-management problem—not merely a vulnerability-detection problem.

Read original(opens in new tab)
gitlab2 min readCurated summary

GitLab Duo Agent Platform with Claude accelerates development

GitLab Duo Agent Platform integrates external AI models such as Anthropic’s Claude and OpenAI’s Codex directly into GitLab workflows. Instead of operating as isolated coding assistants, these agents use project context and organizational standards to handle multi-step development tasks. The result is faster delivery, more consistent quality, and less manual work across the software development lifecycle. ## From an Idea to a Working Application - An agent can use an issue’s title and detailed requirements as the foundation for a complete application. - It analyzes project context and related assets, then generates: - Backend Java classes - Frontend HTML, CSS, and JavaScript - Business logic and UI components - Build configuration - The agent creates a merge request containing the implementation for developers to test and refine through natural-language interaction. ## Automated Code Review - Developers can mention the external agent in a merge request to request a review. - The review can cover: - Code strengths and critical issues - Medium- and low-priority improvements - Security risks - Testing gaps and code metrics - Recommendations and an approval status - This provides consistent review coverage while allowing senior developers to focus on architecture and complex decisions. ## Pipeline and Container Image Creation - When a project lacks CI/CD configuration, the agent can generate the required pipeline. - It creates a Dockerfile with a suitable base image for the project’s Java version. - The pipeline can: - Build the application - Build a Docker image - Push the image to GitLab’s container registry - The resulting workflow runs automatically through build, image creation, and deployment stages. ## Broader Impact on Development - External agents remain within GitLab, reducing context switching between development tools. - They can follow project-specific coding standards and understand broader repository context. - Teams can automate work from initial requirements through implementation, review, and deployment. - Developers spend less time on repetitive tasks while maintaining stronger consistency and quality. GitLab presents Duo Agent Platform as a way to turn external AI models into integrated development collaborators. Teams can use it to accelerate coding, automate reviews, and create deployment pipelines while keeping humans focused on validation, architecture, and innovation.

Read original(opens in new tab)
gitlab2 min readCurated summary

GitLab metrics and registry features help reduce CI/CD bottlenecks

GitLab’s two new beta features target common CI/CD bottlenecks without requiring additional third-party tools. CI/CD Job Performance Metrics provides job-level visibility into duration and failures, while Container Virtual Registry centralizes pulls from multiple registries through a cached GitLab endpoint. Together, they help platform teams identify pipeline problems faster and simplify container management. ## CI/CD Job Performance Metrics - Available in GitLab Premium and Ultimate. - Limited beta on GitLab.com; available on Self-Managed and Dedicated with ClickHouse configured. - Adds a job-focused panel to **Analyze > CI/CD analytics**. - Shows, for the previous 30 days by default: - Median (P50) and worst-case (P95) job duration - Failure rate - Job name and pipeline stage - Supports sorting, searching, and pagination to identify slow or unreliable jobs. - GitLab plans to add stage-level aggregation for build, test, and deploy bottlenecks. ## Container Virtual Registry - Available in GitLab Premium and Ultimate; API-ready in GitLab 18.9. - Provides one GitLab endpoint for pulling images from multiple upstream registries. - Supports registries such as Docker Hub, Harbor, Quay, and other sources using long-lived token authentication. - Uses pull-through caching to: - Reduce repeated downloads and bandwidth costs - Improve availability and reliability - Centralize authentication and registry configuration - Currently configured through the API, with UI management in development. - Cloud registries requiring IAM authentication, including Amazon ECR, Google Artifact Registry, and Azure Container Registry, may be supported later. ## Beta Access and Feedback - GitLab.com users can request access through their customer success manager or the feature’s feedback issue. - Self-managed users can enable the feature flag and configure the virtual registry through the API. - GitLab is seeking feedback to guide future improvements to both features. These betas are worth evaluating if your team needs better visibility into pipeline performance or manages images across several registries. The metrics feature can replace custom dashboards, while the virtual registry can reduce registry-related configuration and operational overhead.

Read original(opens in new tab)