techlist.io
KREN

Kubernetes

144 posts

kakao4 min readCurated summary

Experience Building and Operating a Personalized Airflow Testing Environment

Kakao’s data engineering team built AirZone to make Airflow DAG testing faster, easier, and safer across an ecosystem containing thousands of DAGs and multiple Hadoop clusters. Existing approaches required local setup, repeated Git synchronization, file copying, VPN access, or risky testing on production Airflow. AirZone instead creates an isolated, production-like Airflow environment for each pull request, managed through GitHub comments and Kubernetes automation. ## Limitations of Existing Testing Methods - **Local Airflow** - Requires configuring Airflow, Hadoop authentication, connections, and Docker locally. - Has a high initial setup cost and may differ from production. - **Development Airflow** - Requires committing and pushing every code change. - Git submodule updates and DAG parsing introduce long feedback delays. - **Test Airflow with SSH** - Allows files to be copied directly into a container. - Still requires copying files after every edit. - Access to production Hadoop requires connecting to a production VPN. - **Testing on production Airflow** - Heavy test DAGs consume shared scheduler, worker, and node resources. - A resource-intensive test can delay or interrupt unrelated projects. - Per-user isolation is therefore essential. ## AirZone Requirements - Provide an Airflow environment without requiring users to understand Kubernetes or Helm. - Allow code editing through a browser using Jupyter Notebook. - Execute DAGs against Hadoop and authentication mechanisms similar to production. - Create an independent environment for each pull request. - Prevent one user’s tests from affecting other workflows. ## PR-Based, Isolated Architecture - GitHub pull request comments serve as the user interface. - Users can create or delete an environment directly from a PR. - The resulting environment link is posted back to the PR. - Each PR receives a dedicated Kubernetes namespace based on the repository and PR number. - Airflow web server, scheduler, PostgreSQL, Jupyter, DAG volumes, and logs are isolated. - Multiple PRs can be tested simultaneously. - Cleanup is straightforward because the namespace defines the environment boundary. - A dedicated AirZone Helm chart packages the complete test environment. - Production-only components such as PGBouncer and external database connections are omitted where unnecessary. - Airflow, PostgreSQL, DAG storage, Jupyter, authentication, TLS, and logging are deployed together. ## Separating Requests from Deployment - `airzone-api` only validates requests: - Confirms that the PR exists and is open. - Checks branch information. - Prevents duplicate namespaces. - Kubernetes Jobs perform the long-running work: - Install the Helm release. - Run health checks. - Handle creation and deletion independently from the API process. - Job names include the operation and namespace, such as: - `create-airzone-{namespace}` - `delete-airzone-{namespace}` - Failed Jobs can be removed and recreated for retries. - Independent Job logs and status make deployment failures easier to diagnose. - A daily CronJob removes environments that remain after their PRs are closed. ## Building the Airflow Environment Each Helm deployment includes the components needed for a realistic test environment: - **Git integration:** Synchronizes the PR’s head repository and branch. - **DAG PVC:** Lets the scheduler and Jupyter use the same working directory. - **Airflow configuration:** Uses KubernetesExecutor and test-specific DAG scanning, logging, and Hadoop settings. - **Authentication:** Injects user and shared principals, keytabs, Jupyter tokens, and TLS certificates. - **Infrastructure placement:** Selects suitable node groups and a storage class in the same region. - **Centralized logging:** Connects Airflow logs to Elasticsearch and Kibana. - **Hadoop execution:** Existing infrastructure runs Hadoop tasks in dedicated pods using custom Hadoop images, Kerberos initialization, Spark, and Hive. ## Notifications and Security - KakaoWork sends: - An initial notification when a request is received. - A completion notification after deployment. - Operational error alerts. - Sensitive information, including Jupyter and Kubernetes namespace tokens, is not posted in public PR comments. - Tokens are delivered through KakaoWork instead, keeping authentication data separate from the broader PR audience. AirZone’s main recommendation is to make testing a disposable, reproducible environment tied to the pull request itself. By combining per-PR Kubernetes namespaces, Helm-based deployment, asynchronous Jobs, production-like Hadoop access, and automatic cleanup, teams can test DAGs quickly without burdening shared Airflow or production resources.

Read original(opens in new tab)
KubernetesPostgreSQLApache SparkHelm+3
gitlab2 min readCurated summary

GitLab Secrets Manager adds ESO, Terraform, API support

GitLab Secrets Manager expands beyond CI/CD by supporting Kubernetes, Terraform/OpenTofu, CLI tools, and external automation. Built on OpenBao and compatible with Vault APIs, it provides one centrally managed secret store with consistent access controls and auditing. The result is fewer duplicated credential stores and safer secret retrieval across the software delivery lifecycle. ## Kubernetes with External Secrets Operator - ESO uses its Vault provider to retrieve secrets from GitLab Secrets Manager. - A Kubernetes workload uses a short-lived GitLab-minted JWT to authenticate with OpenBao. - A `SecretStore` configures: - The Vault-compatible server and KV v2 mount - The GitLab organization, group, and project namespace - JWT authentication and the Kubernetes secret containing the token - An `ExternalSecret` maps remote secrets to a Kubernetes `Secret`. - ESO refreshes values according to `refreshInterval`, allowing rotated credentials to reach workloads without redeployment. - `remoteRef.key`, `property`, and `secretKey` define the source path, field, and destination key. ## Terraform and OpenTofu Integration - Terraform can retrieve secrets at plan or apply time instead of storing them in `.tfvars` files or CI/CD variables. - A script obtains a minted JWT and connection metadata through Terraform’s `external` data source. - The Vault provider uses that JWT to authenticate against GitLab Secrets Manager. - The `vault_kv_secret_v2` data source reads the required secret. - Outputs containing secrets should be marked `sensitive`, though downstream Terraform state handling still requires care. ## OpenBao and Vault CLI - Existing Vault-compatible scripts can access GitLab Secrets Manager without using the API directly. - Users configure `VAULT_ADDR` and `VAULT_NAMESPACE`. - A minted JWT is exchanged for an OpenBao client token through the configured JWT authentication path. - The `vault kv get` command then retrieves secrets from the KV mount. ## Secrets Manager API - The API supports automation outside GitLab CI/CD, Kubernetes, and Terraform. - A service account requests an access token through GitLab’s project API. - The response supplies the Vault server, namespace, mount, secrets path, JWT authentication path, and role. - External systems can use this information to authenticate and fetch secrets without hardcoded credentials or separate variable files. GitLab Secrets Manager is most useful when multiple deployment tools need the same credentials. Centralizing secrets in the OpenBao-backed store, using short-lived JWT authentication, and integrating through ESO, Terraform, CLI, or the API can reduce duplication and improve rotation and auditing.

Read original(opens in new tab)
KubernetesTerraformJwtSecrets Management+4
aws4 min readCurated summary

AWS Weekly Roundup: AWS Builder Center at 1 year, Network Scanning in Security Hub, Loom for AWS, and more (July 13, 2026) | Amazon Web Services

AWS’s weekly roundup highlights the first anniversary of AWS Builder Center and its expansion into a broader learning and experimentation ecosystem. It also introduces major security, AI, container, database, and developer-tool updates, including internet reachability scanning, Azure support in Security Hub, Hugging Face integration with SageMaker, lower GPU management fees, and Aurora DSQL change data capture. The overall direction is toward more integrated, governed, and accessible cloud development workflows. ## AWS Builder Center Turns One - Launched on July 9, 2025, Builder Center has expanded from a community hub into an ecosystem with: - Sandbox environments - Workshops and the Builders’ Library - Community Spaces, profiles, badges, and streaks - AWS regional capability listings - Saved items, article series, availability notifications, and GitHub/Amazon sign-in - In its first year: - 5,548 authors published 6,448 articles. - Articles received more than 10.4 million page views. - Builders earned 99,226 badges. - The community submitted 565 feature wishes, with 10 already delivered. - The most popular articles covered MCP and Strands Agents, Linux migration with Kiro, and multimodal AI for neurological screening. ## Free AWS Sandbox Environments - Builder Center now offers free, pre-provisioned AWS accounts for workshop exercises. - Each sandbox: - Remains active for eight hours. - Automatically removes the account and resources afterward. - Requires no personal AWS account, credit card, or manual cleanup. - Users can have one active sandbox and request one sandbox per week. ## Security Hub Adds Network and Azure Coverage - Security Hub Network Scanning probes AWS and Azure environments from the public internet. - It identifies: - Public IP addresses, virtual machines, and load balancers. - Reachable ports. - Services operating behind those ports. - Each reachable port produces a finding, which Security Hub Exposures correlates with other configuration and security findings. - Existing customers can enable the feature by account, Region, or organization policy; it is enabled by default for new customers. - Network Scanning is included with Security Hub Essentials at no extra cost. - Security Hub also now evaluates Azure VMs, container images, Function Apps, and identities for vulnerabilities, misconfigurations, and internet exposure. ## SageMaker Studio and Hugging Face Integration - Users can deploy or customize supported Hugging Face models in SageMaker Studio with one click. - New customers receive automatically configured Studio environments for: - Fine-tuning and reinforcement learning with custom reward functions. - Model evaluation. - Deployment to SageMaker or Bedrock endpoints. - Verified customers receive default GPU access to G5, G6, and G4dn instances, while quota usage is visible in Studio. ## Lower GPU Management Fees - Starting July 1, 2026, EKS Auto Mode and ECS Managed Instances reduced accelerated-instance management fees: - G-series: 35% reduction. - P-series and Trainium: 60% reduction. - Existing customers receive the reductions automatically. - EKS adds accelerator-aware node repair and parallel image pulling. - ECS provides GPU metrics through CloudWatch Container Insights and monitors GPU hardware health. ## Aurora DSQL Change Data Capture - Aurora DSQL CDC is generally available and streams insert, update, and delete events to Kinesis Data Streams. - The events can synchronize microservices, invoke Lambda functions, or feed S3, Redshift, and OpenSearch through Firehose. - CDC is designed to avoid database performance impact and requires no infrastructure management. ## Tools for Governed AI Agents - Loom for AWS is an open-source platform for deploying and managing Strands Agents on Bedrock AgentCore Runtime. - It includes: - Identity-provider integration and RBAC/ABAC. - Lifecycle management for agents, memory, MCP servers, and agent-to-agent connections. - Automated tagging for cost attribution. - Agent Registry integration and human approval for sensitive actions. - The Claude apps gateway provides centralized access, policy, and spending controls for Claude Code and Claude Desktop. - It supports OIDC identity providers, per-user and per-group spending caps, managed request settings, and routing through Amazon Bedrock or Claude Platform on AWS. - AWS MCP Server now supports browser-based OAuth using AWS Console or CLI credentials, including IAM federation and IAM Identity Center, with short-lived tokens and automatic refresh. AWS’s updates favor managed services that reduce operational work while strengthening security and governance. Teams should evaluate Builder Center sandboxes for training, enable Security Hub’s broader scanning where appropriate, and consider the new AI-agent controls and managed GPU options for production workloads.

Read original(opens in new tab)
KubernetesModel Context ProtocolPostgreSQLAWS+4
gitlab3 min readCurated summary

Green DevOps: Why carbon measurement belongs in your CI/CD pipeline

CI/CD pipelines consume energy and generate emissions that standard pipeline metrics do not reveal. The post argues that measuring this impact is the first step toward reducing it, recommending Eco CI for job-level measurements and Carmen for infrastructure and application-level analysis. Both tools integrate with GitLab pipelines with minimal effort and can improve sustainability, performance, and cloud costs. ## Why CI/CD Carbon Measurement Matters - Modern teams run hundreds of jobs daily, including increasingly resource-intensive AI-assisted testing and automation. - Each job has an energy and carbon cost that is usually absent from pipeline logs and architecture diagrams. - Green DevOps involves measuring emissions per pipeline, service, and Kubernetes workload to guide engineering decisions. ## Measuring Emissions at Different Layers ### Pipeline-level measurement with Eco CI - Eco CI measures energy consumption and carbon emissions for individual CI/CD jobs. - It runs as lightweight Bash scripts without additional servers or databases. - Teams can identify expensive jobs, track emissions over time, and add a carbon badge to a project README. - Because it works within existing pipelines, it is a practical starting point. ### Infrastructure and application measurement with Carmen - Carmen, the Carbon Measurement Engine, uses the Green Software Foundation Impact Framework. - It measures emissions from virtual machines, Kubernetes pods, and application workloads. - Reports are provided per component in CSV format and distinguish: - Operational carbon from energy consumption - Embodied carbon from hardware manufacturing and disposal - Outputs such as `EnergykWh` and `TotalCarbonGramsCO2eq` can be imported into Grafana, FinOps dashboards, or custom tools. - Carmen helps teams compare emissions across services, such as an API gateway versus a data-processing layer. ## Adding Carbon Reports to GitLab - Both tools can be integrated through `.gitlab-ci.yml`. - A Carmen job can: - Use a Python 3.12 image - Install Node.js, npm, Git, and the Impact Framework toolchain - Clone and install the Carmen project - Run `carbon-daemon` - Store generated reports as GitLab artifacts - Carmen can run as a separate, non-blocking job, keeping carbon measurement outside the critical delivery path. ## Turning Visibility into Engineering Improvements - Eco CI may reveal that integration tests produce unusually high emissions because dependencies are reinstalled on every run. - Adding dependency caching can reduce runtime, emissions, and CI costs without architectural changes. - Carmen can expose idle infrastructure, such as deprecated services still running in a staging Kubernetes cluster. - Decommissioning unused workloads eliminates both their operational and embodied carbon costs. ## Business and Regulatory Benefits - Emissions data creates a baseline for future reporting requirements, even when a company’s absolute footprint is relatively small. - Carbon-efficient systems are often faster and cheaper because they use fewer compute resources. - Caching, right-sizing runners, and removing idle services support both sustainability and FinOps goals. - Regulations such as the EU’s CSRD, along with enterprise procurement requirements, are increasing pressure for supply-chain and cloud-emissions reporting. Teams can start with Eco CI on one GitLab pipeline, then add Carmen for infrastructure-level visibility. Measuring carbon alongside runtime and cost makes sustainability a practical part of everyday engineering rather than a separate initiative.

Read original(opens in new tab)
KubernetesCI/CDGitlabDevOps+4
aws3 min readCurated summary

AWS Weekly Roundup: Claude Sonnet 5 on AWS, Amazon WorkSpaces for AI agents, AWS service availability updates, and more (July 6, 2026) | Amazon Web Services

The July 6, 2026 AWS roundup highlights new infrastructure, AI, observability, and developer productivity capabilities. Major announcements include Claude Sonnet 5 on AWS, generally available WorkSpaces for AI agents, faster SageMaker inference scaling, and log-query alarms in CloudWatch. It also details AWS service lifecycle changes and upcoming community events. ## Featured AWS Launches - **Claude Sonnet 5 on AWS** - Anthropic’s latest Sonnet model is available for coding, agentic workflows, and professional tasks. - It can navigate large codebases, use tools accurately, and preserve state across long-running tasks. - **Amazon WorkSpaces for AI agents** - AI agents can securely operate desktop applications in managed WorkSpaces environments. - Organizations can use existing applications without modernization or custom integrations. - **OpenSearch log analytics optimization** - A new engine provides up to four times better price-performance in internal benchmarks. - It combines log aggregation with OpenSearch’s full-text search capabilities. - **Faster SageMaker AI inference scaling** - Container image caching can reduce generative AI scale-out time by up to half. - SageMaker supports up to two times faster end-to-end scaling during demand increases. - **CloudWatch alarms from log queries** - Users can create alarms directly from log query results and define thresholds in one workflow. - This removes the need to create metric filters or custom metrics first. ## Additional Infrastructure and Developer Updates - **EC2 C9g and C9gd instances** - Powered by AWS Graviton5 processors. - Offer up to 25% more compute performance than Graviton4 instances, five times more cache, faster memory, and optional local NVMe storage. - **CloudFormation Express mode** - Provides deployment confirmation within seconds. - Helps developers and AI agents iterate more quickly, at no additional cost in commercial Regions. - **Amazon EKS version rollbacks** - Kubernetes upgrades can be reversed within seven days. - Rollbacks avoid rebuilding clusters and reduce the risk of failed upgrades. - **ACME support in AWS Certificate Manager** - Automates issuance and renewal of public TLS certificates using the standard ACME protocol. ## AWS Service Availability Changes AWS updated its lifecycle guidance on June 30, 2026, including alternatives and migration support. - **Moving to maintenance for new customers from July 30** - Amazon Bedrock Agents becomes Bedrock Agents Classic. - Amazon Cognito Sync, Amazon Kendra, Amazon Q Business, Simple AD, and several other services and features will no longer accept new customers. - A number of SageMaker AI features are affected, including Clarify, Debugger, Ground Truth, Model Monitor, and Studio Lab. - AWS IoT Device Defender Detect changes on August 31, 2026. - **Entering sunset** - Amazon WorkSpaces PCoIP and Pool. - AWS Managed Services Advanced. - AWS re:Post Private. - SageMaker AI Profiler. - **End of support** - Amazon Chime SDK Carrier Voice Focus. - SageMaker AI Ground Truth Plus. ## Upcoming AWS Events - AWS Summits will take place throughout the second half of 2026. - AWS Community Day Belo Horizonte is scheduled for August 22. - The AWS Builder Center offers community discussions, technical content, and information about upcoming virtual and in-person events. Organizations using affected AWS services should review the relevant lifecycle documentation and contact AWS Support to plan migrations before availability or support deadlines.

Read original(opens in new tab)
KubernetesAWSAmazon SagemakerAmazon Eks+4
aws2 min readCurated summary

Upgrade Amazon EKS clusters with confidence using Kubernetes version rollbacks | Amazon Web Services

Amazon EKS now supports rolling back Kubernetes minor-version upgrades within seven days, providing a safety net that open-source Kubernetes traditionally lacks. The rollback returns a cluster to its previously validated production version rather than placing it in an emulated transitional state. This can help organizations upgrade more confidently, reduce lengthy upgrade cycles, and avoid remaining on outdated versions. ## Kubernetes Version Rollbacks - Rollbacks support one minor version at a time, matching EKS upgrade behavior. - Administrators can revert an upgrade such as Kubernetes 1.35 to 1.34 within the seven-day rollback window. - EKS uses cluster insights to check rollback readiness, including: - Node version compatibility - Add-on dependencies - Other potential upgrade issues - The `--force` option bypasses these checks when administrators need to proceed quickly. - The capability applies to both self-managed-node clusters and clusters using AWS-managed infrastructure. ## EKS Auto Mode Rollbacks - EKS Auto Mode must roll back both the control plane and managed nodes. - Node rollback operations honor pod disruption budgets, so completion time depends on workload configuration. - A cancel API allows administrators to stop an in-progress node rollback. - Disruption budgets are never bypassed automatically; users can modify or remove them if they need to accelerate the process. ## Rollback Experience and Availability - Rollbacks can be initiated from the EKS console after selecting an eligible cluster. - The console displays the remaining rollback window and relevant readiness insights. - In the example, the control plane rollback took about 20 minutes, while Auto Mode nodes rolled back according to their disruption budgets. - Rollbacks are available at no additional cost in all commercial AWS Regions where EKS operates. - Control plane rollbacks support all EKS clusters; node rollbacks are available for EKS Auto Mode. - Supported Kubernetes versions include those under standard or extended EKS support. Administrators can use the EKS console or documentation to begin using rollbacks, making incremental Kubernetes upgrades safer and easier to recover from.

Read original(opens in new tab)
KubernetesAWSAmazon EksEks Auto Mode+3
discord2 min readCurated summary

Cost Attribution in Discord’s API

Discord’s API runs from a shared Python codebase with more than 1,700 endpoints and 700 background tasks across hundreds of Kubernetes deployments. While existing observability tracks performance and reliability, Discord lacked a way to understand hosting costs by product feature or endpoint. Because deployments share code and workers handle multiple features concurrently, the solution was to extend application profiling to allocate deployment costs according to the time spent serving each feature. ## A Large, Continuously Deployed API - Discord operates a unified Python codebase containing: - Over 1,700 API endpoints - Around 700 background tasks - Engineers deploy changes daily to several hundred Kubernetes deployments. - Phased rollouts and instrumentation help monitor: - Latency - Throughput - Error rates - These metrics make it possible to detect regressions affecting users or infrastructure. ## The Missing Cost Dimension - Discord wanted to determine how hosting costs were distributed across product features. - Example questions included: - How much does it cost to send and receive messages? - What does it cost to start a stream or send a Nitro gift? - How do feature costs change over time? - Did a recent code change materially affect a team’s hosting spend? - The goal was to measure costs at both: - Individual endpoint level - Broader feature level, such as chat ## Why Kubernetes Deployment Costs Were Insufficient - Cloud providers can generally report costs by Kubernetes deployment. - However, Discord’s deployments do not map cleanly to product features: - The same codebase runs across all deployments. - Each deployment handles a particular subset of HTTP traffic or background tasks. - Splitting deployments further would make the system impractical to operate. - Discord therefore needed cost attribution without changing its deployment topology. ## Allocating Costs Through Profiling - API worker processes handle multiple tasks concurrently. - A single worker may simultaneously perform work for many different features. - Existing traffic isolation was not detailed enough for feature-level cost analysis. - Discord’s approach was to allocate a deployment’s cost based on the amount of time spent executing code associated with each feature. - By extending its application profiling tools, Discord could track this execution time and use it to estimate feature and endpoint hosting costs. In practice, the profiling-based approach provides a way to analyze infrastructure spending within shared deployments, without requiring separate services or Kubernetes environments for every product feature.

Read original(opens in new tab)
KubernetesObservabilityPythonAPI Design+2
google3 min readCurated summary

Accelerating Gemini Nano models on Pixel with frozen Multi-Token Prediction

Google introduces a way to add Multi-Token Prediction (MTP) to already-deployed, frozen Gemini Nano models, accelerating on-device generation without a separate drafting model. The approach attaches a lightweight Transformer head to the existing model, reuses its hidden states and KV cache, and preserves identical final outputs through verification. On Pixel 9 and 10 devices, it delivers faster generation, lower memory use, and reduced energy consumption for features such as Notification Summaries and Proofread. ## The Mobile Inference Bottleneck - Autoregressive models generate one token at a time, creating latency and underusing mobile hardware. - Phones face strict RAM and energy constraints that make conventional acceleration techniques difficult. - A standalone speculative-decoding drafter consumes additional memory and must independently process the prompt. ## A “Late Exit” MTP Strategy - Speculative decoding uses: - A small drafter to propose several tokens. - The large model to verify those tokens in parallel. - MTP replaces the separate drafter with a lightweight Transformer head attached near the end of the main model. - The head uses the backbone’s high-dimensional activations to predict future tokens, benefiting from semantic context already computed by the larger model. ## Retrofitting a Frozen Backbone - Google freezes the fully trained Gemini Nano v3 weights and trains only the attached MTP head. - This avoids retraining or fine-tuning the production foundation model. - Incorrect draft tokens are discarded during verification, so the final output remains bit-for-bit identical to the original model. - The method therefore improves efficiency without changing the model’s capabilities or safety alignment. ## Zero-Copy Memory Architecture - The MTP head cross-attends directly to the backbone’s existing KV cache instead of maintaining a duplicate cache. - This eliminates separate prompt-prefill work for the drafter. - It also removes redundant embedding tables, attention variants, and application-specific tuning parameters. - Compared with a standalone drafter, the design saves up to 130 MB per instance. ## Accuracy and Speed Improvements - Access to the backbone’s richer internal representations makes MTP predictions more accurate than those of similarly sized standalone drafters. - Instruction-following tasks such as summarization and rewriting show especially strong gains. - For predictable formats such as smart replies, token acceptance improved by up to 55%. - Pixel 9 experiments showed speedups of 50% or more depending on the task. ## Production Impact - The updated inference stack coordinates drafting and verification on Pixel 9 and 10 devices. - In workloads including AI Notification Summaries and Proofread, MTP predicts nearly two additional tokens per inference pass on average. - Fewer verification cycles reduce processor wake-ups, improving latency and battery efficiency. MTP provides a practical way to accelerate existing on-device models without maintaining a separate drafter for every application. Reusing the frozen backbone’s computations and memory state makes it particularly well suited to mobile deployments where RAM, energy, and backward compatibility are critical.

Read original(opens in new tab)
KubernetesTransformerEdge ComputingSpeculative Decoding+3
line4 min readCurated summary

Flava DBaaS Deep Dive: From Architecture to Migration and Beyond

LY Corporation’s Flava DBaaS is designed to unify the former Verda and YNW cloud platforms on a Kubernetes-based architecture. Its operator pattern separates database business logic from IaaS management, while API servers, managers, and agents divide responsibilities within each DBMS service. The platform expands database support, improves scalability, security, and usability, and treats migration from legacy platforms as part of the DBaaS responsibility. ## Kubernetes Operator-Based Design - Flava DBaaS uses the Kubernetes operator pattern. - Users declare the desired database state through custom resources rather than issuing procedural commands. - Controllers continuously reconcile the actual state with the declared specification. - This approach: - Simplifies troubleshooting through resource status and controller logs. - Handles large database infrastructures efficiently through event-driven processing. - Reuses Kubernetes capabilities for CI/CD and access control. ## Infrastructure Operator Layer - DBaaS must manage IaaS resources such as: - Virtual machines - Storage - Domains and networking - Flava isolates this infrastructure logic in a separate infrastructure operator. - IaaS resources are exposed as Kubernetes custom resources, allowing DBaaS to create infrastructure declaratively without directly calling IaaS APIs. - The resulting layers are: - **DBaaS:** Database-specific business logic - **Infrastructure operator:** Abstraction of IaaS as Kubernetes resources - **IaaS:** Compute, network, and storage services - This separation allows multiple DBMS products to use infrastructure consistently while their developers focus on database operations. ## Custom Resources and DBaaS Components - Each database cluster is represented by a Kubernetes custom resource containing settings such as: - DBMS version - VM size - Storage type and capacity - Replication configuration - These resources are stored in Kubernetes etcd and managed through the Kubernetes API. - Each DBMS implementation consists of three components: - **API server:** Provides REST APIs for creating, modifying, and deleting database resources. Flava UI and IaC tools use these APIs. - **Manager:** Watches resource changes and reconciles the database cluster toward the declared state. - **Agent:** Runs on database VMs and executes local operating-system and database commands. - For example, creating a MySQL cluster causes the API server to create a MySQL custom resource, the manager to provision the required VMs through the infrastructure operator, and the agent to configure replication and database processes inside those VMs. ## Improvements in Flava DBaaS - Flava preserves core DBaaS capabilities such as provisioning, high availability, backup and recovery, scalability, and monitoring. - It combines the DBMS offerings of Verda and YNW, expanding the range of supported database systems. ### Flexible Storage and Scaling - Storage can be configured in 100 GiB increments. - Block-storage-based databases can use up to 5 TiB of storage. - Unlike the legacy platforms, storage is no longer tightly limited by a VM’s local disk capacity. - Custom instance types and separate block storage reduce the need to consider alternatives such as sharding for larger databases. - The 5 TiB limit was selected to cover most analyzed use cases while reducing infrastructure fragmentation. ### Consistent User Experience - All Flava DBaaS products share a common architecture and UI. - Skills learned while changing MySQL server specifications or configuring Cassandra alerts can be applied to other DBMS products. - Users do not need to learn separate operational workflows for each database system. ### Security and Convenience - TDE and TLS are provided as platform-level security features. - Additional features include: - **Custom DB Role:** Reusable database users with configurable permissions. - **Database Parameter Group:** Reusable groups of database configuration parameters. - **Restore backup:** Creation of a new cluster from a selected backup for disaster recovery or realistic performance testing. - Features not yet available for every DBaaS product are planned for broader support. - These improvements reportedly resulted in high internal user-satisfaction scores. ## Migration Responsibilities - A new DBaaS platform is expected to provide migration paths from existing platforms, not merely offer new database clusters. - For migrations between the same DBMS type, the article identifies three general approaches. ### Dump and Restore - Data is backed up from the source database and restored into the destination. - It is the simplest method. - To guarantee consistency, the application generally must be stopped during the migration. ### Replication-Based Migration - The source database is continuously replicated to the destination. - Once replication is caught up, the destination is promoted through failover. - The source database can then be removed. - Data consistency depends on the DBMS’s replication mechanism. - A short application interruption may still occur during primary-node failover. The overall recommendation is to use Flava’s layered, declarative architecture to standardize database operations while continuing to provide practical migration mechanisms from Verda and YNW.

Read original(opens in new tab)
KubernetesInfrastructure As CodeMysqlDatabase Migration+4
netflix3 min readCurated summary

How Netflix Simplified Batch Compute with Kueue

Netflix replaced much of its custom Compute Managed Batch (CMB) queuing and scheduling logic with Kubernetes-native Kueue. The migration preserved the existing user experience while enabling features such as preemption, fair sharing, all-or-nothing scheduling, and topology-aware placement. Kueue now manages millions of batch workloads across Netflix’s Titus-based infrastructure. ## CMB and Titus Architecture - CMB manages workloads that run to completion using: - Hierarchical tenants - Priority-based ordering - Per-tenant capacity management - Workloads ultimately run on Titus, Netflix’s container platform. - Titus provides federation across multiple Kubernetes cells and shared capacity reservations, allowing CMB to interact with a unified endpoint. - CMB tenants are either: - **Internal tenants**, which organize child tenants but do not accept jobs - **Leaf tenants**, which accept jobs through associated queues - Capacity includes: - **Reserved capacity**, providing predictable resources within a tenant hierarchy - **Shared capacity**, a global pool that tenants can burst into - CMB enforced fair sharing only at admission time because it lacked preemption; admitted jobs ran to completion even when demand changed. ## Why Netflix Chose Kueue - CMB was developed before many Kubernetes batch features became available in open source. - Kueue provided capabilities Netflix had previously built or wanted to build, including: - Fair sharing - Hierarchical tenancy - Capacity management - Priority queues - Preemption - Unlike schedulers such as YuniKorn and Volcano, Kueue works with the existing Kubernetes scheduler rather than replacing it. - This allowed Netflix to retain Titus scheduling profiles and avoid inefficient job placement. - Kueue also supports: - Multi-tenant quotas across heterogeneous hardware - Native Kubernetes objects such as `Pod` and `Job` - Higher-level workloads such as `RayJob` and `RayCluster` - All-or-nothing admission and topology-aware scheduling ## Migrating CMB Workloads - The migration, called **Netflix Batch**, was designed to: - Require no changes from CMB users - Avoid regressions in launch rates and maximum throughput - Move queuing and scheduling responsibilities to Kueue - Kueue runs in enabled Titus cells, while a custom router and Titus federation direct workloads to the appropriate cell. - Tenant enrollment was exposed as a simple operator action in Netflix’s UI, making rollout and rollback straightforward. - Internally, the migration mapped: - CMB internal tenants to Kueue **Cohorts** - Leaf tenants to **ClusterQueues** and **LocalQueues** - Capacity configurations to Kueue **resource flavors** and **nominal quotas** ## Lessons from the Rollout - Maintaining API compatibility reduced customer disruption and allowed Netflix to replace backend components incrementally. - Migrating the largest and most complex customer early exposed problems sooner and increased confidence in the broader rollout. - The production migration took approximately four weeks. - Kueue required substantially higher QPS, burst, and `groupKindConcurrency` settings than its defaults. - Netflix validated these settings early through load tests in an environment modeled on Titus. ## Kueue in Production - Kueue is fully deployed at Netflix and manages millions of batch workloads. - Netflix is extending its use to additional Titus batch workloads. - Fair sharing and preemption are being expanded to improve utilization of reserved capacity. - Netflix’s experience is also informing other internal Kubernetes-native systems, including training infrastructure. Netflix’s migration demonstrates that a batch platform can adopt Kubernetes-native scheduling incrementally without forcing users to change APIs or abandoning existing placement infrastructure. For organizations with mature custom systems, preserving the external contract while delegating queueing and admission to Kueue offers a lower-risk path to modern features and simpler long-term operations.

Read original(opens in new tab)
KubernetesContainer OrchestrationMulti TenancyKueue+4
line4 min readCurated summary

From Automation to AI with Infrastructure as Code (IaC): Adopting OpenTofu and ChatOps

LY Corporation’s LINE Plus SRE team migrated Verda cloud infrastructure and IMON monitoring resources from scattered, manual management into an Infrastructure as Code (IaC) and GitOps workflow. Using OpenTofu and Terragrunt, they now manage roughly 1,500 resources across seven services through pull requests, CI/CD, and daily drift detection. The migration required careful automation for importing existing resources, normalizing state, and handling provider limitations and resource dependencies. ## Why IaC Was Needed - Teams previously managed infrastructure through different methods: - Verda’s web dashboard - Scripts - Wiki-based procedures - Personal documents and GitHub repositories - As the number of services and resources grew, this caused: - Inconsistent management practices - Difficult-to-track configuration changes - Greater risk of manual errors - Poor reproducibility and reviewability - The team adopted GitOps so that: - Desired infrastructure state is declared in Git. - All changes go through pull requests. - Infrastructure history is versioned and auditable. - CI/CD applies approved changes automatically. - Their goal was to manage infrastructure with the same engineering standards as application code: reviewable, version-controlled, and reproducible. ## Choosing OpenTofu and Terragrunt - OpenTofu was selected as an open-source Terraform fork. - It retains: - Terraform’s HCL syntax - Provider compatibility - Familiar module and configuration patterns - The team created reusable modules for: - Virtual machines - Load balancers - Monitoring alerts - Modules were versioned so updates could be adopted explicitly rather than affecting every environment immediately. - Terragrunt was added to reduce repetition in environment configuration. - Shared settings are defined once in a parent `root.hcl`. - Individual environments contain only their differing inputs. - OpenTofu reduces duplication in resource definitions, while Terragrunt reduces duplication in environment and backend configuration. ## Planning the Migration - The most difficult part of introducing IaC into an existing environment was importing resources that were already running. - Manual import was considered impractical for hundreds of VMs, load balancers, and DNS records because it would be slow and error-prone. - The migration was split into two phases: - **Phase one:** Automate imports, select one service for a pilot, and establish the complete OpenTofu/Terragrunt pipeline. - **Phase two:** Reuse the validated modules and import scripts to roll the approach out to the remaining services. ## Designing the Import Process - Import scripts were designed to: - Query existing resources - Decide which resources should be managed by IaC - Convert resource data into the desired code structure - Generate Terragrunt configuration - Connect resources to OpenTofu state - Run `plan` to verify that no unintended changes would occur - A key requirement was keeping three representations aligned: - Configuration code - OpenTofu state - Actual cloud resources - Normalization was added because equivalent values could be represented differently—for example, network or image IDs—causing OpenTofu to report misleading differences after import. ## Resource-Specific Import Strategies - Resources could not all be imported using the same procedure. - Different resource types have different identifiers, dependencies, and ownership models: - **VMs:** Imported individually, while distinguishing manually created instances from Kubernetes-managed instances. - **Load balancers:** Imported together with related listeners and pools. - **DNS:** Imported while preserving zone and record relationships. - **Kubernetes:** Structured around clusters and node pools. - **IMON:** Imported according to its hierarchy of teams, alert groups, alert rules, and monitors. - Each resource followed the same broad five-step process, but its implementation was adapted to the resource’s characteristics. ## Problems Discovered During Migration ### Kubernetes-Managed VMs - OpenStack contained both manually created VMs and VMs automatically created by Kubernetes. - Importing Kubernetes-managed VMs into IaC could cause conflicts between OpenTofu and Kubernetes. - The scripts excluded these VMs using naming patterns and metadata. ### IMON’s Hierarchical Structure - IMON alerts are organized as: `Team → Alert Group → Alert Rule → Monitor` - A flat import would lose these relationships. - The team mirrored the hierarchy in the directory structure so ownership and relationships were visible from the file layout. ### Provider and Regional Identifier Issues - The actual cloud platform allowed both hyphens and underscores in load balancer names, but the provider validation logic rejected underscores. - The team fixed this by modifying the provider’s validation logic and contributing the change upstream. - Resource UUIDs such as `flavor_id`, `image_id`, and `network_id` differed by region. - This produced unnecessary changes in `plan`. - The modules added regional mapping logic, allowing users to specify readable names while resolving them to region-specific IDs. The migration demonstrates that successful IaC adoption requires more than writing configuration files: existing infrastructure must be filtered, normalized, modeled according to its dependencies, and validated against real provider behavior. OpenTofu and Terragrunt provided the foundation for scalable GitOps management, while custom import automation and provider improvements made the transition safe for production resources.

Read original(opens in new tab)
KubernetesCI/CDInfrastructure As CodeGitops+3
toss5 min readCurated summary

Spark Connect on Kubernetes #1: Building a Robust Spark Connect

Toss Securities operates Spark Connect as a production service on Kubernetes so analysts and engineers can use Spark without complex setup. Spark Connect replaces per-application Drivers with long-running servers, making clients lighter and sessions faster, but it also introduces shared-failure and resource-contention problems. The post argues that production reliability requires both reducing server-wide failure triggers and distributing sessions across multiple replicas. ## How Classic Spark Works - Spark consists of: - A **Driver**, which plans jobs, schedules tasks, and collects results. - **Executors**, which perform the distributed computations. - In Classic Spark: - **Client mode** runs the Driver inside the client process. - **Cluster mode** launches the Driver in the cluster for each submitted application. - Both modes assume that one application has one Driver and one workload. - Clients also need Spark libraries, JVM support, and configuration. ## What Spark Connect Changes - Spark Connect turns the Driver into a pre-started, long-running server. - Clients send unresolved logical plans encoded with Protocol Buffers over gRPC. - The server handles analysis, optimization, scheduling, and execution. - Results are streamed back using Arrow. - This resembles a database accessed through JDBC. ### Benefits - **Thin clients:** Clients do not need the full Spark runtime or JVM. - **Language and platform independence:** Notebooks, BI tools, SQL clients, and different programming languages can use the same server. - **Fast session creation:** Sessions connect to an already-running server. - **Better client-failure tolerance:** A disconnected notebook does not necessarily terminate server-side work. ## Problems Created by Shared Long-Running Servers Spark’s internal design often assumes “one application equals one workload.” Sharing one application across many users breaks that assumption. ### A Shared Driver Becomes a Single Point of Failure - Multiple sessions share one `SparkContext` and Driver JVM. - A Driver failure terminates all sessions, jobs, and caches attached to it. - Spark’s global `spark.executor.maxNumFailures` counter can shut down the entire application after enough executor failures. - Because all sessions contribute to the same counter, one user’s unstable or memory-intensive query can terminate unrelated users’ workloads. - The counter is global, persists over time, and is separate from per-query task-level fault tolerance such as `spark.task.maxFailures`. ### Resource Contention and Scheduling Limits - `newSession()` isolates SQL state and namespaces, but not CPU, memory, or executors. - Heavy workloads can occupy all task slots and delay smaller queries. - FIFO scheduling favors earlier jobs, and Spark does not preempt tasks already using slots. - Fair Scheduler pools can influence task-slot ordering, but cannot provide true CPU or memory isolation. - Spark Connect does not automatically propagate `spark.scheduler.pool` to the server-side execution thread, causing queries to fall into the default pool unless the server explicitly assigns pools. - Actual resource isolation must therefore be implemented outside Spark’s task scheduler. ### Fixed Server Capacity - A server’s image, Driver and Executor resources, and Spark configuration are fixed when it starts. - Dynamic Resource Allocation can adjust executor counts, but cannot change the server’s basic specification. - Flexible scaling and team-level isolation require creating or replacing servers, which is addressed in a later part of the series. ## Reducing Server-Wide Failures Before adding replicas, Toss Securities reduces the chance that one bad query can kill the shared server. - Set `spark.executor.maxNumFailures` effectively high enough to disable the global shutdown mechanism. - Use `spark.executor.failuresValidityInterval` to periodically clear accumulated failure records. - Rely on query-scoped controls: - `spark.task.maxFailures` stops tasks that repeatedly fail due to OOMs or exceptions. - `spark.stage.maxConsecutiveAttempts` stops jobs whose stages repeatedly fail, such as from shuffle-fetch errors. - These limits must be tuned carefully: overly aggressive values can cause healthy queries to fail during temporary infrastructure problems. - With this approach, executor failures terminate the problematic query rather than the entire Spark Connect server. ## Protecting Driver Memory from Large Results - Spark Connect streams query results through the Driver, so a large `collect()` can threaten Driver memory. - `spark.driver.maxResultSize` aborts an action when accumulated task results exceed the configured limit. - The limit is checked before large executor-side results are fetched into Driver memory. - The default 1 GB value assumes a single workload; in a multi-session server, it should be reduced or tuned based on the number of concurrent queries. ## Replicating Spark Connect Servers - Configuration alone cannot prevent Driver OOMs, node failures, or other catastrophic events. - The stronger isolation boundary is a separate SparkContext. - Multiple identical Spark Connect replicas are deployed: - Each replica has its own Driver, SparkContext, and Executors. - A failure affects only the sessions assigned to that replica. - Other replicas can continue accepting sessions. - Replica-based deployment reduces the blast radius from the entire Spark Connect service to an individual server instance. ## Practical Recommendation For a multi-user Spark Connect service, disable global executor-failure shutdown, enforce query-level failure limits, protect Driver memory with `spark.driver.maxResultSize`, and use multiple replicas to contain unavoidable Driver or node failures. Scheduler pools can improve ordering, but they should not be treated as true resource isolation.

Read original(opens in new tab)
KubernetesApache SparkGrpcFault Tolerance+4
aws2 min readCurated summary

Announcing Amazon EC2 G7 instances accelerated by NVIDIA RTX PRO 4500 Blackwell Server Edition GPUs | Amazon Web Services

Amazon EC2 G7 instances are now generally available with NVIDIA RTX PRO 4500 Blackwell Server Edition GPUs and custom sixth-generation Intel Xeon processors. Compared with G6 instances, they provide up to 4.6× higher AI inference performance and 2.1× better graphics performance. AWS positions them for AI inference, rendering, video, virtual desktops, spatial computing, and GPU-accelerated analytics. ## GPU and Performance Improvements - Each GPU provides 32 GB of memory, with up to 256 GB across eight GPUs. - GPU memory capacity is 1.33× higher and bandwidth is 2.45× higher than G6. - GPUs include fifth-generation Tensor Cores and fourth-generation RT Cores. - G7 instances accelerate analytics workloads running on Amazon EMR with Amazon EKS. ## Networking and Storage - Up to 700 Gbps of EFA-enabled networking—seven times the G6 throughput. - Up to 7.6 TB of local NVMe SSD storage keeps large models and datasets close to the GPUs. - Support for NVIDIA GPUDirect P2P and GPUDirect RDMA with EFA enables low-latency GPU communication across GPUs, nodes, and FSx for Lustre. ## Video Processing - Ninth-generation NVENC and sixth-generation NVDEC engines support 4:2:2 encoding and decoding. - They deliver up to 1.5× more concurrent video streams than G6 instances. ## Instance Configurations - Seven instance sizes are available. - Configurations offer up to: - 8 NVIDIA RTX PRO 4500 GPUs - 192 vCPUs - 768 GiB of system memory - 700 Gbps network bandwidth - 7.6 TB local NVMe storage - Detailed instance specifications were listed as “coming soon” in the announcement. ## Software and Availability - AWS provides Deep Learning AMIs and NVIDIA Workstation AMIs with preinstalled drivers. - Amazon EKS users should build AMIs with NVIDIA driver version R595. - Supported operating systems include Amazon Linux, Ubuntu, RHEL, and Windows Server. - NVIDIA integration supports DirectX, Vulkan, and OpenGL. - G7 instances are initially available in US East (Ohio) and US West (Oregon). - Purchasing options include On-Demand, Savings Plans, Spot Instances, and Dedicated Instances for selected sizes. G7 instances are a strong option for GPU-intensive workloads requiring higher inference, graphics, networking, and video performance. Organizations can launch them through the EC2 console and evaluate pricing across the available purchasing models.

Read original(opens in new tab)
KubernetesAWSAmazon Ec2Gpu Acceleration+4
gitlab2 min readCurated summary

Introducing the 2026 EMEA GitLab Partner Award winners

GitLab announced its 2026 EMEA Partner Award winners, recognizing organizations that drove customer success, technical innovation, certification, business growth, and joint marketing. The awards highlight partners helping enterprises adopt DevSecOps, cloud-native platforms, managed services, and AI-enabled software development across the region. ## Regional Partners of the Year - **Central Europe: cc cloud GmbH** — Combines infrastructure and DevOps expertise to manage cloud applications, platforms, and IT operations. - **Northern Europe: Eficode** — Supports more than 1,600 customers through consulting, managed services, toolchain implementation, and AI-augmented development. - **Southern Europe: Kiratech** — Helps enterprises modernize infrastructure using cloud-native, DevOps, and PlatformOps practices. - **Eastern Europe and Israel: Bynet** — An established systems integrator supporting enterprise IT, cloud, cybersecurity, modernization, DevSecOps, and AI adoption. ## Technical and Enablement Awards - **Best Technical Solution/Project: Capgemini | Sogeti** — Recognized for impactful, complex technical solutions using AI-driven quality engineering, data, and cloud capabilities. - **Most Certified and Enabled Partner: Devoteam** — Awarded for having the largest number of GitLab-certified professionals. - **Rookie of the Year: ITDOTCOM** — A Uzbekistan-based technology distributor that achieved rapid success supporting software, infrastructure, cybersecurity, and business automation across Central Asia. ## Growth and Collaboration Awards - **First Order Master: Linux Polska** — Recognized for winning new customers and business through open-source consulting, DevOps, automation, containerization, and data analytics. - **Co-marketing Partner of the Year: Conoa, a PROACT Company** — Honored for joint marketing efforts and expertise in Kubernetes, cloud-native technologies, container platforms, and managed operations. The awards demonstrate the breadth of GitLab’s EMEA partner ecosystem, from regional systems integrators and cloud specialists to technical consultants and Kubernetes providers. Together, these partners are helping customers modernize delivery practices and adopt DevSecOps and AI capabilities.

Read original(opens in new tab)
KubernetesAIGitlabDevSecOps+4
google3 min readCurated summary

A low-carbon computing platform from your retired phones

Retired smartphones can become low-carbon cloud infrastructure by reusing their still-capable motherboards instead of manufacturing new servers. Researchers at UC San Diego, with Google’s support, are developing clusters of stripped-down Pixel phones managed by Kubernetes. Their planned 2,000-phone datacenter aims to provide affordable computing for education and research while reducing hardware-related emissions. ## The Carbon Case for Reusing Smartphones - Computing emissions come from: - **Operational carbon**, produced by electricity consumed during use. - **Embodied carbon**, produced during hardware manufacturing and raw-material extraction. - Reusing phones primarily addresses embodied carbon by extending the life of components that remain functional. - Since people typically replace phones every four years, many retired devices still contain capable processors, accelerators, memory, and storage. ## Smartphone Performance and Limitations - Modern smartphone performance cores can match or exceed the per-core performance of some data-center servers. - Smartphones have significant limitations compared with servers: - Fewer, heterogeneous processor cores. - Only 8–12 GB of memory. - Less capacity for large, multithreaded workloads. - The platform therefore targets workloads that fit on a phone or can be distributed across multiple devices. ## Converting Phones into Datacenter Hardware - Unmodified phones are unsuitable for datacenters because they include unnecessary and potentially hazardous components such as: - Displays and cameras. - Batteries not designed for sustained datacenter operation. - Consumer-oriented chassis and peripherals. - Researchers remove everything except the motherboard, which accounts for roughly 50% of a phone’s embodied carbon. - Android’s mobile userspace is replaced with a general-purpose Linux distribution. - This removes mobile-specific restrictions such as Android’s “low memory killer” and enables broader server-style programmability. - Kubernetes manages containerized applications across clusters of approximately 25–50 phones, equivalent to roughly one conventional server. ## Applications for Education and Research - Many university workloads—including Jupyter notebooks, grading systems, and research applications—require modest resources that a single smartphone can provide. - Early tests showed that a 20-phone cluster could handle peak grading demand for a class of more than 75 students while achieving latency below a typical AWS backend. - The planned 2,000-phone cluster could support around 100 comparable classes simultaneously. - The deployment would provide approximately 50 server-equivalents at substantially lower cost. ## Testing Computing at Scale - The project will evaluate whether consumer smartphone hardware can operate reliably under sustained datacenter workloads. - It will also serve as a large-scale testbed for distributed smartphone computing. - The system is expected to launch at UC San Diego in fall 2026. Repurposing retired phones offers a practical way to reduce demand for newly manufactured computing hardware, especially for lightweight academic and cloud workloads. The approach is most promising when applications can tolerate distributed resources and the reliability challenges of consumer-grade components.

Read original(opens in new tab)
KubernetesAndroidDistributed SystemsCloud Computing+4