Infrastructure As Code

15 posts

gitlab3 min readCurated summary

Secure every commit to production with Claude and GitLab

Claude’s security tools can identify and fix vulnerabilities during coding, but they do not govern everything that happens afterward. The post argues that GitLab should secure the remaining path to production through enforced scans, approval policies, data controls, lifecycle-wide coverage, and audit evidence. Together, Claude handles in-session authoring while GitLab provides organization-wide governance. ## From In-Session Findings to Enforced Controls - GitLab applies security configuration profiles across projects and pipelines, preventing teams from bypassing required scans. - Merge request approval policies enforce separation of duties, ensuring an agent or developer cannot approve and merge its own changes. - Critical vulnerabilities can block merges until a designated approver reviews them. - Vulnerability reports and security dashboards permanently track whether findings were detected, dismissed, or resolved. ## Turning Security Activity into Audit Evidence - Compliance controls require scans to run on every merge request and expose findings for human review. - Pipeline logs, approval records, and audit events provide reproducible evidence of what was scanned and who approved each change. - Compliance frameworks such as SOC 2, PCI DSS, and FedRAMP can map evidence to named requirements. - Compliance status reports show which controls have passed, failed, or remain pending, including activity involving agents. ## Controlling Sensitive Data Sent to Models - Context exclusions prevent secrets, proprietary files, and regulated data from being sent to AI models. - Self-managed GitLab environments and self-hosted models can keep code and inference within an organization’s boundary. - Teams can select approved models, restrict model access, and prevent code from being used for training. - GitLab Duo prompt guardrails detect secrets and limit the content available to prompts, reducing prompt-injection risk. ## Scanning Beyond the Coding Session - Session-based reviews cannot detect vulnerabilities introduced after code is written, such as newly disclosed dependency flaws like Log4Shell. - GitLab provides dependency, container, infrastructure-as-code, secret, and DAST scanning across the delivery lifecycle. - Security Review Flow uses reasoning to identify business-logic flaws, authorization errors, and race conditions that deterministic scanners may miss. - Advanced SAST produces reproducible, CWE-mapped results, providing more consistent evidence than potentially variable LLM-based findings. ## Consistent Coverage for Humans and Agents - Claude’s plugin primarily reviews code written and committed within its session. - Developer shell commands, including Claude’s `!` shell escape, may fall outside that review scope. - Claude Security can review broader codebases when explicitly invoked by a developer or administrator. - GitLab pipeline scans and merge request policies apply to every change, regardless of whether it was written by a human or an agent. GitLab’s policies and lifecycle-wide scanning provide the durable controls needed after an AI coding session ends. Teams can use Claude for immediate security feedback while relying on GitLab to enforce consistent review, protect sensitive data, block unsafe releases, and maintain audit-ready evidence through production.

Read original(opens in new tab)
aws2 min readCurated summary

Accelerate your infrastructure deployments by up to 4x with AWS CloudFormation Express mode | Amazon Web Services

AWS CloudFormation Express mode speeds deployments by marking them complete once resource configuration is applied, rather than waiting for full stabilization checks. AWS says this can reduce deployment times by up to four times, while resources continue becoming operational in the background. It is intended for rapid infrastructure iteration and scenarios where eventual stabilization is acceptable, not workflows requiring resources to be fully ready before proceeding. ## How Express Mode Works - Standard CloudFormation deployments wait for post-configuration stabilization checks. - Express mode completes earlier, immediately after configuration is applied. - Resources continue stabilizing asynchronously. - CloudFormation retries dependent resources that encounter transient provisioning failures. - The provisioning process itself is unchanged; only the point at which deployment completion is reported changes. ## Performance Improvements - Creating an SQS queue with a dead-letter queue took: - Standard mode: 64 seconds - Express mode: up to 10 seconds - Deleting a Lambda function with attached network interfaces took: - Standard mode: 20–30 minutes - Express mode: up to 10 seconds in AWS’s benchmark ## Best Use Cases - Iteratively building infrastructure one component at a time. - Testing individual application components. - AI-assisted infrastructure development requiring sub-minute feedback. - Production workflows that can tolerate resources stabilizing after deployment completion. ## Enabling Express Mode - In the AWS Console, select **Enable** under stack deployment options. - With the CLI or SDKs, set the deployment configuration mode to `EXPRESS`: ```bash aws cloudformation create-stack \ --stack-name my-app \ --template-body file://template.yaml \ --deployment-config '{"mode": "EXPRESS", "disableRollback": true}' ``` - AWS CDK supports: ```bash cdk deploy --express ``` - No CloudFormation template changes are required. - Express mode supports existing templates, change sets, nested stacks, and IaC or AI tools such as Kiro. - Enabling it on a parent stack also applies it to nested stacks. ## Rollback and Operational Considerations - Rollback is disabled by default in Express mode to maximize iteration speed. - For production use, rollback can be restored with `"disableRollback": false`. - Teams should otherwise provide monitoring and cleanup procedures for failed deployments. - IAM templates should continue following least-privilege principles. ## Availability - Express mode is available at no additional cost in all AWS commercial Regions. - AWS recommends standard deployment behavior when resources must be fully operational before traffic shifting or testing. For fast development and AI-driven infrastructure iteration, Express mode is a useful optimization. Use it selectively, while retaining standard mode—or explicitly enabling rollback—when deployment readiness and failure recovery are critical.

Read original(opens in new tab)
github2 min readCurated summary

Transitioning as a hubber

Arthur Searle describes transitioning at GitHub as a largely smooth experience, enabled by an inclusive, remote-first culture and strong workplace support. Using handles, written communication, flexible avatars, and gender-affirming benefits reduced many common sources of stress. His experience shows that transition can involve both bureaucratic challenges and profound joy when colleagues respond with acceptance and care. ## A Career Built at GitHub - Searle began in IT support and operations before teaching himself to code. - He joined GitHub’s IT Engineering team after a colleague’s referral and moved to Enterprise Security six months later. - His work has included: - Helping migrate GitHub’s main SaaS platform to infrastructure as code. - Speaking at Oxford University about version control. - Throughout his transition, his handle—“gleeblezoid”—remained constant, providing continuity at work. ## How GitHub’s Culture Supported Transition - GitHub’s remote-first structure reduced anxiety around appearance, commuting, and in-person interactions. - Much of Searle’s work happened through written communication in Slack and GitHub, limiting the pressure of speaking while undergoing voice training and hormone-related voice changes. - Employees commonly use handles and informal avatars, making gender assumptions based on appearance less central. - Searle was able to update his name and pronouns in internal systems, with colleagues consistently using them. ## Gender-Affirming Benefits - GitHub covered gender-affirming healthcare for employees. - Benefits included reimbursement for: - Voice training. - Hormone replacement therapy prescriptions. - Therapy. - The main remaining difficulty was ordinary administrative friction, such as changing his legal name in payroll systems. ## Acceptance, Joy, and Belonging - Searle contrasts his experience with people who remain closeted, repeatedly come out to new coworkers, or face extensive bureaucracy. - Colleagues treated his transition as a normal part of his life and expressed genuine happiness for him. - Small gestures had a major emotional impact, including hearing his name and pronouns used at work for the first time and receiving a shaving kit from a teammate. - He emphasizes that being trans is not defined only by hardship; there is also joy in living openly and being supported by others. GitHub’s example suggests that inclusive policies, flexible communication practices, and everyday respect can make workplace transition significantly safer and more affirming. For organizations, support should extend beyond formal benefits to the culture and systems employees use every day.

Read original(opens in new tab)
line4 min readCurated summary

Flava DBaaS Deep Dive: From Architecture to Migration and Beyond

LY Corporation’s Flava DBaaS is designed to unify the former Verda and YNW cloud platforms on a Kubernetes-based architecture. Its operator pattern separates database business logic from IaaS management, while API servers, managers, and agents divide responsibilities within each DBMS service. The platform expands database support, improves scalability, security, and usability, and treats migration from legacy platforms as part of the DBaaS responsibility. ## Kubernetes Operator-Based Design - Flava DBaaS uses the Kubernetes operator pattern. - Users declare the desired database state through custom resources rather than issuing procedural commands. - Controllers continuously reconcile the actual state with the declared specification. - This approach: - Simplifies troubleshooting through resource status and controller logs. - Handles large database infrastructures efficiently through event-driven processing. - Reuses Kubernetes capabilities for CI/CD and access control. ## Infrastructure Operator Layer - DBaaS must manage IaaS resources such as: - Virtual machines - Storage - Domains and networking - Flava isolates this infrastructure logic in a separate infrastructure operator. - IaaS resources are exposed as Kubernetes custom resources, allowing DBaaS to create infrastructure declaratively without directly calling IaaS APIs. - The resulting layers are: - **DBaaS:** Database-specific business logic - **Infrastructure operator:** Abstraction of IaaS as Kubernetes resources - **IaaS:** Compute, network, and storage services - This separation allows multiple DBMS products to use infrastructure consistently while their developers focus on database operations. ## Custom Resources and DBaaS Components - Each database cluster is represented by a Kubernetes custom resource containing settings such as: - DBMS version - VM size - Storage type and capacity - Replication configuration - These resources are stored in Kubernetes etcd and managed through the Kubernetes API. - Each DBMS implementation consists of three components: - **API server:** Provides REST APIs for creating, modifying, and deleting database resources. Flava UI and IaC tools use these APIs. - **Manager:** Watches resource changes and reconciles the database cluster toward the declared state. - **Agent:** Runs on database VMs and executes local operating-system and database commands. - For example, creating a MySQL cluster causes the API server to create a MySQL custom resource, the manager to provision the required VMs through the infrastructure operator, and the agent to configure replication and database processes inside those VMs. ## Improvements in Flava DBaaS - Flava preserves core DBaaS capabilities such as provisioning, high availability, backup and recovery, scalability, and monitoring. - It combines the DBMS offerings of Verda and YNW, expanding the range of supported database systems. ### Flexible Storage and Scaling - Storage can be configured in 100 GiB increments. - Block-storage-based databases can use up to 5 TiB of storage. - Unlike the legacy platforms, storage is no longer tightly limited by a VM’s local disk capacity. - Custom instance types and separate block storage reduce the need to consider alternatives such as sharding for larger databases. - The 5 TiB limit was selected to cover most analyzed use cases while reducing infrastructure fragmentation. ### Consistent User Experience - All Flava DBaaS products share a common architecture and UI. - Skills learned while changing MySQL server specifications or configuring Cassandra alerts can be applied to other DBMS products. - Users do not need to learn separate operational workflows for each database system. ### Security and Convenience - TDE and TLS are provided as platform-level security features. - Additional features include: - **Custom DB Role:** Reusable database users with configurable permissions. - **Database Parameter Group:** Reusable groups of database configuration parameters. - **Restore backup:** Creation of a new cluster from a selected backup for disaster recovery or realistic performance testing. - Features not yet available for every DBaaS product are planned for broader support. - These improvements reportedly resulted in high internal user-satisfaction scores. ## Migration Responsibilities - A new DBaaS platform is expected to provide migration paths from existing platforms, not merely offer new database clusters. - For migrations between the same DBMS type, the article identifies three general approaches. ### Dump and Restore - Data is backed up from the source database and restored into the destination. - It is the simplest method. - To guarantee consistency, the application generally must be stopped during the migration. ### Replication-Based Migration - The source database is continuously replicated to the destination. - Once replication is caught up, the destination is promoted through failover. - The source database can then be removed. - Data consistency depends on the DBMS’s replication mechanism. - A short application interruption may still occur during primary-node failover. The overall recommendation is to use Flava’s layered, declarative architecture to standardize database operations while continuing to provide practical migration mechanisms from Verda and YNW.

Read original(opens in new tab)
line4 min readCurated summary

From Automation to AI with Infrastructure as Code (IaC): Adopting OpenTofu and ChatOps

LY Corporation’s LINE Plus SRE team migrated Verda cloud infrastructure and IMON monitoring resources from scattered, manual management into an Infrastructure as Code (IaC) and GitOps workflow. Using OpenTofu and Terragrunt, they now manage roughly 1,500 resources across seven services through pull requests, CI/CD, and daily drift detection. The migration required careful automation for importing existing resources, normalizing state, and handling provider limitations and resource dependencies. ## Why IaC Was Needed - Teams previously managed infrastructure through different methods: - Verda’s web dashboard - Scripts - Wiki-based procedures - Personal documents and GitHub repositories - As the number of services and resources grew, this caused: - Inconsistent management practices - Difficult-to-track configuration changes - Greater risk of manual errors - Poor reproducibility and reviewability - The team adopted GitOps so that: - Desired infrastructure state is declared in Git. - All changes go through pull requests. - Infrastructure history is versioned and auditable. - CI/CD applies approved changes automatically. - Their goal was to manage infrastructure with the same engineering standards as application code: reviewable, version-controlled, and reproducible. ## Choosing OpenTofu and Terragrunt - OpenTofu was selected as an open-source Terraform fork. - It retains: - Terraform’s HCL syntax - Provider compatibility - Familiar module and configuration patterns - The team created reusable modules for: - Virtual machines - Load balancers - Monitoring alerts - Modules were versioned so updates could be adopted explicitly rather than affecting every environment immediately. - Terragrunt was added to reduce repetition in environment configuration. - Shared settings are defined once in a parent `root.hcl`. - Individual environments contain only their differing inputs. - OpenTofu reduces duplication in resource definitions, while Terragrunt reduces duplication in environment and backend configuration. ## Planning the Migration - The most difficult part of introducing IaC into an existing environment was importing resources that were already running. - Manual import was considered impractical for hundreds of VMs, load balancers, and DNS records because it would be slow and error-prone. - The migration was split into two phases: - **Phase one:** Automate imports, select one service for a pilot, and establish the complete OpenTofu/Terragrunt pipeline. - **Phase two:** Reuse the validated modules and import scripts to roll the approach out to the remaining services. ## Designing the Import Process - Import scripts were designed to: - Query existing resources - Decide which resources should be managed by IaC - Convert resource data into the desired code structure - Generate Terragrunt configuration - Connect resources to OpenTofu state - Run `plan` to verify that no unintended changes would occur - A key requirement was keeping three representations aligned: - Configuration code - OpenTofu state - Actual cloud resources - Normalization was added because equivalent values could be represented differently—for example, network or image IDs—causing OpenTofu to report misleading differences after import. ## Resource-Specific Import Strategies - Resources could not all be imported using the same procedure. - Different resource types have different identifiers, dependencies, and ownership models: - **VMs:** Imported individually, while distinguishing manually created instances from Kubernetes-managed instances. - **Load balancers:** Imported together with related listeners and pools. - **DNS:** Imported while preserving zone and record relationships. - **Kubernetes:** Structured around clusters and node pools. - **IMON:** Imported according to its hierarchy of teams, alert groups, alert rules, and monitors. - Each resource followed the same broad five-step process, but its implementation was adapted to the resource’s characteristics. ## Problems Discovered During Migration ### Kubernetes-Managed VMs - OpenStack contained both manually created VMs and VMs automatically created by Kubernetes. - Importing Kubernetes-managed VMs into IaC could cause conflicts between OpenTofu and Kubernetes. - The scripts excluded these VMs using naming patterns and metadata. ### IMON’s Hierarchical Structure - IMON alerts are organized as: `Team → Alert Group → Alert Rule → Monitor` - A flat import would lose these relationships. - The team mirrored the hierarchy in the directory structure so ownership and relationships were visible from the file layout. ### Provider and Regional Identifier Issues - The actual cloud platform allowed both hyphens and underscores in load balancer names, but the provider validation logic rejected underscores. - The team fixed this by modifying the provider’s validation logic and contributing the change upstream. - Resource UUIDs such as `flavor_id`, `image_id`, and `network_id` differed by region. - This produced unnecessary changes in `plan`. - The modules added regional mapping logic, allowing users to specify readable names while resolving them to region-specific IDs. The migration demonstrates that successful IaC adoption requires more than writing configuration files: existing infrastructure must be filtered, normalized, modeled according to its dependencies, and validated against real provider behavior. OpenTofu and Terragrunt provided the foundation for scalable GitOps management, while custom import automation and provider improvements made the transition safe for production resources.

Read original(opens in new tab)
gitlab2 min readCurated summary

Mythos-class Claude Fable 5 arrives on GitLab Duo Agent Platform

Claude Fable 5, Anthropic’s Mythos-class model, is now available through GitLab Duo Agent Platform across all tiers and deployment models. The post presents it as a major step beyond incremental model improvements, emphasizing stronger first-attempt accuracy, sustained autonomous workflows, and better code review and incident analysis. Its main conclusion is that teams should apply it to complex, previously difficult engineering problems rather than routine tasks. ## More Accurate First-Pass Results - Produces more correct implementations for complex, well-defined problems with fewer iterations. - Reduces back-and-forth in Duo Agentic Chat. - Performs especially well on: - Multi-file refactoring - Incident investigation - Infrastructure-as-code definitions - Interprets technical images, web applications, and detailed screenshots more accurately while often using fewer output tokens. ## Longer and More Reliable Agent Workflows - Sustains goal-directed work across extended, multi-day runs and millions of tokens. - Maintains instructions and task focus without frequent manual checkpoints or re-prompting. - Uses verification loops to identify and correct its own mistakes. - Handles parallel sub-agents more reliably across multiple repositories or services. - Reduces the amount of human oversight required per agent run, allowing teams to review results asynchronously. ## Improved Bug Detection and Incident Response - Provides higher bug-finding recall than previous models. - Improves outage triage, root-cause analysis, and repository-history investigation. - Finds deeper code-path issues and edge cases during merge request reviews. - Produces more actionable review comments instead of generic feedback. - Helps reduce production defects and improve mean time to resolution. ## Recommended Use Cases - The post recommends testing the model on difficult, unsolved engineering problems rather than routine work. - Suggested applications include: - Complex multi-file refactors - Production incident investigations - Implementations developers would normally write manually - Teams should allow agents to scope work, ask clarifying questions, and execute multi-step solutions. Claude Fable 5 became available on GitLab Duo Agent Platform on June 9, 2026. Teams can access it through a free trial, the GitLab free tier, or included GitLab Credits for Premium and Ultimate subscribers.

Read original(opens in new tab)
cloudflare3 min readCurated summary

Turning Cloudflare’s threat indicators into real-time WAF rules

Cloudflare now lets security teams turn live Threat Events intelligence into proactive WAF rules. Instead of manually blocking known malicious IPs, organizations can filter traffic by threat actor, targeted industry or country, attack type, and recency. The integration separates always-on detection from mitigation, allowing teams to validate intelligence before enforcing blocks without sacrificing visibility. ## Always-on Threat Detection - The system builds on Cloudflare’s always-on Attack Signature Detection framework. - Threat intelligence runs continuously in the background and enriches HTTP requests with metadata before a WAF action is taken. - This removes the traditional “log versus block” trade-off: teams can observe detections in analytics and later create blocking rules. - Cloudforce One subscribers can review threat actors, targeted industries, and traffic patterns before enabling mitigation. - Detection is designed to add negligible latency. - The initial release supports IP-based matching, with planned support for JA3 fingerprints and domain-based indicators to identify attackers who rotate IP addresses. ## New WAF Intelligence Fields Cloudflare exposes threat context through fields that can be used in WAF custom rules and rate-limiting policies: - `cf.intel.ip.attacker_names`: Known threat groups, such as `CRAVENFLEA`. - `cf.intel.ip.target_industries`: Industries previously targeted by the IP. - `cf.intel.ip.attacker_countries`: Countries associated with the threat event. - `cf.intel.ip.target_countries`: Countries targeted by the event. - `cf.intel.ip.datasets`: Intelligence sources or attack categories, such as `ddos` and `waf`. Because an IP may have multiple associated actors, industries, or datasets, these values are arrays. Rules use `any()` with the `[*]` wildcard to match individual values. ## Example WAF Rules - Block IPs involved in DDoS activity against France: ```text any(cf.intel.ip.target_countries[*] == "FR") and any(cf.intel.ip.datasets[*] == "ddos") ``` - Block BLACKBASTA-associated traffic targeting banking and financial services: ```text any(cf.intel.ip.target_industries[*] == "Banking & Financial Services") and any(cf.intel.ip.attacker_names[*] == "BLACKBASTA") ``` - Apply broad protection against traffic originating from Iran: ```text any(cf.intel.ip.attacker_countries[*] == "IR") ``` ## Deployment Through Existing Workflows - The fields are available in the WAF rule builder for custom rules and rate limiting. - Teams can configure them through the Cloudflare API or Terraform. - Rules can be automated across individual domains or an entire account. - Security Analytics records matches, including the triggered rule and the specific intelligence indicator involved. - Threat Events users can create Saved Views—such as IPs targeting the financial sector within the last seven days—and convert them into WAF rules with one click. ## Global Distribution and Performance - Cloudflare compresses threat intelligence datasets into a high-performance format. - The data is distributed to Cloudflare data centers worldwide. - The WAF can therefore evaluate threat indicators at the edge without requiring local IP lists or introducing significant request latency. - The post begins to describe this lookup process as O(1), but the provided content ends before explaining the implementation in detail. Cloudflare’s integration gives teams a practical path from threat intelligence to enforcement: investigate indicators, validate them in analytics, and deploy precise WAF or rate-limiting rules through the UI, API, or Terraform. Teams should use the always-on visibility to tune criteria before blocking, while preparing for future indicators such as JA3 fingerprints and malicious domains.

Read original(opens in new tab)
aws3 min readCurated summary

Introducing the next generation of AWS Resilience Hub for generative AI-based SRE resilience journey | Amazon Web Services

AWS introduces the next generation of AWS Resilience Hub to help organizations manage application resilience consistently across large portfolios. The update adds a new business-oriented application model, modular resilience policies, dependency discovery, generative AI failure-mode analysis, and organization-wide reporting. Its goal is to help SREs and development teams define expectations, identify weaknesses, implement fixes, and demonstrate resilience through testing. ## Modular Resilience Policies - Policies are built from composable requirements rather than a single fixed policy type. - Requirements can include: - Availability service-level objectives (SLOs) - Multi-AZ and multi-Region disaster recovery - Recovery time objectives (RTOs) - Recovery point objectives (RPOs) - Backup data recovery requirements - Example policies can define 99.95% availability, a 15-minute RTO, and a 5-minute RPO for financial applications. ## Business-Oriented Application Modeling - A **system** represents a business application. - **User journeys** describe critical end-user paths and business outcomes. - **Services** represent deployable units such as microservices, AWS resources, code, and observability components. - Resilience Hub automatically maps resource relationships into a topology showing data flow, containment, and permissions. ## Dependency Discovery - Resilience Hub analyzes VPC DNS query logs to identify dependencies that may not be documented. - It can discover: - AWS services - Internal endpoints - Third-party endpoints - Unexpected cross-Region calls - Critical external dependencies - Dependency discovery can be enabled or disabled for individual services. ## Generative AI Failure-Mode Assessments - Assessments analyze services against: - Configured resilience policies - AWS Well-Architected best practices - The AWS Resilience Analysis Framework - Findings explain the failure mode, its architectural impact, recommended remediation, and related policy requirement. - Users can add or modify assertions to guide the AI agents and improve assessment accuracy. - Findings can be marked as resolved after remediation or irrelevant when they do not apply. ## Getting Started and Enterprise Management - Users must configure an invoker IAM role granting read-only access to AWS resources. - Cross-account roles or AWS Organizations service-linked roles can support multi-account environments. - AWS Organizations integration allows a delegated administrator to assess resilience across the enterprise without signing into individual accounts. - A typical workflow is: - Create a resilience policy. - Create a system and service. - Associate resources through tags, CloudFormation, Terraform, or Amazon EKS. - Enable dependency discovery. - Run a failure-mode assessment. - Review and implement recommendations. ## Migration, Availability, and Pricing - Migration APIs convert older Resilience Hub policies and applications into the new model, including mapping multiple related applications into one system with multiple services. - The new version is generally available in supported AWS commercial Regions. - Pricing uses a service-based model and includes two failure-mode assessments per service per month, with automated dependency assessment available optionally. - AWS offers a free trial. The updated Resilience Hub is most useful for organizations that need consistent resilience standards and evidence across many AWS accounts and applications. Teams should begin by defining reusable policies, modeling critical services and dependencies, and using the AI assessments to prioritize remediation.

Read original(opens in new tab)
line4 min readCurated summary

LY Corporation's Cloud Infrastructure Reorganization: Introducing the Architecture of Flava, a Next-Generation Platform Integrating Two Massive Clouds

LY Corporation is consolidating the former LINE “Verda” and Yahoo Japan “YNW” private clouds into Flava, a next-generation platform designed for large-scale, uninterrupted operations. Its approach assumes failures will occur, prioritizing stateless services, application-led availability, rapid IaC-based recovery, and extensive automation. Flava also restructures the architecture around shared resources, upstream OpenStack, default VPC networking, and user-driven cost optimization. ## Failure-Aware Design and Operations - VM root disks are treated as temporary; persistent data is placed in external storage so instance failures have limited service impact. - Availability is achieved through cooperation between infrastructure and applications rather than excessive infrastructure-side guarantees. - Recovery focuses on maintaining service continuity, rebuilding environments quickly with infrastructure as code, and avoiding lengthy root-cause investigations during incidents. - The company promotes KaaS and PaaS to help developers build resilient services without managing low-level infrastructure. - OS configuration, package installation, networking, and other changes are managed as code through CI/CD. - Deployments are performed by availability zone to limit the blast radius of failures. ## Observability from Fleet-Wide Trends to Root Causes - Prometheus, Grafana, and custom dashboards monitor overall cloud health and long-term trends. - When anomalies appear, engineers investigate at a deeper level using kernel traces, packet captures, and other low-level diagnostics. - This combination of broad monitoring and detailed investigation allows teams to move between “forest” and “tree” perspectives. - The operational model depends not only on tools but also on engineers capable of tracing problems down to their fundamental causes. ## OSS, Software-Defined Infrastructure, and Custom Development - The platform relies heavily on OpenStack, Envoy, Linux kernel technologies such as eBPF/XDP, FRR, and Ceph. - LY contributes patches and new capabilities upstream instead of maintaining long-lived private forks. - It has developed SRv6 BGP functionality required for Flava’s VPCs and contributed related work to FRRouting and the Linux kernel. - Compute, VPC, DNS, and load-balancing services run primarily on commodity x86 servers rather than specialized appliances. - XDP-based data planes, hardware offload, and system tuning are used to achieve near-wire-speed throughput and low latency. - Where OSS cannot meet internal requirements, LY builds systems from scratch, including the Dragon object store, SDN control-plane components, load-balancer health agents, and service discovery tools written in Rust, Go, and Python. ## Autonomous Hardware Operations - With tens of thousands of hypervisors and petabyte-scale storage, hardware failures occur continuously. - Failure detection, requests to data-center technicians, hardware replacement, and cluster reintegration are largely automated. - Some exceptional cases still require engineers, but LY plans to use LLMs to automate more of these operational tasks. ## Flava’s Architectural Improvements ### Shared Resource Pools - Older clouds used many dedicated clusters and resource pools, making capacity planning complex and reducing utilization. - Flava consolidates most products and services into one large shared resource pool. - This reduces planning variables, improves resource efficiency, and accelerates provisioning. ### Upstream-Compatible OpenStack - Excessive customization in the legacy environment made upgrades difficult. - Flava minimizes private patches, follows upstream OpenStack, and contributes necessary improvements back to the project. - This enables regular upgrade cycles and keeps security fixes and features current. ### VPC by Default - VPC networking is the standard security model for multi-tenant workloads. - Logical isolation replaces many cases where dedicated VLANs or firewalls previously required months of preparation. - Equivalent security environments can now be provisioned in minutes. - The VPC data plane is being redesigned with XDP to support the reliability and performance required at company-wide scale. ### Built-In Cost Optimization - Development environments require resource lifetimes, allowing unused “zombie” resources to be deleted automatically. - Object storage offers bucket classes such as “High Performance” and “Scalable.” - Users can change storage classes without changing endpoints, adapting cost and performance as access patterns evolve. ## Remaining Challenges - Flava currently offers only a limited set of products and must expand its capabilities while addressing post-launch bugs and overlooked requirements. - The largest challenge is migrating users from the legacy platforms. - LY is working to provide transparent migration tools and reduce manual effort while shortening the period of duplicate investment in old and new infrastructure. ## Team and Engineering Culture - The team includes specialists ranging from kernel developers to web-front-end engineers. - Engineers are expected to understand and control infrastructure rather than treat it as a black box. - Deep source-level expertise enables upstream OSS contributions and informed negotiations with commercial vendors. - This culture of ownership and technical control is presented as a core reason the platform can evolve at LY’s scale. LY’s experience demonstrates that large private clouds can combine OSS, custom software, commodity hardware, and rigorous automation effectively. The practical recommendation is to design for failure, keep infrastructure reproducible through IaC, contribute changes upstream where possible, and use custom development selectively for requirements that general-purpose platforms cannot satisfy.

Read original(opens in new tab)
cloudflare3 min readCurated summary

The truly programmable SASE platform

Cloudflare argues that true SASE programmability goes beyond APIs, Terraform, webhooks, and alerts. It means intercepting security events, enriching them with external context, and making real-time decisions through custom logic. By running Cloudflare One and its Developer Platform on the same global edge network, Cloudflare aims to let customers apply programmable, low-latency policies without stitching together separate infrastructure. ## What “Programmability” Means - Traditional programmability supports configuration and automation, such as sending Slack alerts when policies trigger. - True programmability allows security systems to: - Inspect an event before access is granted. - Query external systems for additional context. - Make or change an access decision in real time. - Example: a request to a regulated application could be checked against a learning management system to confirm that the user’s compliance training is current. Expired or missing certification would result in denial and redirection to training. ## Cloudflare’s Programmable SASE Architecture - Cloudflare’s network spans more than 330 cities and reaches approximately 95% of Internet-connected users within 50 milliseconds. - Cloudflare One and the Developer Platform run on the same infrastructure and use shared network primitives. - This enables Workers to extend inline services such as Access without requiring separate cloud infrastructure. - Customers can: - Call external risk APIs. - Add dynamic request headers. - Validate browser attributes. - Route traffic according to custom business logic. - Running custom logic at the edge reduces latency and avoids the operational overhead of webhook-based integrations and disconnected systems. ## Custom Actions in Security Policies - Conventional gateways generally limit policy outcomes to actions such as allow, block, isolate, or quarantine. - Cloudflare is expanding policies to support managed and custom actions. - Potential uses include: - Injecting headers based on user identity claims. - Obtaining real-time verdicts from external risk engines. - Restricting access based on location or working hours. - Updating risk lists based on scheduled analysis of user activity. - Custom actions can invoke a Worker when a Gateway HTTP policy matches, giving the code access to request context and allowing decisions in milliseconds. - Managed actions will offer templates for common use cases such as IT service management, redirects, and compliance workflows. ## Automated Device Session Revocation - One customer needed periodic re-authentication for Cloudflare One Client users, similar to traditional VPN session expiration. - Cloudflare’s built-in session controls were application-specific rather than global and time-based. - The customer implemented a scheduled Worker that: - Queries the Cloudflare Devices API. - Handles cursor-based pagination to retrieve registrations. - Calculates how long each device has been inactive. - Deletes registrations exceeding a configured inactivity threshold. - Forces affected users to authenticate again through their identity provider. - The example also supports environment-based configuration and a dry-run mode for testing before revocations are applied. Cloudflare’s recommendation is to treat SASE policies as programmable decision points rather than fixed allow-or-block rules. Combining Cloudflare One with edge Workers can provide faster, more context-aware security automation while reducing integration complexity.

Read original(opens in new tab)
line4 min readCurated summary

Creating the Cloud of the Future

LY Corporation is consolidating Yahoo! JAPAN and LINE’s internal cloud services into Flava, a private cloud for application development. The article outlines how Flava could evolve over the next two to three years through unified developer platforms, stronger yet more usable security, scalable multimedia storage, AI infrastructure, and intelligent cloud management. Its ultimate goal is to make complex infrastructure easier to consume while automating operational work. ## Platform Flavaization - Flava currently focuses on infrastructure, databases, and containers, while other development services are spread across separate internal platforms. - Developers must learn different systems for: - Access control and approvals - Logging, monitoring, metering, and billing - APIs, CLIs, and user interfaces - Multi-region and availability-zone operations - “Flavaization” means offering all development platforms through a consistent cloud experience. - LY expects much of this integration to be completed within the next one to two years. ## Stronger, More Usable Security - Flava incorporates security governance from the architecture and product-planning stages, working with the CISO organization. - Data environments are separated by security level: - Default - Secret - Top secret - Sensitive changes require role-based permissions, organizational reporting, expert review, and formal approval. - The main challenge is usability: - Resources can now be provisioned within minutes, but access may still require around ten workflows, such as VDI and Box account creation, taking up to two months. - VPC ACL controls can add several milliseconds of latency, which may affect latency-sensitive services such as LINE messaging. - Flava must provide “usable security” that preserves strong governance without making development excessively slow or difficult. ## Storage for Growing Multimedia Data - Users continuously generate and retain large volumes of photos, videos, and other multimedia content. - Storage demand can grow even when service traffic remains stable. - Flava needs storage technologies suited to different data lifecycles, balancing: - Cost - Throughput and latency - Searchability - Compression and deduplication - Encryption - Efficient tiered storage will be essential for managing long-lived user data economically. ## AI Operations Platforms - LY is adopting AI tools and agents across its organizations, creating demand for shared AIOps infrastructure. - Potential platform capabilities include: - Approved MCP server development and management - Vector databases - AI observability tools such as Langfuse - AI model management - Because AI systems handle internal data, these platforms must comply with company security and data-processing policies. - Flava aims to rapidly evaluate emerging AI technologies and provide compliant, standardized services across the company. ## Network and Storage Infrastructure for AI - AI workloads process larger datasets while requiring very low network latency and high throughput. - Relevant technologies include: - DPUs - Smart NICs - High-speed NVMe storage - Automated storage tiering - Operating networks and storage at cloud scale introduces major challenges in latency, reliability, fault tolerance, throughput, change management, and security. - Flava’s existing network and storage engineering teams have experience supporting LINE and Yahoo! JAPAN at large scale and will adapt that expertise for AI workloads. ## The Intelligent Cloud - Future users may describe infrastructure requirements in natural language rather than manually configuring resources through consoles, APIs, CLIs, or Terraform. - For example, Flava could translate requirements for image processing, AI-based content labeling, messaging, and tiered storage into an architecture and deployable system. - An intelligent Flava could also: - Generate network diagrams and ACL matrices - Identify vulnerabilities and prioritize remediation - Recommend cost optimizations - Detect underutilized resources - Find unencrypted personal information - Manage OSS vulnerability responses - Chatbots could automate tasks such as identifying low-utilization resources while excluding standby failover servers or proposing cost reductions for them. - Operational campaigns currently requiring substantial engineer participation could increasingly be handled by AI agents. Flava’s recommended direction is to combine a unified cloud experience with practical security, lifecycle-aware storage, AI-ready infrastructure, and natural-language automation. The article argues that building this future cloud requires both deep infrastructure expertise and strong attention to developer and user experience.

Read original(opens in new tab)
aws3 min readCurated summary

AWS Weekly Roundup: Amazon Bedrock agent workflows, Amazon SageMaker private connectivity, and more (February 2, 2026) | Amazon Web Services

The AWS Weekly Roundup highlights new capabilities for AI agents, private connectivity, encryption management, and resilience testing. Major launches include Bedrock server-side tools and longer prompt caching, SageMaker Unified Studio support for PrivateLink, and S3 encryption changes without data movement. Additional updates strengthen event-driven architectures, observability, zero-trust access, and AI-assisted AWS deployments. ## AI Agents and Developer Workflows - Amazon Bedrock’s Responses API now supports server-side tools such as web search, code execution, and database updates within AWS security boundaries. - Bedrock also offers a one-hour prompt-cache TTL for select Anthropic Claude models, improving performance and reducing costs for long-running, multi-turn agents. - AWS MCP Server deployment SOPs, currently in preview, let agents deploy applications from natural-language prompts using CDK, CloudFormation, and CI/CD workflows. - The deployment preview supports React, Vue.js, Angular, and Next.js through tools such as Kiro, Cursor, and Claude Code. - CloudWatch Application Signals integration with Kiro provides AI-assisted investigation of service health, SLO compliance, and observability issues. ## Private Connectivity and Zero-Trust Security - SageMaker Unified Studio now supports AWS PrivateLink, allowing VPC traffic to remain within the AWS network instead of traversing the public internet. - IAM policies can govern private SageMaker connectivity for stricter security and compliance requirements. - AWS Verified Access guidance demonstrates centralized zero-trust application access across multi-account environments using IAM Identity Center and AWS RAM. - AWS Network Firewall adds predefined web categories for identifying and controlling generative AI application traffic, with full-URL filtering available alongside TLS inspection. ## Storage, Encryption, and Database Performance - Amazon S3’s `UpdateObjectEncryption` API changes encryption for existing objects without moving or re-uploading data. - Supported operations include switching from SSE-S3 to SSE-KMS, rotating customer-managed KMS keys, and standardizing encryption with S3 Batch Operations. - Amazon Keyspaces table pre-warming prepares tables for predictable high-throughput workloads, reducing throttling and cold-start delays during traffic spikes. - Pre-warming works with on-demand and provisioned capacity, including multi-Region tables. - DynamoDB MRSC global tables now integrate with AWS Fault Injection Service, enabling simulated Regional failures and validation of replication and application resilience. ## Event-Driven Systems and Observability - EventBridge’s event payload limit increased from 256 KB to 1 MB, allowing events to carry richer JSON, telemetry, ML, and generative AI data without external storage or fragmentation. - Lambda’s enhanced observability for Kafka event source mappings adds CloudWatch logs and metrics for polling, scaling, processing state, permissions, and failures. - The feature supports both Amazon MSK and self-managed Apache Kafka sources. ## CloudFormation and Community - AWS’s 2025 CloudFormation review covers improved troubleshooting, drift-aware change sets, stack refactoring, StackSets, the CloudFormation language server, and IaC MCP tooling. - AWS Community Day Romania will take place April 23–24, 2026, featuring technical sessions, AWS experts, and networking opportunities. Together, these updates point toward more private, observable, resilient, and AI-assisted AWS operations. Teams should evaluate the new capabilities against their security, scalability, and automation needs, particularly Bedrock agent tooling, S3 encryption updates, PrivateLink connectivity, and resilience testing.

Read original(opens in new tab)
tossOriginal article

The story of how I destroyed (opens in new tab)

Toss Payments modernized its inherited legacy infrastructure by building an OpenStack-based private cloud to operate alongside public cloud providers in an Active-Active hybrid configuration. By overcoming extreme technical debt—including servers burdened with nearly 2,000 manual routing entries—the team achieved a cloud-agnostic deployment environment that ensures high availability and cost efficiency. The transformation demonstrates how a small team can successfully implement complex open-source infrastructure through automation and the rigorous technical internalization of Cluster API and OpenStack. ### The Challenge of Legacy Networking - The inherited infrastructure relied on server-side routing rather than network equipment, meaning every server carried its own routing table. - Some legacy servers contained 1,997 individual routing entries, making manual management nearly impossible and preventing efficient scaling. - Initial attempts to solve this via public cloud (AWS) faced limitations, including rising costs due to exchange rates, lack of deep visibility for troubleshooting, and difficulties in disaster recovery (DR) configuration between public and on-premise environments. ### Scaling OpenStack with a Two-Person Team - Despite having only two engineers with no prior OpenStack experience, the team chose the open-source platform to maintain 100% control over the infrastructure. - The team internalized the technology by installing three different versions of OpenStack dozens of times and simulating various failure scenarios. - Automation was prioritized using Ansible and Terraform to manage the lifecycle of VMs and load balancers, enabling new instance creation in under 10 seconds. - Deep technical tuning was applied, such as modifying the source code of the Octavia load balancer to output custom log formats required for their specific monitoring needs. ### High Availability and Monitoring Strategy - To ensure reliability, the team built three independent OpenStack clusters operating in an Active-Active configuration. - This architecture allows for immediate traffic redirection if a specific cluster fails, minimizing the impact on service availability. - A comprehensive monitoring stack was implemented using Zabbix, Prometheus, Mimir, and Grafana to collect and visualize every essential metric across the private cloud. ### Managing Kubernetes with Cluster API - To replicate the convenience of Public Cloud PaaS (like EKS), the team implemented Cluster API to manage the Kubernetes lifecycle. - Cluster API treats Kubernetes clusters themselves as resources within a management cluster, allowing for standardized and rapid deployment across the private environment. - This approach ensures that developers can deploy applications without needing to distinguish between the underlying cloud providers, fulfilling the goal of "cloud-agnostic" infrastructure. ### Practical Recommendation For organizations dealing with massive technical debt or high public cloud costs, the Toss Payments model suggests that a "Private-First" hybrid approach is viable even with limited headcount. The key is to avoid proprietary black-box solutions and instead invest in the technical internalization of open-source tools like OpenStack and Cluster API, backed by a "code-as-infrastructure" philosophy to ensure scalability and reliability.

tossOriginal article

Managing Thousands of API/ (opens in new tab)

Toss Payments manages thousands of API and batch server configurations that handle trillions of won in transactions, where a single typo in a JVM setting can lead to massive financial infrastructure failure. To solve the risks associated with manual "copy-paste" workflows and configuration duplication, the team developed a sophisticated system that treats configuration as code. By implementing layered architectures and dynamic templates, they created a testable, unified environment capable of managing complex hybrid cloud setups with minimal human error. ## Overlay Architecture for Hierarchical Control * The team implemented a layered configuration system consisting of `global`, `cluster`, `phase`, and `application` levels. * Settings are resolved by priority, where lower-level layers override higher-level defaults, allowing servers to inherit common settings while maintaining specific overrides. * This structure allows the team to control environment-specific behaviors, such as disabling canary deployments in development environments, from a single centralized directory. * The directory structure maps files 1:1 to their respective layers, ensuring that naming conventions drive the CI/CD application process. ## Solving Duplication with Template Patterns * Standard YAML overlays often fail when dealing with long strings or arrays, such as `JVM_OPTION`, because changing a single value usually requires redefining the entire block. * To prevent the proliferation of nearly identical environment variables, the team introduced a template pattern using placeholders like `{{MAX_HEAP}}`. * Developers can modify specific parameters at the application layer while the core string remains defined at the global layer, significantly reducing the risk of typos. * This approach ensures that critical settings, like G1GC parameters or heap region sizes, remain consistent across the infrastructure unless explicitly changed. ## Dynamic and Conditional Configuration Logic * The system allows for "evolutionary" configurations where Python scripts can be injected to generate dynamic values, such as random JMX ports or data fetched from remote APIs. * Advanced conditional logic was added to handle complex deployment scenarios, enabling environment variables to change their values automatically based on the target cluster name (e.g., different profiles for AWS vs. IDC). * By treating configuration as a living codebase, the team can adapt to new infrastructure requirements without abandoning their core architectural principles. ## Reliable Batch Processing through Simplicity * For batch operations handling massive settlement volumes, the team prioritized "appropriate technology" and simplicity to minimize failure points. * They chose Jenkins for its low learning curve and reliability, despite its lack of native GitOps support. * To address inconsistencies in manual UI entries and varying Java versions across machines, they standardized the batch infrastructure to ensure that high-stakes financial calculations are executed in a controlled, predictable environment. The most effective way to manage large-scale infrastructure is to transition from static, duplicated configuration files to a dynamic, code-centric system. By combining an overlay architecture for hierarchy and a template pattern for granular changes, organizations can achieve the flexibility needed for hybrid clouds while maintaining the strict safety standards required for financial systems.

microsoft3 min readCurated summary

Dev Box Ready-To-Code Dev Box images template

Microsoft announced Team customizations and imaging for Microsoft Dev Box to make development environments faster to create, more consistent, and easier to maintain. The feature builds on Microsoft’s internal One Engineering System (1ES) “ready to code” environments, already used by more than 35,000 developers. Its central approach is to use reusable templates, automated image builds, and centrally managed improvements to reduce setup time and eliminate environment inconsistencies. ## The Challenge of Large-Scale Development Environments - Large teams often work with enormous repositories, proprietary or legacy tools, and slow build processes. - Many setup steps are shared across teams, but creating and maintaining reusable customizations requires significant effort. - Teams also need flexibility for unique requirements without duplicating large amounts of configuration. - The 1ES approach addresses this through reusable templates with conditional logic and shared modules. ## How 1ES Ready-to-Code Environments Work - Templates define common environment requirements while allowing teams to customize: - Repositories to clone - Build configurations - Default tools - Additional setup tasks - Image artifacts, implemented as scripts or CI/CD tasks, install and configure environment components. - Azure Managed Identity provides secure access to required repositories and assets. - Azure Bicep modules hide template complexity while allowing reusable infrastructure definitions. - Azure Pipelines manage image creation and refreshes, making troubleshooting familiar to Microsoft developers. ## Benefits of Team Customizations - **Security:** Images use Managed Identity, approved sources, and validated artifacts. - **Performance:** Dev Drive and security settings are preconfigured for development workloads. - **Consistency:** Smart defaults reduce configuration differences and “works on my machine” problems. - **Flexibility:** Teams can tailor repositories, tools, builds, and other customizations. - **Maintainability:** Shared Bicep modules allow central teams to deliver improvements broadly. - **Easy updates:** Automated Azure Pipelines simplify image refreshes and maintenance. ## Testing and Controlled Releases - 1ES creates hundreds of Ready-to-Code images for Microsoft teams. - Template pull requests are tested with a small set of images covering core features. - Before each template release, larger test runs simulate real customer image definitions. - Releases are phased through internal dogfooding before wider deployment. - Bicep Module Registry tags distinguish release phases and support targeted hotfixes. ## Community Sample Template Microsoft is sharing a simplified version of its internal approach using Azure Image Builder. The sample demonstrates how to create Ready-to-Code images for open-source repositories such as MSBuildSdks, eShop, and Axios. Key components include: - `README.md` with setup instructions - Bicep image definitions for different repository types - `devbox-image` as the main reusable module - `build_images.yml` for Azure DevOps image builds - PowerShell artifacts for image configuration ## Capabilities of the Sample - Clones repositories, restores packages, builds projects, and creates shortcuts. - Supports MSBuild and .NET projects with automatic SDK installation. - Configures repository and artifact authentication through Managed Identity. - Sets up Dev Drive automatically. - Uses a Visual Studio-based Azure Marketplace image by default. - Installs tools such as VS Code, Visual Studio extensions, Git, Sysinternals, WinGet, and the Azure Artifacts Credential Provider. - Applies developer-focused Windows and Defender settings. - Supports image chaining, Compute Gallery publishing, and configurable build VM sizing. The recommended approach is to adopt a reusable, templated image definition rather than maintaining one-off setup scripts. Teams can start with Microsoft’s sample and extend it for their repositories, while Team customizations evolves toward bringing the 1ES capabilities directly into Microsoft Dev Box.

Read original(opens in new tab)