Amazon S3 | Techlist.io

awsDec 2, 2025Original article

New capabilities to optimize costs and improve scalability on Amazon RDS for SQL Server and Oracle (opens in new tab)

Amazon Web Services has introduced several key updates to Amazon RDS for SQL Server and Oracle designed to reduce operational overhead and licensing expenses. By integrating SQL Server Developer Edition and high-performance M7i/R7i instances with customizable CPU options, organizations can now scale their development and production environments more efficiently. These enhancements allow teams to mirror production features in testing environments and right-size resource allocation without the financial burden of traditional enterprise licensing. ### SQL Server Developer Edition for Non-Production Workloads * Amazon RDS now supports SQL Server Developer Edition, providing the full feature set of the Enterprise Edition at no licensing cost for development and testing environments. * The update allows for consistency across the database lifecycle, as developers can utilize RDS features such as automated backups, software updates, and encryption while testing Enterprise-level functionalities. * To deploy, users upload SQL Server binary files to Amazon S3; existing data can be migrated from Standard or Enterprise editions using native backup and restore operations. ### Performance and Licensing Optimization via M7i/R7i Instances * RDS for SQL Server now supports M7i and R7i instance types, which offer up to 55% lower costs compared to previous generation instances. * The billing structure for these instances provides improved transparency by separating Amazon RDS DB instance costs from software licensing fees. * The "Optimize CPU" capability allows users to customize the number of vCPUs on license-included instances, enabling them to reduce licensing costs while maintaining the high memory and storage performance of larger instance classes. ### Expanded Storage and Scalability for RDS * The updates include expanded storage capabilities for both Amazon RDS for Oracle and RDS for SQL Server to accommodate growing data requirements. * These enhancements are designed to support a wide range of workloads, providing flexibility for diverse compute and storage needs across development, testing, and production tiers. These updates represent a significant shift toward providing more granular control over database expenditures and performance. For organizations running heavy SQL Server or Oracle workloads, leveraging the Developer Edition for non-production tasks and migrating to M7i/R7i instances with optimized CPU settings can drastically reduce total cost of ownership while maintaining high scalability.

Amazon S3 Amazon Rds Cost Optimization Cloud Databases+3

awsDec 2, 2025Original article

Amazon CloudWatch introduces unified data management and analytics for operations, security, and compliance (opens in new tab)

Amazon CloudWatch has evolved into a unified platform for managing operational, security, and compliance log data, significantly reducing the need for redundant data stores and complex ETL pipelines. By standardizing ingestion through industry-standard formats like OCSF and OpenTelemetry, the service enables seamless cross-source analytics while lowering operational overhead and storage costs. This update allows organizations to move away from fragmented data silos toward a centralized, Iceberg-compatible architecture for deeper technical and business insights. **Data Ingestion and Schema Normalization** * Automatically collects AWS-vended logs across accounts and regions via AWS Organizations, including CloudTrail, VPC Flow Logs, WAF access logs, and Route 53 resolver logs. * Includes pre-built connectors for a wide range of third-party sources, such as endpoint security (CrowdStrike, SentinelOne), identity providers (Okta, Entra ID), and network security (Zscaler, Palo Alto Networks). * Utilizes managed Open Cybersecurity Schema Framework (OCSF) and OpenTelemetry (OTel) conversion to ensure data consistency across disparate sources. * Provides built-in processors, such as Grok for custom parsing and field-level operations, to transform and manipulate strings during the ingestion phase. **Unified Architecture and Cost Optimization** * Consolidates log management into a single service with built-in governance, eliminating the need to store and maintain duplicate copies of data across different tools. * Introduces Apache Iceberg-compatible access via Amazon S3 Tables, allowing data to be queried in place by external tools. * Removes the requirement for complex ETL pipelines by providing a unified data store that is accessible to Amazon Athena, Amazon SageMaker Unified Studio, and other Iceberg-compatible analytics engines. **Advanced Analytics and Discovery Tools** * Supports multiple query interfaces, allowing users to interact with logs using natural language, SQL, LogsQL, or PPL (Piped Processing Language). * The new "Facets" interface enables intuitive filtering by application, account, region, and log type, featuring intelligent parameter inference for cross-account queries. * Enables the correlation of operational logs with business data from third-party tools like ServiceNow CMDB or GitHub to provide a more comprehensive view of organizational health. Organizations should leverage these unified management features to consolidate their security and operational monitoring into a single source of truth. By adopting OCSF normalization and the new S3 Tables integration, teams can reduce the technical debt associated with managing multiple log silos while improving their ability to run cross-functional analytics.

Amazon S3 Log Management Apache Iceberg Data Analytics+4

awsDec 2, 2025Original article

Amazon S3 Vectors now generally available with increased scale and performance (opens in new tab)

Amazon S3 Vectors has reached general availability, establishing the first cloud object storage service with native support for storing and querying vector data. This serverless solution allows organizations to reduce total ownership costs by up to 90% compared to specialized vector database solutions while providing the performance required for production-grade AI applications. By integrating vector capabilities directly into S3, AWS enables a simplified architecture for retrieval-augmented generation (RAG), semantic search, and multi-agent workflows. ### Massive Scale and Index Consolidation The move to general availability introduces a significant increase in data capacity, allowing users to manage massive datasets without complex infrastructure workarounds. * **Increased Index Limits:** Each index can now store and search across up to 2 billion vectors, representing a 40x increase from the 50 million limit during the preview phase. * **Bucket Capacity:** A single vector bucket can now scale to house up to 20 trillion vectors. * **Simplified Architecture:** The increased scale per index removes the need for developers to shard data across multiple indexes or implement custom query federation logic. ### Performance and Latency Optimizations The service has been tuned to meet the low-latency requirements of interactive applications like conversational AI and real-time inference. * **Query Response Times:** Frequent queries now achieve latencies of approximately 100ms or less, while infrequent queries consistently return results in under one second. * **Enhanced Retrieval:** Users can now retrieve up to 100 search results per query (increased from 30), providing broader context for RAG applications. * **Write Throughput:** The system supports up to 1,000 PUT transactions per second for streaming single-vector updates, ensuring new data is immediately searchable. ### Serverless Efficiency and Ecosystem Integration S3 Vectors functions as a fully serverless offering, eliminating the need to provision or manage underlying instances while paying only for active storage and queries. * **Amazon Bedrock Integration:** It is now generally available as a vector storage engine for Bedrock Knowledge Bases, facilitating the building of RAG applications. * **OpenSearch Support:** Integration with Amazon OpenSearch allows users to utilize S3 Vectors for storage while leveraging OpenSearch for advanced analytics and search features. * **Expanded Footprint:** The service is now available in 14 AWS Regions, up from five during the preview period. With its massive scale and 90% cost reduction, S3 Vectors is a primary candidate for organizations looking to move AI prototypes into production. Developers should consider migrating high-volume vector workloads to S3 Vectors to benefit from the serverless operational model and the native integration with the broader AWS AI stack.

Amazon S3 AI Amazon Bedrock RAG+4

figmaMay 28, 20254 min readCurated summary

From Multi-Day Latency to Near Real-Time Insights: Figma’s Data Pipeline Upgrade | Figma Blog

Figma replaced a daily full-table export system that could take hours or days with an incremental synchronization pipeline designed for near real-time analytics. The new architecture combines database snapshots, change data capture (CDC), and Snowflake merge logic to transfer only recent changes. By building the system in-house, Figma gained greater flexibility, lower projected costs, and a design that can scale with continued growth. ## Why the Legacy Pipeline Failed - Since 2020, a daily cron job ran `SELECT * FROM <TABLE>`, exported results to S3, and loaded them into Snowflake. - As Figma’s tables and insert volume grew: - Daily syncs reached roughly six hours by 2023. - The largest tables took several days or longer. - Additional database replicas were required for exports. - Replica maintenance cost millions of dollars annually. - The delays limited access to timely company KPIs and analytical insights. ## Choosing Incremental Synchronization Figma evaluated three options: - Continue using the legacy process, which was increasingly expensive and too slow. - Add parallelism, which might improve throughput temporarily but would not scale sustainably. - Rebuild the synchronization system around incremental updates. Incremental synchronization transfers only new and changed records instead of repeatedly copying entire tables, reducing data movement, processing time, and infrastructure usage. ## Buy vs. Build Figma decided to build the pipeline internally because available proprietary tools did not meet its requirements. - **Flexibility:** Generic SQL tools did not take advantage of capabilities such as Amazon RDS for PostgreSQL snapshot exports. - **Cost:** Commercial solutions were projected to cost five to ten times more than an in-house implementation. - **Scale:** Building internally allowed Figma to optimize the system for its infrastructure and adapt it as the company grows. ## Pipeline Components The bespoke system combines several lower-level technologies: - **Snapshots:** Amazon RDS exports initial table copies to S3. - **Change data capture:** Kafka Connect streams database changes through Amazon MSK. - **Warehouse ingestion:** A Snowflake Connector loads CDC events into Snowflake. - **Incremental merging:** Custom Snowflake stored procedures and scheduled tasks merge changes into base tables. ## Architecture Principles The redesign was guided by four goals: - Reduce end-to-end synchronization latency. - Control costs as data volume increases. - Meet regulatory and compliance requirements. - Preserve data accuracy, completeness, consistency, and trustworthiness. The resulting architecture uses two workflows: a bootstrap workflow for onboarding tables and a validation workflow for checking data correctness. ## Bootstrap Workflow The automated onboarding process includes: - The CDC service begins capturing the new Postgres table and publishes events to a per-table Kafka topic. - Amazon RDS exports the latest database snapshot to S3. - Snowflake’s `COPY INTO <table>` loads the snapshot into a per-entity base table. - An MSK Connect Snowflake Sink Connector streams Kafka events into a separate CDC table, with offsets arranged so changes before the snapshot timestamp are retained. - A scheduled Snowflake task runs a custom `MERGE` procedure to combine the snapshot and CDC data. - Once the process catches up with current changes, Figma creates a lightweight user-facing view over the base table. ## Zero-Downtime Re-Bootstrapping - Bootstrap artifacts are versioned, while the final user-facing view remains stable. - New versions can be built in parallel without interrupting queries. - Promotion is completed through an atomic view update. - This supports schema evolution and other situations requiring a fresh bootstrap without downtime. ## Data Validation - Even well-designed pipelines can suffer corruption from partial failures, configuration errors, software bugs, or unexpected source-data anomalies. - Figma therefore added a validation workflow to verify correctness as data moves through snapshot exports, CDC capture, and incremental merging. Figma’s experience shows that incremental synchronization is a more sustainable alternative to repeated full-table exports. Combining managed infrastructure with custom orchestration can deliver lower latency, better cost control, and stronger operational flexibility than a one-size-fits-all commercial pipeline.

Read original

Amazon S3 PostgreSQL Apache Kafka Data Pipeline+4

coupangNov 15, 2024Original article

Optimizing Operating Costs through Cloud Service (opens in new tab)

Coupang’s Finance and Engineering teams collaborated to optimize cloud expenditures by focusing on resource efficiency and the company's "Hate Waste" leadership principle. Through a dedicated optimization project team and the implementation of data-driven analytics, the company successfully reduced on-demand costs by millions of dollars without compromising business growth. This initiative transformed cloud management from a reactive expense into a proactive engineering culture centered on financial accountability and technical efficiency. ### Forming the Optimization Project Team * A specialized team consisting of Cloud Infrastructure Engineers and Technical Program Managers (TPMs) was established to bridge the gap between finance and engineering. * The project team focused on educating domain teams about the variable cost model of cloud services, moving away from a fixed-cost mindset. * Technical experts helped domain teams identify opportunities to use cost-efficient technologies, such as ARM-based AWS Graviton processors and AWS Spot Instances for data processing. * The initiative established clear ownership, ensuring that each domain team understood and managed their specific cloud resource usage. ### Analytics and Dashboards for Visibility * Engineers developed custom dashboards using Amazon Athena to process Amazon CloudWatch data, providing deep insights into resource performance. * The team utilized AWS Cost & Usage Reports (CUR) within internal Business Intelligence (BI) tools to provide granular visibility into spending patterns. * Finance teams worked alongside engineers to align technical roadmaps with monthly and quarterly budget goals, making cost management a shared responsibility. ### Strategies for Usage and Cost Reduction * **Spend Less (Usage Reduction):** Coupang implemented automation to ensure that non-production environment resources were only active when needed, resulting in a 25% cost saving for those environments. * **Pay Less (Right-sizing):** The team analyzed usage patterns to manually identify and decommission unused EC2 resources across all domain teams. * **Instance and Storage Optimization:** The project prioritized migrating workloads to the latest instance generations and optimizing Amazon S3 storage structures to reduce costs for data at rest. To achieve sustainable cloud efficiency, organizations should move beyond simple monitoring and foster an engineering culture where resource management is a core technical discipline. Prioritizing automated resource scheduling and adopting modern, high-efficiency hardware like Graviton instances are essential steps for any large-scale cloud operation looking to maximize its return on investment.

Amazon S3 AWS Amazon Ec2 Finops+3

coupangNov 14, 2024Original article

Cloud expenditure optimization for cost efficiency (opens in new tab)

Coupang addressed rising cloud costs by establishing a cross-functional Central team to bridge the gap between engineering usage and financial accountability. Through a data-driven approach involving custom analytics and automated resource management, the company successfully reduced on-demand expenditure by millions of dollars. This initiative demonstrates that aligning technical infrastructure with financial governance is essential for maintaining growth without unnecessary waste. **The Central Team and Data-Driven Governance** * Coupang formed a specialized Central team consisting of infrastructure engineers and technical program managers to identify efficiency opportunities across the organization. * The team developed custom BI dashboards utilizing Amazon CloudWatch, AWS Cost and Usage Reports (CUR), and Amazon Athena to provide domain teams with actionable insights into their spending. * The finance department partnered with engineering to enforce strict budget compliance, ensuring that domain teams managed their resources within assigned monthly and quarterly limits. **Strategies for Spending and Paying Less** * The company implemented "Spending Less" strategies by automating the launch of resources in non-production environments only when needed, resulting in a 25% cost reduction for those areas. * "Paying Less" initiatives focused on rightsizing, where the Central team worked with domain owners to manually identify and eliminate unutilized or underutilized EC2 resources. * Workloads were migrated to more efficient hardware and pricing models, specifically leveraging ARM-based AWS Graviton processors and AWS Spot Instances for data processing and storage. **Targeted Infrastructure Optimization** * Engineering teams focused on instance generation alignment, ensuring that services were running on the most cost-effective hardware generations available. * Storage costs were reduced by optimizing Amazon S3 structures at rest, improving how data is organized and stored. * The team refined Amazon EMR (Elastic MapReduce) configurations to enhance processing efficiency, significantly lowering the cost of large-scale data analysis. To achieve sustainable cloud efficiency, engineering organizations should move beyond viewing cloud costs as a purely financial concern and instead treat resource management as a core technical metric. By integrating financial accountability directly into the engineering workflow through shared analytics and automated resource controls, companies can foster a culture of efficiency that supports long-term scalability.

Amazon S3 AWS Finops Amazon Athena+4

figmaJul 20, 20202 min readCurated summary

A deep dive on deep search | Figma Blog

Figma’s deep search lets users find files by searching text inside them rather than relying on file names or metadata. Building it required extending infrastructure originally created for Design System Analytics to process `.fig` files stored in Amazon S3. Because analyzing large file trees is expensive, Figma accepted briefly stale results and processed deduplicated changes hourly. ## Deep Search in a Browser-Based Product - Figma’s web-based architecture provides detailed access to files and usage data. - This enables features such as: - Component usage analytics - File-view frequency - Inspection of file structure - Searching content inside files - Deep search builds on the browser’s collaboration and discoverability advantages. ## Reusing Design System Analytics Infrastructure - Design System Analytics already opened recently edited files, retrieved them from storage, and traversed their contents. - Analytics extracted shared-library usage information. - Deep search applies the same general workflow to extract text from Figma files. - The existing file-analyzer worker platform provided support for computationally intensive, periodic processing. ## Regular Search vs. Deep Search - Regular search indexes database metadata, including: - File name - Creator - Folder ID - Team ID - Its pipeline: - Database changes are streamed into a messaging system. - Search indexers retrieve current records. - The metadata is indexed in Elasticsearch. - Deep search cannot rely on database metadata because the actual file contents are stored as `.fig` documents in Amazon S3. - A `.fig` file is represented as a tree of nodes, such as frames, rectangles, vectors, ellipses, and text objects, each with its own properties. ## Managing the Cost of File Analysis - Retrieving and traversing a complete Figma file is significantly more expensive than reading database records. - Files may contain thousands of nodes, and users can trigger saves approximately every 30 seconds. - Re-indexing every save would produce substantial duplicated computation. - Figma therefore: - Deduplicates file changes over one-hour windows. - Sends changed files to file-analyzer workers. - Allows deep-search results to be temporarily stale. - This tradeoff reduces server workload while maintaining useful search functionality. Deep search demonstrates how content-aware features require different infrastructure from conventional metadata search. Periodic, deduplicated processing offers a practical balance between timely results and the high computational cost of analyzing complete design files.

Read original

Amazon S3 Distributed Systems Data Pipeline Elasticsearch+2