techlist.io
KR EN

amazon-s3

6 posts

aws

Announcing replication support and Intelligent-Tiering for Amazon S3 Tables | AWS News Blog (opens in new tab)

AWS has expanded the capabilities of Amazon S3 Tables by introducing Intelligent-Tiering for automated cost optimization and cross-region replication for enhanced data availability. These updates address the operational overhead of managing large-scale Apache Iceberg datasets by automating storage lifecycle management and simplifying the architecture required for global data distribution. By integrating these features, organizations can reduce storage costs without manual intervention while ensuring consistent data access across multiple AWS Regions and accounts. ### Cost Optimization with S3 Tables Intelligent-Tiering This feature automatically shifts data between storage tiers based on access frequency to maximize cost efficiency without impacting application performance. * The system utilizes three low-latency tiers: Frequent Access, Infrequent Access (offering 40% lower costs), and Archive Instant Access (offering 68% lower costs than Infrequent Access). * Data transitions are automated, moving to Infrequent Access after 30 days of inactivity and to Archive Instant Access after 90 days. * Automated table maintenance tasks, such as compaction and snapshot expiration, are optimized to skip colder files; for example, compaction only processes data in the Frequent Access tier to minimize unnecessary compute and storage costs. * Users can configure Intelligent-Tiering as the default storage class at the table bucket level using the AWS CLI commands `put-table-bucket-storage-class` and `get-table-bucket-storage-class`. ### Cross-Region and Cross-Account Replication New replication support allows users to maintain synchronized, read-only replicas of their S3 Tables across different geographic locations and ownership boundaries. * Replication maintains chronological consistency and preserves parent-child snapshot relationships, ensuring that replicas remain identical to the source for query purposes. * Replica tables are typically updated within minutes of changes to the source table and support independent encryption and retention policies to meet specific regional compliance requirements. * The service eliminates the need for complex, custom-built architectures to track metadata transformations or manually sync objects between Iceberg tables. * This functionality is primarily designed to reduce query latency for geographically distributed teams and provide robust data protection for disaster recovery scenarios. ### Practical Implementation To maximize the benefits of these new features, organizations should consider setting Intelligent-Tiering as the default storage class at the bucket level for all new datasets to ensure immediate cost savings. For global operations, setting up read-only replicas in regions closest to end-users will significantly improve query performance for analytics tools like Amazon Athena and Amazon SageMaker.

amazon-s3amazon-s3-tablesapache-icebergdata-replication+3
aws

Amazon S3 Storage Lens adds performance metrics, support for billions of prefixes, and export to S3 Tables | AWS News Blog (opens in new tab)

Amazon S3 Storage Lens has introduced three significant updates designed to provide deeper visibility into storage performance and usage patterns at scale. By adding dedicated performance metrics, support for billions of prefixes, and direct export capabilities to Amazon S3 Tables, AWS enables organizations to better optimize application latency and storage costs. These enhancements allow for more granular data-driven decisions across entire AWS organizations or specific high-performance workloads. ## Enhanced Performance Metric Categories The update introduces eight new performance-related metric categories available through the S3 Storage Lens advanced tier. These metrics are designed to pinpoint specific architectural bottlenecks that could impact application speed. * **Request and Storage Distributions:** New metrics track the distribution of read/write request sizes and object sizes, helping identify small-object patterns that might be better suited for Amazon S3 Express One Zone. * **Error and Latency Tracking:** Users can now monitor concurrent PUT 503 errors to identify throttling and analyze FirstByteLatency and TotalRequestLatency to measure end-to-end request performance. * **Data Transfer Efficiency:** Metrics for cross-Region data transfer help identify high-cost or high-latency data access patterns, suggesting where compute resources should be co-located with storage. * **Access Patterns:** Tracking unique objects accessed per day identifies "hot" datasets that could benefit from higher-performance storage tiers or caching solutions. ## Support for Billions of Prefixes S3 Storage Lens has expanded its analytical scale to support the monitoring of billions of prefixes. This allows organizations with massive, complex data structures to maintain granular visibility without sacrificing performance or detail. * **Granular Visibility:** Users can drill down into massive datasets to find specific prefixes causing performance degradation or cost spikes. * **Scalable Analysis:** This expansion ensures that even the largest data lakes can be monitored at a level of detail previously limited to smaller buckets. ## Integration with Amazon S3 Tables The service now supports direct export of storage metrics to Amazon S3 Tables, a feature optimized for high-performance analytics. This integration streamlines the workflow for administrators who need to perform complex queries on their storage metadata. * **Analytical Readiness:** Exporting to S3 Tables makes it easier to use SQL-based tools to query storage trends and performance over time. * **Automation:** This capability allows for the creation of automated reporting pipelines that can handle the massive volume of data generated by prefix-level monitoring. To take full advantage of these features, users should enable the S3 Storage Lens advanced tier and configure prefix-level monitoring for buckets containing mission-critical or high-throughput data. Organizations experiencing latency issues should specifically review the new request size distribution metrics to determine if batching objects or migrating to S3 Express One Zone would improve performance.

amazon-s3amazon-s3-tablesstorage-optimizationcloud-monitoring+2
aws

New capabilities to optimize costs and improve scalability on Amazon RDS for SQL Server and Oracle | AWS News Blog (opens in new tab)

Amazon Web Services has introduced several key updates to Amazon RDS for SQL Server and Oracle designed to reduce operational overhead and licensing expenses. By integrating SQL Server Developer Edition and high-performance M7i/R7i instances with customizable CPU options, organizations can now scale their development and production environments more efficiently. These enhancements allow teams to mirror production features in testing environments and right-size resource allocation without the financial burden of traditional enterprise licensing. ### SQL Server Developer Edition for Non-Production Workloads * Amazon RDS now supports SQL Server Developer Edition, providing the full feature set of the Enterprise Edition at no licensing cost for development and testing environments. * The update allows for consistency across the database lifecycle, as developers can utilize RDS features such as automated backups, software updates, and encryption while testing Enterprise-level functionalities. * To deploy, users upload SQL Server binary files to Amazon S3; existing data can be migrated from Standard or Enterprise editions using native backup and restore operations. ### Performance and Licensing Optimization via M7i/R7i Instances * RDS for SQL Server now supports M7i and R7i instance types, which offer up to 55% lower costs compared to previous generation instances. * The billing structure for these instances provides improved transparency by separating Amazon RDS DB instance costs from software licensing fees. * The "Optimize CPU" capability allows users to customize the number of vCPUs on license-included instances, enabling them to reduce licensing costs while maintaining the high memory and storage performance of larger instance classes. ### Expanded Storage and Scalability for RDS * The updates include expanded storage capabilities for both Amazon RDS for Oracle and RDS for SQL Server to accommodate growing data requirements. * These enhancements are designed to support a wide range of workloads, providing flexibility for diverse compute and storage needs across development, testing, and production tiers. These updates represent a significant shift toward providing more granular control over database expenditures and performance. For organizations running heavy SQL Server or Oracle workloads, leveraging the Developer Edition for non-production tasks and migrating to M7i/R7i instances with optimized CPU settings can drastically reduce total cost of ownership while maintaining high scalability.

amazon-s3amazon-rdssql-serveroracle-database+3
aws

Amazon S3 Vectors now generally available with increased scale and performance | AWS News Blog (opens in new tab)

Amazon S3 Vectors has reached general availability, establishing the first cloud object storage service with native support for storing and querying vector data. This serverless solution allows organizations to reduce total ownership costs by up to 90% compared to specialized vector database solutions while providing the performance required for production-grade AI applications. By integrating vector capabilities directly into S3, AWS enables a simplified architecture for retrieval-augmented generation (RAG), semantic search, and multi-agent workflows. ### Massive Scale and Index Consolidation The move to general availability introduces a significant increase in data capacity, allowing users to manage massive datasets without complex infrastructure workarounds. * **Increased Index Limits:** Each index can now store and search across up to 2 billion vectors, representing a 40x increase from the 50 million limit during the preview phase. * **Bucket Capacity:** A single vector bucket can now scale to house up to 20 trillion vectors. * **Simplified Architecture:** The increased scale per index removes the need for developers to shard data across multiple indexes or implement custom query federation logic. ### Performance and Latency Optimizations The service has been tuned to meet the low-latency requirements of interactive applications like conversational AI and real-time inference. * **Query Response Times:** Frequent queries now achieve latencies of approximately 100ms or less, while infrequent queries consistently return results in under one second. * **Enhanced Retrieval:** Users can now retrieve up to 100 search results per query (increased from 30), providing broader context for RAG applications. * **Write Throughput:** The system supports up to 1,000 PUT transactions per second for streaming single-vector updates, ensuring new data is immediately searchable. ### Serverless Efficiency and Ecosystem Integration S3 Vectors functions as a fully serverless offering, eliminating the need to provision or manage underlying instances while paying only for active storage and queries. * **Amazon Bedrock Integration:** It is now generally available as a vector storage engine for Bedrock Knowledge Bases, facilitating the building of RAG applications. * **OpenSearch Support:** Integration with Amazon OpenSearch allows users to utilize S3 Vectors for storage while leveraging OpenSearch for advanced analytics and search features. * **Expanded Footprint:** The service is now available in 14 AWS Regions, up from five during the preview period. With its massive scale and 90% cost reduction, S3 Vectors is a primary candidate for organizations looking to move AI prototypes into production. Developers should consider migrating high-volume vector workloads to S3 Vectors to benefit from the serverless operational model and the native integration with the broader AWS AI stack.

amazon-s3airagvector-db+5
coupang

Optimizing operational costs through cloud (opens in new tab)

Coupang’s Finance and Engineering teams collaborated to optimize cloud expenditures by focusing on resource efficiency and the company's "Hate Waste" leadership principle. Through a dedicated optimization project team and the implementation of data-driven analytics, the company successfully reduced on-demand costs by millions of dollars without compromising business growth. This initiative transformed cloud management from a reactive expense into a proactive engineering culture centered on financial accountability and technical efficiency. ### Forming the Optimization Project Team * A specialized team consisting of Cloud Infrastructure Engineers and Technical Program Managers (TPMs) was established to bridge the gap between finance and engineering. * The project team focused on educating domain teams about the variable cost model of cloud services, moving away from a fixed-cost mindset. * Technical experts helped domain teams identify opportunities to use cost-efficient technologies, such as ARM-based AWS Graviton processors and AWS Spot Instances for data processing. * The initiative established clear ownership, ensuring that each domain team understood and managed their specific cloud resource usage. ### Analytics and Dashboards for Visibility * Engineers developed custom dashboards using Amazon Athena to process Amazon CloudWatch data, providing deep insights into resource performance. * The team utilized AWS Cost & Usage Reports (CUR) within internal Business Intelligence (BI) tools to provide granular visibility into spending patterns. * Finance teams worked alongside engineers to align technical roadmaps with monthly and quarterly budget goals, making cost management a shared responsibility. ### Strategies for Usage and Cost Reduction * **Spend Less (Usage Reduction):** Coupang implemented automation to ensure that non-production environment resources were only active when needed, resulting in a 25% cost saving for those environments. * **Pay Less (Right-sizing):** The team analyzed usage patterns to manually identify and decommission unused EC2 resources across all domain teams. * **Instance and Storage Optimization:** The project prioritized migrating workloads to the latest instance generations and optimizing Amazon S3 storage structures to reduce costs for data at rest. To achieve sustainable cloud efficiency, organizations should move beyond simple monitoring and foster an engineering culture where resource management is a core technical discipline. Prioritizing automated resource scheduling and adopting modern, high-efficiency hardware like Graviton instances are essential steps for any large-scale cloud operation looking to maximize its return on investment.

amazon-s3awsamazon-ec2cloud-cost-optimization+3
coupang

Cloud expenditure optimization for cost efficiency | by Coupang Engineering | Coupang Engineering Blog | Medium (opens in new tab)

Coupang addressed rising cloud costs by establishing a cross-functional Central team to bridge the gap between engineering usage and financial accountability. Through a data-driven approach involving custom analytics and automated resource management, the company successfully reduced on-demand expenditure by millions of dollars. This initiative demonstrates that aligning technical infrastructure with financial governance is essential for maintaining growth without unnecessary waste. **The Central Team and Data-Driven Governance** * Coupang formed a specialized Central team consisting of infrastructure engineers and technical program managers to identify efficiency opportunities across the organization. * The team developed custom BI dashboards utilizing Amazon CloudWatch, AWS Cost and Usage Reports (CUR), and Amazon Athena to provide domain teams with actionable insights into their spending. * The finance department partnered with engineering to enforce strict budget compliance, ensuring that domain teams managed their resources within assigned monthly and quarterly limits. **Strategies for Spending and Paying Less** * The company implemented "Spending Less" strategies by automating the launch of resources in non-production environments only when needed, resulting in a 25% cost reduction for those areas. * "Paying Less" initiatives focused on rightsizing, where the Central team worked with domain owners to manually identify and eliminate unutilized or underutilized EC2 resources. * Workloads were migrated to more efficient hardware and pricing models, specifically leveraging ARM-based AWS Graviton processors and AWS Spot Instances for data processing and storage. **Targeted Infrastructure Optimization** * Engineering teams focused on instance generation alignment, ensuring that services were running on the most cost-effective hardware generations available. * Storage costs were reduced by optimizing Amazon S3 structures at rest, improving how data is organized and stored. * The team refined Amazon EMR (Elastic MapReduce) configurations to enhance processing efficiency, significantly lowering the cost of large-scale data analysis. To achieve sustainable cloud efficiency, engineering organizations should move beyond viewing cloud costs as a purely financial concern and instead treat resource management as a core technical metric. By integrating financial accountability directly into the engineering workflow through shared analytics and automated resource controls, companies can foster a culture of efficiency that supports long-term scalability.

amazon-s3awscloud-cost-optimizationfinops+4