Cloud Storage | Techlist.io

gitlabMar 31, 20262 min readCurated summary

Changes to packages.gitlab.com: What you need to know

GitLab is migrating `packages.gitlab.com` to a new package hosting system while keeping the base domain unchanged. Existing configurations will continue working through rewrite rules until September 30, 2026, but old URL formats, GPG key locations, network requirements, and download paths must be updated before then. New installations already use the updated formats. ## Timeline and Required Updates - The legacy PackageCloud system and UI shut down on March 31, 2026. - Backward-compatible URL rewrites will be removed by the end of September 2026. - Existing users should: - Re-run the latest DEB or RPM installation script, or update repository configuration manually. - Change the GPG key URL to `https://packages.gitlab.com/gpgkey/gpg.key`. - Allow `https://storage.googleapis.com/packages-ops` through firewalls and proxies. - Update repository mirrors and direct-download automation. - Change GitLab Runner RPM references from `noarch` to `x86_64`. ## DEB Repository URL Changes - `gitlab/*` repositories such as `gitlab-ee` and `gitlab-ce` now include the distribution codename in the URL path. - Example: - Old: `.../ubuntu/` - New: `.../ubuntu/jammy` - This produces standard Debian paths containing `dists/jammy` and `pool`. - Re-running the installation script is the simplest migration method: - DEB: `script.deb.sh` - RPM: `script.rpm.sh` - `runner/*` DEB repositories, including `runner/gitlab-runner`, are unchanged. ## GPG Keys and Installation Scripts - Replace: - `https://packages.gitlab.com/gpg.key` - With: - `https://packages.gitlab.com/gpgkey/gpg.key` - Previously saved installation scripts should not be reused; download current versions from GitLab. ## Direct Package Downloads - The old PackageCloud UI used URLs ending in `download.deb` or `download.rpm`. - The new UI links directly to the actual package paths. - Automation that scrapes the old UI or constructs these legacy URLs must be changed to use the new paths or standard package-manager repositories. ## GitLab Runner RPM Packages - Runner RPM packages that were previously under a `noarch` path have moved to `x86_64`. - This affects RPM-based systems such as EL8 and EL9. - Debian-based Runner packages are unaffected. - Update scripts or configurations that reference the old `noarch/Packages` paths. ## Firewall and Network Requirements - Package downloads now redirect to Google Cloud Storage instead of AWS CloudFront. - Permit both `packages.gitlab.com` and `storage.googleapis.com/packages-ops`. - Missing allowlist entries can cause HTTP 503 errors or connection timeouts. Existing installations should be migrated well before September 30, 2026. For new setups, following GitLab’s current installation documentation handles the changes automatically.

Read original

Cloud Storage Gitlab Gpg Deb+4

awsMar 13, 20263 min readCurated summary

Twenty years of Amazon S3 and building what’s next | Amazon Web Services

Amazon S3 began in 2006 as a simple web service for storing and retrieving objects, but its emphasis on security, durability, availability, performance, and elasticity enabled it to become foundational infrastructure. Over two decades, it scaled from roughly one petabyte to hundreds of exabytes while preserving API compatibility, reducing prices, and expanding beyond object storage. Amazon’s long-term vision is for S3 to serve as a universal foundation for data, analytics, and AI workloads. ## The Original S3 Philosophy - S3 introduced two basic operations: - `PUT` to store an object - `GET` to retrieve it - The service abstracted away complex infrastructure so developers could focus on applications. - Its five enduring design principles are: - **Security:** Data is protected by default. - **Durability:** Designed for 11 nines of durability, with a lossless operating model. - **Availability:** Failure is assumed and handled throughout the system. - **Performance:** Storage capacity can grow without degrading performance. - **Elasticity:** Capacity expands and contracts automatically. ## From One Petabyte to Hundreds of Exabytes - At launch, S3 had approximately: - One petabyte of capacity - 400 storage nodes across 15 racks and three data centers - 15 Gbps of bandwidth - A maximum object size of 5 GB - A price of $0.15 per GB - Today, S3: - Stores more than 500 trillion objects. - Serves over 200 million requests per second. - Operates across 123 Availability Zones in 39 AWS Regions. - Supports objects up to 50 TB—10,000 times larger than the original limit. - Storage prices have fallen by roughly 85%, to slightly above 2 cents per GB. - S3 Intelligent-Tiering has saved customers more than $6 billion in storage costs. - The S3 API has become an industry standard, with many other storage systems offering compatible interfaces. ## Backward Compatibility and Long-Term Reliability - Code written against S3 in 2006 still works without modification. - AWS has repeatedly replaced disks, storage systems, and request-processing code while preserving access to older data. - This compatibility reflects S3’s goal of remaining infrastructure that “just works” despite continuous internal change. ## Engineering for Durability and Scale - Microservices continuously inspect every byte across the fleet. - Auditor services detect degradation and automatically trigger repair and re-replication. - Automated formal methods mathematically verify correctness in areas such as: - The index subsystem - Cross-Region replication - Access policies - AWS has progressively rewritten performance-critical components in Rust over the past eight years. - Rust improves performance while preventing memory-safety bugs and other classes of errors at compile time. - S3 follows the principle that scale should improve the service: larger, more distributed workloads become increasingly decorrelated, improving reliability for all customers. ## S3 as a Foundation for Data and AI Amazon’s future vision is for customers to store data once in S3 and work with it directly, avoiding costly copies and specialized systems. - **S3 Tables** provides managed Apache Iceberg tables with automated maintenance to improve query performance and reduce storage costs. - **S3 Vectors** supports semantic search and retrieval-augmented generation, with up to 2 billion vectors per index and sub-100 ms query latency. - Within five months of launch, customers created over 250,000 indexes, ingested more than 40 billion vectors, and executed over 1 billion queries. - **S3 Metadata** enables centralized, faster data discovery without recursively listing large buckets. These additions extend S3 from inexpensive object storage into a broader platform for analytics, search, and AI while retaining its scale and cost advantages.

Read original

Cloud Storage Database Design Rust API+4