Gpu Acceleration | Techlist.io

awsJun 18, 20262 min readCurated summary

Announcing Amazon EC2 G7 instances accelerated by NVIDIA RTX PRO 4500 Blackwell Server Edition GPUs | Amazon Web Services

Amazon EC2 G7 instances are now generally available with NVIDIA RTX PRO 4500 Blackwell Server Edition GPUs and custom sixth-generation Intel Xeon processors. Compared with G6 instances, they provide up to 4.6× higher AI inference performance and 2.1× better graphics performance. AWS positions them for AI inference, rendering, video, virtual desktops, spatial computing, and GPU-accelerated analytics. ## GPU and Performance Improvements - Each GPU provides 32 GB of memory, with up to 256 GB across eight GPUs. - GPU memory capacity is 1.33× higher and bandwidth is 2.45× higher than G6. - GPUs include fifth-generation Tensor Cores and fourth-generation RT Cores. - G7 instances accelerate analytics workloads running on Amazon EMR with Amazon EKS. ## Networking and Storage - Up to 700 Gbps of EFA-enabled networking—seven times the G6 throughput. - Up to 7.6 TB of local NVMe SSD storage keeps large models and datasets close to the GPUs. - Support for NVIDIA GPUDirect P2P and GPUDirect RDMA with EFA enables low-latency GPU communication across GPUs, nodes, and FSx for Lustre. ## Video Processing - Ninth-generation NVENC and sixth-generation NVDEC engines support 4:2:2 encoding and decoding. - They deliver up to 1.5× more concurrent video streams than G6 instances. ## Instance Configurations - Seven instance sizes are available. - Configurations offer up to: - 8 NVIDIA RTX PRO 4500 GPUs - 192 vCPUs - 768 GiB of system memory - 700 Gbps network bandwidth - 7.6 TB local NVMe storage - Detailed instance specifications were listed as “coming soon” in the announcement. ## Software and Availability - AWS provides Deep Learning AMIs and NVIDIA Workstation AMIs with preinstalled drivers. - Amazon EKS users should build AMIs with NVIDIA driver version R595. - Supported operating systems include Amazon Linux, Ubuntu, RHEL, and Windows Server. - NVIDIA integration supports DirectX, Vulkan, and OpenGL. - G7 instances are initially available in US East (Ohio) and US West (Oregon). - Purchasing options include On-Demand, Savings Plans, Spot Instances, and Dedicated Instances for selected sizes. G7 instances are a strong option for GPU-intensive workloads requiring higher inference, graphics, networking, and video performance. Organizations can launch them through the EC2 console and evaluate pricing across the available purchasing models.

Read original

Gpu Acceleration Kubernetes AWS Amazon Ec2+4

metaFeb 24, 20263 min readCurated summary

RCCLX: Innovating GPU communications on AMD platforms

RCCLX is Meta’s open-source enhancement of RCCL for AMD GPUs, integrated with Torchcomms to support portable distributed AI workloads. It introduces Direct Data Access (DDA) and low-precision collectives, targeting communication bottlenecks in inference and training. On AMD MI300X systems, these optimizations deliver lower latency and higher throughput while maintaining acceptable accuracy. ## RCCLX and Torchcomms Integration - RCCLX is based on RCCL and tested on Meta’s internal workloads. - It integrates CTran transport technology for AMD platforms. - CTran enables features such as `AllToAllvDynamic`, a GPU-resident collective; additional CTran capabilities are planned for future releases. - Through Torchcomms, applications can use a common communication API across AMD, NVIDIA, and other backends without major code changes. - RCCLX is intended to achieve feature parity with Meta’s NCCLX backend for NVIDIA systems. ## Direct Data Access for Intra-Node Collectives - LLM inference has two distinct phases: - **Prefill** is compute-bound and generates the model’s key-value cache. - **Decoding** is memory-bound and generates tokens incrementally. - Tensor parallelism can make AllReduce responsible for up to 30% of end-to-end latency. - RCCLX introduces two DDA algorithms: - **DDA flat** lets each rank directly read other ranks’ memory and perform local reductions. It reduces latency from O(N) to O(1) for small messages by increasing data exchange from O(n) to O(n²). - **DDA tree** divides AllReduce into reduce-scatter and all-gather phases, retaining ring-like data movement while reducing latency for somewhat larger messages. - On AMD MI300X GPUs, DDA improves over RCCL by: - 10–50% for decode workloads. - 10–30% for prefill workloads. - Approximately 10% lower time-to-incremental-token. ## Low-Precision Collectives - RCCLX provides optimized low-precision versions of AllReduce, AllGather, AlltoAll, and ReduceScatter. - These target AMD Instinct MI300 and MI350 GPUs and support FP32 and BF16 inputs. - FP8 quantization provides up to 4:1 compression, reducing communication overhead for messages of at least 16 MB. - Parallel peer-to-peer mesh communication uses AMD Infinity Fabric for bandwidth and low latency. - Computation remains in FP32 to improve numerical stability. - Users can enable the feature with: ```bash RCCL_LOW_PRECISION_ENABLE=1 ``` - Internal evaluations showed: - About a 0.3% change on GSM8K accuracy evaluations. - 9–10% lower latency. - Approximately 7% higher throughput. - The current implementation is tuned for single-node deployments. ## Getting Started - Install Torchcomms with the RCCLX backend. - Create an RCCLX communicator through Torchcomms using the `"rcclx"` backend and a HIP device. - Existing Torchcomms operations such as `allreduce` can then run without backend-specific API changes. - Distributed initialization uses standard `torchrun` environment variables such as `MASTER_ADDR`, `MASTER_PORT`, `RANK`, and `WORLD_SIZE`. RCCLX is positioned as a practical way to improve AMD-based AI training and inference without requiring applications to adopt a new communication API. Teams can use DDA for lower inference latency and selectively enable low-precision collectives for higher throughput, while evaluating numerical accuracy for their own workloads.

Read original

Gpu Acceleration Distributed Computing Llm Inference Rcclx+4

dropboxFeb 12, 20263 min readCurated summary

How low-bit inference enables efficient AI

Low-bit inference reduces the memory, compute, and energy required to serve modern AI models by representing values with fewer bits. Quantization can substantially increase GPU throughput, but its benefits depend on model accuracy, hardware support, and whether workloads prioritize latency or throughput. The article presents low-bit inference as a production trade-off rather than a universally optimal technique. ## The Rising Cost of Modern Models - Models are growing rapidly, increasing demand for: - Memory capacity - Compute power - Energy - Low-latency serving infrastructure - Dropbox uses attention-based models for Dash and other capabilities involving: - Text, image, video, and audio understanding - Search and summarization - Reasoning over large collections of content - Production deployment requires balancing model capability with hardware utilization, cost, and responsiveness. ## Where Inference Compute Is Spent - Most computation comes from repeated matrix multiplications in two areas: - **Linear layers**, including attention projections, MLP layers, and final output layers. - **Attention mechanisms**, which calculate relationships between input tokens and become increasingly expensive with longer contexts. - GPUs accelerate these operations using specialized hardware: - NVIDIA Tensor Cores - AMD Matrix Cores - These cores execute matrix multiply-accumulate operations much faster than general-purpose CUDA cores. ## How Lower Precision Improves Efficiency - Quantization reduces the number of bits used to represent model values. - Converting values from 16-bit to 8-bit or 4-bit formats: - Reduces memory usage - Lowers memory-transfer costs - Can increase matrix-operation throughput - Reduces energy consumption - GPU throughput generally improves as precision decreases; halving precision can approximately double the number of operations performed per second in suitable workloads. - Eight-bit quantization maps values into 256 discrete levels. Formats below 8 bits typically require **bitpacking**, combining multiple values into types such as `uint8` or `int32` because 4-bit values are not normally stored as native hardware types. - Newer hardware, such as Blackwell GPUs with FP4 support, can provide major energy savings compared with higher-precision systems like the H100. ## Limits of Extremely Low-Bit Formats - Binary and ternary quantization restricts weights to two or three possible values, offering greater theoretical savings. - These formats are not well matched to today’s GPUs because they cannot fully use Tensor or Matrix Cores. - Specialized accelerators could make them more practical, but adoption remains limited by: - Weak ecosystem support - Hardware availability - Concerns about model quality - Practical gains therefore depend not only on bit width, but also on how well the format is supported by existing hardware and software. ## Quantization Formats and Deployment Trade-offs - Quantization is a family of techniques with different choices for: - Numerical representation - Scaling - Execution strategy - These choices affect: - Model accuracy - Inference speed - Memory consumption - Hardware utilization - Different workloads have different priorities: - Latency-sensitive applications need fast individual requests. - Throughput-oriented workloads prioritize processing large volumes efficiently. - Depending on the workload, inference may be limited by software overhead, memory bandwidth, or specialized GPU compute units. ## Pre-MXFP and MXFP Approaches - The article divides modern low-bit formats into two broad groups following the introduction of **MXFP microscaling**: - **Pre-MXFP formats** rely on software-managed scaling and explicit dequantization. - **MXFP formats** move scaling and related operations into Tensor Core hardware. - MXFP aims to standardize low-bit data types while making them more directly usable by modern GPUs. - The choice between these approaches depends on the hardware generation and the specific performance requirements of each production workload. Low-bit inference is most effective when quantization formats, model quality, and hardware capabilities are considered together. Teams should select formats based on the actual bottleneck—memory, bandwidth, latency, or compute—rather than assuming that the fewest possible bits will always deliver the best result.

Read original

Gpu Acceleration Large Language Models Model Optimization Quantization+3

awsJan 20, 2026Original article

Announcing Amazon EC2 G7e instances accelerated by NVIDIA RTX PRO 6000 Blackwell Server Edition GPUs (opens in new tab)

Amazon has announced the general availability of EC2 G7e instances, a new hardware tier powered by NVIDIA RTX PRO 6000 Blackwell Server Edition GPUs designed for generative AI and high-end graphics. These instances deliver up to 2.3 times the inference performance of their G6e predecessors while providing significant upgrades to memory and bandwidth. This launch aims to provide a cost-effective solution for running medium-sized AI models and complex spatial computing workloads at scale. **Blackwell GPU and Memory Advancements** * The G7e instances feature NVIDIA RTX PRO 6000 Blackwell GPUs, which provide twice the memory and 1.85 times the memory bandwidth of the G6e generation. * Each GPU provides 96 GB of memory, allowing users to run medium-sized models—such as those with up to 70 billion parameters—on a single GPU using FP8 precision. * The architecture is optimized for both spatial computing and scientific workloads, offering the highest graphics performance currently available in the EC2 portfolio. **High-Speed Connectivity and Multi-GPU Scaling** * To support large-scale models, G7e instances utilize NVIDIA GPUDirect P2P, enabling direct communication between GPUs over PCIe interconnects with minimal latency. * These instances offer four times the inter-GPU bandwidth compared to the L40s GPUs found in G6e instances, facilitating more efficient data transfer in multi-GPU configurations. * Total GPU memory can scale up to 768 GB within a single node, supporting massive inference tasks across eight interconnected GPUs. **Networking and Storage Performance** * G7e instances provide up to 1,600 Gbps of network bandwidth, a four-fold increase over previous generations, making them suitable for small-scale multi-node clusters. * Support for NVIDIA GPUDirect Remote Direct Memory Access (RDMA) via Elastic Fabric Adapter (EFA) reduces latency for remote GPU-to-GPU communication. * The instances support GPUDirect Storage with Amazon FSx for Lustre, achieving throughput speeds up to 1.2 Tbps to ensure rapid model loading and data processing. **System Specifications and Configurations** * Under the hood, G7e instances are powered by Intel Emerald Rapids processors and support up to 192 vCPUs and 2,048 GiB of system memory. * Local storage options include up to 15.2 TB of NVMe SSD capacity to handle high-speed data caching and local processing. * The instance family ranges from the g7e.2xlarge (1 GPU, 8 vCPUs) to the g7e.48xlarge (8 GPUs, 192 vCPUs). For developers ready to transition to Blackwell-based architecture, these instances are accessible through AWS Deep Learning AMIs (DLAMI). They represent a major step forward for organizations needing to balance the high memory requirements of modern LLMs with the cost efficiencies of the G-series instance family.

Gpu Acceleration AI Generative AI Amazon Ec2+5

awsDec 2, 2025Original article

Amazon OpenSearch Service improves vector database performance and cost with GPU acceleration and auto-optimization (opens in new tab)

Amazon OpenSearch Service has introduced serverless GPU acceleration and auto-optimization features designed to enhance the performance and cost-efficiency of large-scale vector databases. These updates allow users to build vector indexes up to ten times faster at a quarter of the traditional indexing cost, enabling the creation of billion-scale databases in under an hour. By automating complex tuning processes, OpenSearch Service simplifies the deployment of generative AI and high-speed search applications. ### GPU Acceleration for Rapid Indexing The new serverless GPU acceleration streamlines the creation of vector data structures by offloading intensive workloads to specialized hardware. * **Performance Gains:** Indexing speed is increased by 10x compared to non-GPU configurations, significantly reducing the time-to-market for data-heavy applications. * **Cost Efficiency:** Indexing costs are reduced to approximately 25% of standard costs, and users only pay for active processing through OpenSearch Compute Units (OCU) rather than idle instance time. * **Serverless Management:** There is no need to provision or manage GPU instances manually; OpenSearch Service automatically detects acceleration opportunities and isolates workloads within the user's Amazon VPC. * **Operational Scope:** Acceleration is automatically applied to both initial indexing and subsequent force-merge operations. ### Automated Vector Index Optimization Auto-optimization removes the requirement for deep vector expertise by automatically balancing competing performance metrics. * **Simplified Tuning:** The system replaces manual index tuning—which can traditionally take weeks—with automated configurations. * **Resource Balancing:** The tool finds the optimal trade-off between search latency, search quality (recall rates), and memory requirements. * **Improved Accuracy:** Users can achieve higher recall rates and better cost savings compared to using default, unoptimized index configurations. ### Configuration and Integration These features can be integrated into new or existing OpenSearch Service domains and Serverless collections through the AWS Console or CLI. * **CLI Activation:** Users can enable acceleration on existing domains using the `update-domain-config` command with the `--aiml-options` flag set to enable `ServerlessVectorAcceleration`. * **Index Settings:** To leverage GPU processing, users must create a vector index with specific settings, notably setting `index.knn.remote_index_build.enabled` to `true`. * **Supported Workloads:** The service supports standard OpenSearch operations, including the Bulk API for adding vector data and text embeddings. For organizations managing large-scale vector workloads for RAG (Retrieval-Augmented Generation) or semantic search, enabling GPU acceleration is a highly recommended step to reduce operational overhead. Developers should transition existing indexes to include the `remote_index_build` setting to take immediate advantage of the improved speed and reduced OCU pricing.

Gpu Acceleration AI Generative AI AWS+4

googleSep 30, 2025Original article

Introducing interactive on-device segmentation in Snapseed (opens in new tab)

Google has introduced a new "Object Brush" feature in Snapseed that enables intuitive, real-time selective photo editing through a novel on-device segmentation technology. By leveraging a high-performance interactive AI model, users can isolate complex subjects with simple touch gestures in under 20 milliseconds, bridging the gap between professional-grade editing and mobile convenience. This breakthrough is achieved through a sophisticated teacher-student training architecture that prioritizes both pixel-perfect accuracy and low-latency performance on consumer hardware. ### High-Performance On-Device Inference * The system is powered by the Interactive Segmenter model, which is integrated directly into the Snapseed "Adjust" tool to facilitate immediate object-based modifications. * To ensure a fluid user experience, the model utilizes the MediaPipe framework and LiteRT’s GPU acceleration to process selections in less than 20ms. * The interface supports dynamic refinement, allowing users to provide real-time feedback by tracing lines or tapping to add or subtract specific areas of an image. ### Teacher-Student Model Distillation * The development team first created "Interactive Segmenter: Teacher," a large-scale model fine-tuned on 30,000 high-quality, pixel-perfect manual annotations across more than 350 object categories. * Because the Teacher model’s size and computational requirements are prohibitive for mobile use, researchers developed "Interactive Segmenter: Edge" through knowledge distillation. * This distillation process utilized a dataset of over 2 million weakly annotated images, allowing the smaller Edge model to inherit the generalization capabilities of the Teacher model while maintaining a footprint suitable for mobile devices. ### Training via Synthetic User Prompts * To make the model universally capable across all object types, the training process uses a class-agnostic approach based on the Big Transfer (BiT) strategy. * The model learns to interpret user intent through "prompt generation," which simulates real-world interactions such as random scribbles, taps, and lasso (box) selections. * During training, both the Teacher and Edge models receive identical prompts—such as red foreground scribbles and blue background scribbles—to ensure the student model learns to produce high-quality masks even from imprecise user input. This advancement significantly lowers the barrier to entry for complex photo manipulation by moving heavy-duty AI processing directly onto the mobile device. Users can expect a more responsive and precise editing experience that handles everything from fine-tuning a subject's lighting to isolating specific environmental elements like clouds or clothing.

Gpu Acceleration AI Machine Learning Computer Vision+5