dynamodb

6 posts

aws

AWS Weekly Roundup: AWS Local Zones in Istanbul, open-source ExtendDB, Kiro Web, and more (May 25, 2026) | Amazon Web Services (opens in new tab)

AWS’s latest updates focus on expanding regional infrastructure, improving developer workflows, and making cloud and AI services more portable. The Istanbul Local Zone strengthens data residency and low-latency capabilities in Türkiye, while tools such as ExtendDB, OpenAI-compatible SageMaker APIs, and Kiro Web reduce migration and development friction. Together, these releases emphasize flexibility, operational resilience, and easier local testing. ## AWS Local Zone in Istanbul - AWS opened a new Local Zone in Istanbul, Türkiye. - It provides nearby compute, storage, and networking with single-digit millisecond latency. - Organizations can keep and process data within Turkish borders to support residency and compliance requirements. - The zone supports latency-sensitive workloads in sectors such as finance, government, telecommunications, and healthcare. - Applications can combine Istanbul infrastructure with the broader AWS Region, enabling hybrid architectures without operating a private data center. ## Security and AI Service Updates - **Security Hub Extended** now integrates with 21 curated partner solutions across nine security categories, including endpoint protection, threat intelligence, and cloud security posture management. - **Amazon SageMaker AI** supports OpenAI-compatible inference APIs, allowing existing OpenAI-based applications to use SageMaker with minimal or no SDK changes. - **Secrets Manager Agent** can pre-fetch secrets at startup, reducing cold-start delays, and can assume IAM roles for workloads with different permission boundaries. - **Amazon Bedrock** introduced tools for advanced prompt optimization and migration across foundation models. ## Open-Source and Local Development Tools - AWS open-sourced **ExtendDB**, a DynamoDB-compatible adapter for alternative storage backends. - It supports local development and testing without a live AWS connection. - It can help teams retain DynamoDB API semantics while controlling the underlying storage layer. - **AWS SAM CLI** now supports CloudFormation Language Extensions locally, improving consistency between local testing and production deployments. ## Developer Experience and Reliability - **Kiro Web** brings AWS’s AI-assisted, spec-driven development environment to browsers, providing access to chat and agent capabilities without installing the desktop IDE. - AWS updated default retry behavior across SDKs and CLI tools. - Improvements include smarter backoff and better throttling handling. - Production applications should become more resilient to transient failures without additional configuration. ## Container Image Changes - Bitnami images are being removed from Amazon ECR Public. - Teams currently using those images should review the migration timeline and update image references to Bitnami’s own registry to avoid interruptions. ## Upcoming AWS Events - AWS Summit Amsterdam: May 27 - AWS Summit Bangkok: May 28 - AWS Summit Milan: May 28 Builders should evaluate the Istanbul Local Zone for residency- or latency-sensitive systems, consider ExtendDB and SAM improvements for local workflows, and review the Bitnami registry change before images are removed from ECR Public.

aws

Announcing Amazon Aurora PostgreSQL serverless database creation in seconds | Amazon Web Services (opens in new tab)

Amazon’s new Aurora PostgreSQL express configuration lets developers create a serverless database in seconds with two console clicks or a single CLI/API call. It uses preconfigured defaults, IAM authentication, and an internet access gateway to simplify secure connections without requiring a VPC, VPN, or Direct Connect. The feature is designed to accelerate prototyping and application development while preserving Aurora capabilities such as read replicas and automated failover. ## Express Configuration for Aurora PostgreSQL - Creates an Aurora PostgreSQL serverless cluster and instance within seconds. - Uses preconfigured defaults to reduce setup complexity. - Allows customization of: - Cluster identifier - Serverless capacity range during creation - Read replicas and parameter groups after creation - Express-configured clusters do not require an Amazon VPC. - An internet access gateway is enabled by default for secure connections from development tools worldwide. - The gateway is distributed across multiple Availability Zones for high availability. - IAM authentication is configured for the administrator, enabling passwordless database authentication. ## Creating a Database - In the Aurora and RDS console: - Open the Dashboard. - Choose **Create** with the rocket icon. - Review or adjust the express configuration. - Choose **Create database**. - The AWS CLI and SDKs support the `--with-express-configuration` parameter. - A single `create-db-cluster` call creates both the cluster and its instance: ```bash aws rds create-db-cluster \ --db-cluster-identifier channy-express-db \ --engine aurora-postgresql \ --with-express-configuration ``` - The database becomes ready when its status changes to **Available**. ## Connecting to the Database The **Connectivity & security** tab provides several connection methods: - **Code snippets** - Generates connection examples for .NET, Go, JDBC, Node.js, PHP, PostgreSQL, Python, and TypeScript. - Python examples use `boto3` to generate an IAM authentication token and `psycopg2` to connect over SSL. - **AWS CloudShell** - Launches a shell with a preconfigured `psql` connection command. - Developers can immediately run SQL commands at the PostgreSQL prompt. - **Endpoints** - Supports tools such as pgAdmin that use username-and-password fields. - The password is an IAM authentication token valid for 15 minutes. - A new token must be generated if the connection ends or the token expires. ## Application Development Integrations - Aurora is now included among eligible AWS Free Tier database services. - AWS’s enhanced Free Tier offers up to $200 in credits: - $100 upon signup - Up to another $100 through usage of services such as RDS, Lambda, and Bedrock - Integrations with Vercel and v0 allow developers to create or connect to AWS databases quickly. - v0 can use natural-language prompts to generate full-stack applications backed by Aurora PostgreSQL, Aurora DSQL, or DynamoDB. - Existing Aurora databases created with express configuration can also be connected to Vercel. The express configuration is best suited for quickly starting development, experimentation, and prototypes. Developers can begin with minimal networking and authentication setup, then add capacity, replicas, and other Aurora features as their application grows.

dropbox

Inside the feature store powering real-time AI in Dropbox Dash (opens in new tab)

Dropbox Dash’s ranking system depends on a hybrid feature store that can combine real-time user behavior with large-scale historical data. Because Dropbox operates across on-premises and cloud environments, and because each query can trigger thousands of feature lookups, off-the-shelf systems could not meet its latency, scale, and integration requirements. The resulting architecture uses Feast for orchestration, Spark for computation, Dynovault for low-latency storage, and a custom Go serving layer, achieving roughly 25–35 ms p95 latency while keeping features fresh. ## Goals and Requirements - Dash ranks documents, images, and conversations using behavioral, contextual, and real-time signals. - A single query can fan out into thousands of feature lookups across many candidate files. - The feature store needed to: - Support sub-100 ms search latency. - Reflect user actions within seconds or minutes. - Bridge Dropbox’s on-premises services and Spark-based cloud infrastructure. - Handle both streaming-style updates and batch computations. - Let engineers develop features without managing serving and orchestration details. ## Choosing a Hybrid Architecture - Dropbox evaluated Feast, Hopsworks, Featureform, Feathr, Databricks, and Tecton. - Feast was selected because: - It separates feature definitions from infrastructure concerns. - Engineers can focus on PySpark transformations. - Its modular adapter system supports existing Dropbox infrastructure. - Feast’s DynamoDB adapter enabled integration with Dynovault, Dropbox’s DynamoDB-compatible storage system. - The architecture combines: - Feast for orchestration and serving APIs. - Spark jobs for feature computation and ingestion. - Cloud storage for offline indexing and data management. - Dynovault for online, low-latency lookups. - A custom Go service replacing Feast’s Python online serving path. - Dynovault is colocated with inference workloads and provides approximately 20 ms client-side latency. - Monitoring covers job failures, feature freshness, and data lineage. ## Replacing Python with Go for Low Latency - The initial Feast-based Python service struggled under heavy concurrency. - CPU-bound JSON parsing and Python’s Global Interpreter Lock became bottlenecks. - A multi-process design helped temporarily but introduced coordination overhead. - The serving layer was rewritten in Go using: - Lightweight goroutines. - Shared memory. - Faster JSON parsing. - The Go service now handles thousands of requests per second. - It adds only about 5–10 ms beyond Dynovault latency and achieves roughly 25–35 ms p95 latency. ## Keeping Features Fresh - Fresh signals are essential for ranking quality; actions such as opening a document should influence subsequent searches quickly. - Fully real-time computation is impractical for features requiring large joins, aggregations, and historical context. - Dropbox therefore built a three-part ingestion strategy. - Batch ingestion handles complex, high-volume transformations using a medallion architecture. - Intelligent change detection updates only modified records rather than rewriting every feature. - This reduced online-store writes from hundreds of millions to fewer than one million per run and significantly shortened update time. ## Practical Takeaway The system demonstrates that a feature store does not need to be entirely off-the-shelf or entirely real-time. Combining a modular framework with custom serving, colocated storage, batch optimization, and freshness monitoring allowed Dropbox to meet demanding latency and scale requirements while keeping feature development manageable.

figma

The Infrastructure Behind AI Search in Figma | Figma Blog (opens in new tab)

Figma’s AI search lets users find designs and components through text, screenshots, or layer selections. It relies on multimodal embeddings, vector nearest-neighbor search, and large-scale indexing of frames and components. The main infrastructure challenge was generating and indexing billions of embeddings efficiently while controlling costs. ## AI-Powered Search Flows - **Search for designs** indexes frames across users’ files, including unlabeled frames buried in complex files. - Users can search designs: - Lexically with a text description - Visually with a screenshot - By selecting similar Figma layers - **Search for components** enhances the Assets panel with semantic matching. - A component representing 😀 can be found with terms such as “smiley,” “happy,” “face,” or “grin.” - Designers no longer need to manually add every possible keyword to component descriptions. - Components can also be found using visual queries. ## Multimodal Embeddings - An embedding model converts text or images into numerical vectors that represent their meaning. - Figma uses the open-source **CLIP** model, which places text and images in the same embedding space. - The embedding for the word “cat” should be numerically close to an embedding generated from an image of a cat. - Figma’s models were not trained on private customer files or data. - Fine-tuning used interface images from public, free Community files. - Search works by: - Generating embeddings for indexed content - Creating an embedding for the user’s query - Finding indexed vectors that are nearest to the query vector - Figma tested embeddings based on textual representations such as JSON, but image-based embeddings produced better results and supported the same workflow as screenshot search. - Layer selections are converted into screenshots before being passed to the embedding model. ## Populating the Vector Search Index - Each searchable item requires: - A thumbnail or rendered screenshot - An embedding - Metadata stored in the search index - Figma uses DynamoDB for metadata and embeddings because the workload primarily requires high-throughput key-value reads and writes. - Identifying searchable frames is difficult because unpublished frames are not directly enumerable. - Figma runs a headless, server-side version of its C++ editor in asynchronous jobs to discover frames within files. - These jobs use server-side sandboxing techniques to safely run the editor. Figma’s approach combines CLIP-based multimodal representations, server-side rendering, asynchronous processing, and scalable vector storage to make visual and semantic search practical across large design systems.

figma

Making multiplayer more reliable | Figma Blog (opens in new tab)

Figma improved multiplayer reliability by adding a durable write-ahead journal alongside its existing checkpoint system. Instead of relying on full-file snapshots every 30–60 seconds, Figma now records incremental changes frequently, allowing crashed servers to recover nearly to the latest state and reducing deployment-related database spikes. The goal was to reduce potential data loss from up to 60 seconds to less than one second. ## How Figma’s Multiplayer System Worked - Browsers connect to Figma’s multiplayer service over WebSockets. - The service authoritatively handles: - Validation - Ordering - Conflict resolution - Broadcasting updates to connected clients - File state is held in memory for speed. - Every 30–60 seconds, the entire file is: - Encoded into a binary format - Compressed - Uploaded to Amazon S3 as a checkpoint ## Problems with Checkpoint-Only Persistence - A multiplayer crash could lose up to 60 seconds of server-side work. - Checkpoints become increasingly expensive as files grow in size and complexity. - Redeployments caused large write spikes: - All in-memory files had to be closed. - Each file needed a final checkpoint. - The resulting burst increased database load. ## Introducing the Journal - Figma added a durable transaction log, or journal, backed by DynamoDB. - Each accepted change receives an incrementing sequence number. - Checkpoints store the latest sequence number they include. - During recovery: - Multiplayer loads the latest checkpoint. - It queries the journal for entries with higher sequence numbers. - It replays those incremental changes to reconstruct the latest file state. - Journal entries are much smaller than full-file checkpoints, since they contain only user edits. - Figma writes journal data roughly every 0.5 seconds rather than waiting 60 seconds between full snapshots. - The target was less than one second of data loss in rare failure scenarios. ## Smoother Deployments - During deployment, Figma can close connections and wait for unsaved changes to reach the journal. - The 99th percentile persistence time is under one second. - Since journal writes happen continuously during normal operation, deployments no longer create a sudden checkpoint-writing surge. - Database write load is therefore steadier and more predictable. ## Datastore and Batching Decisions - Figma selected DynamoDB because the journal requires a horizontally scalable datastore with high write capacity. - Postgres was considered but rejected because the anticipated write volume exceeded Figma’s current horizontal-scaling approach for Postgres. - Clients send updates at approximately 30 frames per second, or every 33 milliseconds. - The journal does not need that granularity, so multiple changes can be batched before being persisted, improving performance. Figma’s approach combines inexpensive, frequent incremental journal writes with larger periodic checkpoints. This provides faster recovery, minimizes data loss, and avoids deployment-related load spikes while retaining checkpoints for efficient long-term storage and features such as version history.

datadog

How Datadog's IT team automated monitoring third-party accounts (opens in new tab)

Datadog built “Clarity,” an automated system for auditing SaaS accounts against Workday’s employee records. It regularly identifies accounts that do not belong to active employees, then logs, tickets, stores, and communicates findings through Datadog, Freshservice, Slack, and DynamoDB. The system replaces infrequent, manual reviews with continuous visibility and faster remediation of security and cost risks. ## The Need for Automated SaaS Audits - Modern companies rely on dozens or hundreds of external applications. - Manual account reviews are difficult to scale and may fail to detect unauthorized or abandoned accounts promptly. - An unexpected account in an identity provider or SaaS application could give a bad actor access to sensitive systems. - Datadog needed recurring audits as its SaaS portfolio continued to expand. ## Clarity’s Requirements - Use a single source of truth for employee status: - Datadog uses Workday. - Other organizations could use Okta, OneLogin, ADP, or Active Directory. - Run frequently enough to provide timely visibility. - Support manual execution when needed. - Integrate with existing communication, ticketing, and observability tools, such as Slack, Freshservice, and Datadog. - Minimize disruption to IT workflows and encourage adoption across a globally distributed organization. ## Audit Pipeline - A CloudWatch Event Rule triggers the audit Monday through Friday at 10 a.m. EST. - Clarity concurrently retrieves: - Active employees from Workday. - Active users from primary SaaS applications such as Slack, GitHub, and Zoom. - It compares SaaS user email addresses with active employee records. - Accounts without a matching active employee are flagged. - Results are: - Sent to Datadog as logs and metrics. - Added to DynamoDB for historical tracking. - Converted into Freshservice tickets. - Reported through Slack notifications with an audit summary. ## Datadog Metrics and Investigation - Clarity sends a metric for every flagged account using the Datadog Metrics API and Python SDK. - It uses a gauge metric to track flagged accounts over time. - Metrics include tags such as: - Environment, such as production. - Responsible team. - SaaS service. - Flagged user’s email address. - These tags provide the context needed to investigate the account and support alerting and visualization within Datadog. ## Practical Outcome Clarity provides a repeatable, automated control for SaaS account governance. Organizations implementing a similar system should connect an authoritative employee directory to their SaaS inventory, run audits regularly, and integrate findings with their existing monitoring, ticketing, and notification workflows.