toss·Nov 30, 2025Original article

From Legacy Payment Ledger to Scalable System (opens in new tab)

Toss Payments successfully modernized a 20-year-old legacy payment ledger by transitioning to a decoupled, MySQL-based architecture designed for high scalability and consistency. By implementing strategies like INSERT-only immutability and event-driven domain isolation, they overcame structural limitations such as the inability to handle split payments. Ultimately, the project demonstrates that robust system design must be paired with resilient operational recovery mechanisms to manage the complexities of large-scale financial migrations. ### Legacy Ledger Challenges * **Inconsistent Schemas:** Different payment methods used entirely different table structures; for instance, a table named `REFUND` unexpectedly contained only account transfer data rather than all refund types. * **Domain Coupling:** Multiple domains (settlement, accounting, and payments) shared the same tables and columns, meaning a single schema change required impact analysis across several teams. * **Structural Limits:** A rigid 1:1 relationship between a payment and its method prevented the implementation of modern features like split payments or "Dutch pay" models. ### New Ledger Architecture * **Data Immutability:** The system shifted from updating existing rows to an **INSERT-only** principle, ensuring a reliable audit trail and preventing database deadlocks. * **Event-Driven Decoupling:** Instead of direct database access, the system uses Kafka to publish payment events, allowing independent domains to consume data without tight coupling. * **Payment-Approval Separation:** By separating the "Payment" (the transaction intent) from the "Approval" (the specific financial method), the system now supports multiple payment methods per transaction. ### Safe Migration and Data Integrity * **Asynchronous Mirroring:** To maintain zero downtime, data was initially written to the legacy system and then asynchronously loaded into the new MySQL ledger. * **Resource Tuning:** Developers used dedicated migration servers within the same AWS Availability Zone to minimize latency and implemented **Bulk Inserts** to handle hundreds of millions of rows efficiently. * **Verification Batches:** A separate batch process ran every five minutes against a Read-Only (RO) database to identify and correct any data gaps caused by asynchronous processing failures. ### Operational Resilience and Incident Response * **Query Optimization:** During a load spike, the MySQL optimizer chose "Full Scans" over indexes; the team resolved this by implementing SQL hints and utilizing a 5-version Docker image history for rapid rollbacks. * **Network Cancellation:** To handle timeouts between Toss and external card issuers, the system uses specific logic to automatically send cancellation requests and synchronize states. * **Timeout Standardization:** Discrepancies between microservices were resolved by calculating the maximum processing time of approval servers and aligning all upstream timeout settings to prevent merchant response mismatches. * **Reliable Event Delivery:** While using the **Outbox pattern** for events, the team added log-based recovery (Elasticsearch and local disk) and idempotency keys in event headers to handle both missing and duplicate messages. For organizations tackling significant technical debt, this transition highlights that initial design is only half the battle. True system reliability comes from building "self-healing" structures—such as automated correction batches and standardized timeout chains—that can survive the unpredictable nature of live production environments.

MicroservicesDatabase DesignApache KafkaMysql+4

netflix·Nov 14, 2025Original article

How and Why Netflix Built a Real-Time Distributed Graph: Part 1 — Ingesting and Processing Data Streams at Internet Scale | by Netflix Technology Blog | Netflix TechBlog (opens in new tab)

Netflix has developed a Real-Time Distributed Graph (RDG) to unify member interaction data across its expanding business verticals, including streaming, live events, and mobile gaming. By transitioning from siloed microservice data to a graph-based model, the company can perform low-latency, relationship-centric queries that were previously hindered by expensive manual joins and data fragmentation. The resulting system enables Netflix to track user journeys across various devices and platforms in real-time, providing a foundation for deeper personalization and pattern detection. ### Challenges of Data Isolation in Microservices * While Netflix’s microservices architecture facilitates independent scaling and service decomposition, it inherently leads to data isolation where each service manages its own storage. * Data scientists and engineers previously had to "stitch" together disparate data from various databases and the central data warehouse, which was a slow and manual process. * The RDG moves away from table-based models to a relationship-centric model, allowing for efficient "hops" across nodes without the need for complex denormalization. * This flexibility allows the system to adapt to new business entities (like live sports or games) without requiring massive schema re-architectures. ### Real-Time Ingestion and Normalization * The ingestion layer is designed to capture events from diverse upstream sources, including Change Data Capture (CDC) from databases and request/response logs. * Netflix utilizes its internal data pipeline, Keystone, to funnel these high-volume event streams into the processing framework. * The system must handle "Internet scale" data, ensuring that events from millions of members are captured as they happen to maintain an up-to-date view of the graph. ### Stream Processing with Apache Flink * Netflix uses Apache Flink as the core stream processing engine to handle the transformation of raw events into graph entities. * Incoming data undergoes normalization to ensure a standardized format, regardless of which microservice or business vertical the data originated from. * The pipeline performs data enrichment, joining incoming streams with auxiliary metadata to provide a comprehensive context for each interaction. * The final step of the processing layer involves mapping these enriched events into a graph structure of nodes (entities) and edges (relationships), which are then emitted to the system's storage layer. ### Practical Conclusion Organizations operating with a highly decoupled microservices architecture should consider a graph-based ingestion strategy to overcome the limitations of data silos. By leveraging stream processing tools like Apache Flink to build a real-time graph, engineering teams can provide stakeholders with the ability to discover hidden relationships and cross-domain insights that are often lost in traditional data warehouses.

MicroservicesDistributed SystemsReal Time DataData Architecture+3

line·Nov 13, 2025Original article

Pushsphere: The Secret to Fast and (opens in new tab)

LINE developed Pushsphere to overcome the inherent instability and rate-limiting challenges of delivering high-volume push notifications via providers like APNs and FCM. By implementing a sophisticated gateway architecture rather than relying on naive retry logic, the system ensures reliable delivery even during massive traffic spikes or regional emergencies. This approach has successfully stabilized the messaging pipeline, drastically reducing operational overhead and system-wide failures. ## Limitations of Standard Push Architectures * External push providers are frequently unstable, exhibiting misbehaving instances, sudden disconnections, and unpredictable timeouts. * Naive retry strategies often lead to "retry storms," which quickly exhaust rate-limit quotas and result in HTTP 429 (Too Many Requests) errors. * At massive scales, manual management of hundreds of server connections becomes impossible, necessitating automated decisions on when to abandon or switch between faulty nodes. ## Unified Gateway Design and High-Performance Transport * Pushsphere provides a single entry point for all push platforms, abstracting the complexities of mTLS for Apple and OAuth 2.0 for Firebase. * The system is built on the Armeria microservice framework and utilizes Netty for high-performance, non-blocking communication within the Java Virtual Machine. * The architecture includes a client library and gateway server that support zone-aware routing, ensuring low latency and efficient traffic distribution across data centers. ## Intelligent Retry and Load Balancing Strategies * The "retry-aware" load balancer uses a Round Robin base strategy but is designed to skip previously attempted endpoints during a retry cycle to avoid repeated failures on faulty nodes. * Quota-aware logic monitors rate limits in real-time, preventing the system from retrying endpoints that are nearing their capacity. * These smarter traffic distribution rules balance high delivery success rates with the preservation of provider quotas, preventing service-wide blocking. ## Resilient Endpoint Management via Circuit Breakers * Pushsphere assigns a dedicated circuit breaker to every endpoint to report success and failure rates continuously. * When a circuit opens due to frequent failures, the unhealthy endpoint is immediately removed from the active pool and replaced with a fresh candidate from a DNS-refreshed pool. * This automated replacement mechanism maintains a consistent pool of healthy endpoints, allowing the system to remain stable without manual intervention during hardware or network degradations. Pushsphere has transformed LINE's notification infrastructure, reducing annual on-call alerts from over 30 to just four, despite implementing stricter monitoring thresholds. For developers managing high-volume messaging services, adopting a gateway-based approach with automated circuit breaking and quota awareness is a proven path to achieving carrier-grade reliability.

MicroservicesPush NotificationsArmeriaNetty+4

netflix·Sep 22, 2025Original article

Scaling Muse: How Netflix Powers Data-Driven Creative Insights at Trillion-Row Scale | by Netflix Technology Blog | Netflix TechBlog (opens in new tab)

Netflix’s Muse platform has evolved from a simple dashboard into a high-scale Online Analytical Processing (OLAP) system that processes trillions of rows to provide creative insights for promotional media. To meet growing demands for complex audience affinity analysis and advanced filtering, the engineering team modernized the data serving layer by moving beyond basic batch pipelines. By integrating HyperLogLog sketches for approximate counting and leveraging in-memory precomputed aggregates, the system now delivers low-latency performance and high data accuracy at an immense scale. ### Approximate Counting with HyperLogLog (HLL) Sketches To track metrics like unique impressions and qualified plays without the massive overhead of comparing billions of profile IDs, Muse utilizes the Apache Datasketches library. * The system trades a small margin of error (approximately 0.8% with a logK of 17) for significant gains in processing speed and memory efficiency. * Sketches are built during Druid ingestion using the HLLSketchBuild aggregator with rollup enabled to reduce data volume. * In the Spark ETL process, all-time aggregates are maintained by merging new daily HLL sketches into existing ones using the `hll_union` function. ### Utilizing Hollow for In-Memory Aggregates To reduce the query load on the Druid cluster, Netflix uses Hollow, an internal open-source tool designed for high-density, near-cache data sets. * Muse stores precomputed, all-time aggregates—such as lifetime impressions per asset—within Hollow’s in-memory data structures. * When a user requests "all-time" data, the application retrieves the results from the Hollow cache instead of forcing Druid to scan months or years of historical segments. * This approach significantly lowers latency for the most common queries and frees up Druid resources for more complex, dynamic filtering tasks. ### Optimizing the Druid Data Layer Efficient data retrieval from Druid is critical for supporting the application’s advanced grouping and filtering capabilities. * The team transitioned from hash-based partitioning to range-based partitioning on frequently filtered dimensions like `video_id` to improve data locality and pruning. * Background compaction tasks are utilized to merge small segments into larger ones, reducing metadata overhead and improving scan speeds across the cluster. * Specific tuning was applied to the Druid broker and historical nodes, including adjusting processing threads and buffer sizes to handle the high-concurrency demands of the Muse UI. ### Validation and Data Accuracy Because the move to HLL sketches introduces approximation, the team implemented rigorous validation processes to ensure the data remained actionable. * Internal debugging tools were developed to compare results from the new architecture against the "ground truth" provided by legacy batch systems. * Continuous monitoring ensures that HLL error rates remain within the expected 1–2% range and that data remains consistent across different time grains. For organizations building large-scale OLAP applications, the Muse architecture demonstrates that performance bottlenecks can often be solved by combining approximate data structures with specialized in-memory caches to offload heavy computations from the primary database.

MicroservicesGraphQLApache SparkApache Druid+4

airbnb·Sep 15, 2025·4 min readCurated summary

Viaduct, Five Years On: Modernizing the Data-Oriented Service Mesh

Viaduct, Airbnb’s data-oriented service mesh, has evolved substantially over five years while retaining its core model: a central schema, hosted business logic, and re-entrant composition through GraphQL. Its usage has grown eightfold, supporting more than 130 teams and over 1.5 million lines of production code, without increasing operational overhead. Viaduct Modern now aims to simplify its developer API and establish stronger architectural boundaries, alongside the project’s release as open source. ## Adoption and Evolution - Viaduct traffic has increased by a factor of eight since 2020. - More than 130 teams now host code in Viaduct, supported by hundreds of weekly active developers. - The hosted codebase has grown to over 1.5 million lines, with roughly the same amount of test code. - Operational overhead has remained constant, incident-minutes have been cut in half, and costs have grown linearly with QPS. - Viaduct is now available as open-source software. ## Core Principles That Remain - **Central schema:** Viaduct provides one integrated schema connecting domains across Airbnb. - More than 75% of requests are internal. - The schema is developed by many teams but exposed as a connected graph. - **Hosted business logic:** Teams run business logic directly in Viaduct rather than maintaining separate microservices. - This reduces operational overhead and can allow standalone services to be retired. - Viaduct provides a serverless environment so developers can focus on application logic. - **Re-entrancy:** Hosted logic composes with other hosted logic through GraphQL fragments and queries. - This supports modularity. - It helps avoid the tightly coupled structure and maintenance problems associated with traditional monoliths. ## Problems with the Earlier Design - Viaduct’s APIs evolved reactively in response to individual use cases. - Multiple mechanisms emerged for accomplishing similar tasks, creating confusion for developers. - Some capabilities were well supported while others were not. - The framework’s layers had loose, inconsistent interfaces. - The boundary between Viaduct and hosted application code was weak. - These issues made framework improvements increasingly risky because changes could disrupt existing users. ## Simplifying the Tenant API - Viaduct Modern overhauls the developer-facing API and execution engine. - The new Tenant API reduces the implementation choices to two mechanisms: - **Node resolvers** - **Field resolvers** - The choice is determined by the schema rather than by ad hoc behavioral distinctions. - Resolver APIs have been unified wherever possible. - The goal is a smaller, more consistent surface that preserves successful ideas from the old API while removing unnecessary alternatives. ## Tenant Modules and Re-Entrant Composition - Viaduct uses modules and re-entrancy to provide boundaries similar to service definitions and RPC APIs in microservice architectures. - A tenant module combines: - Schema owned by a team - The code implementing that schema - Modules can create rich connections in the shared graph, but direct code dependencies between teams are discouraged. - Instead, teams declare their data requirements through GraphQL fragments and queries. ### Example: Extending the `User` Type - A Core User team owns the base `User` type and resolves fields such as `firstName` and `lastName`. - A Messaging team can extend `User` with a `displayName` field. - Its resolver declares that it needs `firstName` and `lastName`. - Messaging does not depend directly on Core User’s implementation or need to know where those fields originate. - This declarative model lets teams collaborate through the schema while preserving ownership and modularity. ## Framework Modularity - Viaduct Modern also restructures the framework itself. - The system consists of: - The GraphQL execution engine - The Tenant API - Hosted application code - Historically, the interfaces between these layers were weak, making performance and reliability improvements difficult to introduce safely. - The redesign focuses on stronger abstraction boundaries so the framework can evolve independently of application code. Viaduct’s modernization is intended to preserve its centralized, data-oriented model while making development simpler and framework evolution safer. The open-source release provides an opportunity for other organizations to evaluate or adopt this approach to schema-driven, modular service composition.

Read original(opens in new tab)

MicroservicesDistributed SystemsServerlessAPI Design+3

datadog·Jun 17, 2025·1 min readCurated summary

Breaking up a monolith: How we’re unwinding a shared database at scale | Datadog

The provided text does not contain the blog post itself. It mainly includes Datadog’s navigation menu and a promotional link announcing its Gartner Magic Quadrant recognition, so the article’s argument and technical conclusions cannot be reliably summarized. ## Content Present in the Extract - A promotional banner links to Datadog’s recognition as a **Leader in the Gartner Magic Quadrant for Observability Platforms**. - The page navigation lists Datadog offerings across: - Infrastructure and application monitoring - Logs, databases, and data observability - Security - Digital experience monitoring - Software delivery - Service management - AI capabilities - The URL suggests the intended article is **“Unwinding a Shared Database”**, but its body text is missing. ## Practical Conclusion Please provide the article’s actual text or a complete page extract for a meaningful technical summary.

Read original(opens in new tab)

MicroservicesDatabase DesignDatadogObservability

coupang·Nov 15, 2024Original article

Coupang SCM Workflow: Developing (opens in new tab)

Coupang has developed an internal SCM Workflow platform to streamline the complex data and operational needs of its Supply Chain Management team. By implementing low-code and no-code functionalities, the platform enables developers, data scientists, and business analysts to build data pipelines and launch services without the traditional bottlenecks of manual development. ### Addressing Inefficiencies in SCM Data Management * The SCM team manages a massive network of suppliers and fulfillment centers (FCs) where demand forecasting and inventory distribution require constant data feedback. * Traditionally, non-technical stakeholders like business analysts (BAs) relied heavily on developers to build or modify data pipelines, leading to high communication costs and slower response times to changing business requirements. * The new platform aims to simplify the complexity found in traditional tools like Jenkins, Airflow, and Jupyter Notebooks, providing a unified interface for data creation and visualization. ### Democratizing Access with the No-code Data Builder * The "Data Builder" allows users to perform data queries, extraction, and system integration through a visual interface rather than writing backend code. * It provides seamless access to a wide array of data sources used across Coupang, including Redshift, Hive, Presto, Aurora, MySQL, Elasticsearch, and S3. * Users can construct workflows by creating "nodes" for specific tasks—such as extracting inventory data from Hive or calculating transfer quantities—and linking them together to automate complex decisions like inter-center product transfers. ### Expanding Capabilities through Low-code Service Building * The platform functions as a "Service Builder," allowing users to expand domains and launch simple services without building entirely new infrastructure from scratch. * This approach enables developers to focus on high-level algorithm development while allowing data scientists to apply and test new models directly within the production environment. * By reducing the need for code changes to reflect new requirements, the platform significantly increases the agility of the SCM pipeline. Organizations managing complex, data-driven ecosystems can significantly reduce operational friction by adopting low-code/no-code platforms. Empowering non-technical stakeholders to handle data processing and service integration not only accelerates innovation but also allows engineering resources to be redirected toward core architectural challenges.

MicroservicesAutomationData PipelineData Engineering+3

datadog·May 12, 2022·3 min readCurated summary

How Datadog's IT team automated account inactivity and SaaS spend management

Datadog expanded its Clarity auditing tool into Clarity License Manager (CLM), a system that tracks SaaS usage, reduces licensing costs, and improves security. CLM identifies inactive accounts, notifies employees, automatically deactivates unused access, and restores it quickly when needed. Its microservice architecture and application-specific adapters allow the system to scale across many SaaS products. ## The SaaS License Management Problem - Datadog used many commercial SaaS tools with substantial per-user costs. - License usage data was outdated and collected through quarterly manual audits. - IT Support had to contact employees individually, creating administrative overhead and a poor user experience. - Unused accounts also created security risks, including stale credentials that could be compromised. ## Goals of Clarity License Manager - Monitor and automatically deactivate inactive accounts, especially in sensitive services such as cloud providers. - Reduce the risk of leaked or abused stale credentials. - Limit the potential impact of security incidents. - Lower SaaS spending and support data-driven licensing decisions. - Preserve employee productivity through an easy account restoration process. ## Usage Monitoring and Automated Workflows - CLM gathers activity data through: - Direct integrations with individual SaaS APIs. - Google Workspace SAML audit logs for indirect integrations. - Employee activity is stored per application in an Amazon RDS-backed PostgreSQL database. - Employees receive email and Slack notifications after a configurable period of inactivity, with 90 days as the default. - Notifications explain the specific login or application action required to remain active. - If the employee does not respond after multiple reminders, CLM deactivates the account automatically. - Employees can reopen access by submitting a Freshservice ticket. - Accounts are restored within seconds, including their previous roles and permissions. ## Microservice Architecture - CLM consists of Python microservices running on AWS Lambda. - The services share a central PostgreSQL database. - Microservices provide: - Easier scaling as Datadog adds more SaaS applications. - Greater resilience and flexibility. - A modular foundation for future development. - The architecture introduced complexity because services required different APIs and libraries with overlapping functionality. ## Application-Specific Adapters - Each SaaS product is represented by an adapter shared across CLM microservices. - Adapters isolate application-specific API logic from the core workflows. - A typical adapter supports operations such as: - Retrieving users. - Fetching login activity. - Activating and deactivating accounts. - Onboarding and offboarding users. - This design provides: - Clear separation of responsibilities. - Reusable and flexible integration code. - Simpler microservices that do not need to handle each application’s unique behavior. CLM demonstrates how automated usage monitoring can simultaneously improve SaaS security, reduce unnecessary spending, and minimize disruption for employees. A modular adapter-based architecture is particularly useful when managing a growing portfolio of third-party applications.

Read original(opens in new tab)

MicroservicesDatabase DesignPostgreSQLPython+3

airbnb·Nov 10, 2020·3 min readCurated summary

Taming Service-Oriented Architecture Using A Data-Oriented Service Mesh

Airbnb’s Viaduct rethinks the service mesh as a data-oriented layer rather than a network for routing procedural service calls. Built on GraphQL, it presents a unified data graph that hides microservice dependencies from consumers and improves modularity in large SOAs. The central schema can also coordinate service APIs, database models, and serverless data transformations, making system-wide changes more agile. ## The Problem with Large SOAs - Modern organizations may operate thousands of microservices connected through highly tangled dependency graphs. - These graphs resemble “spaghetti code” at the service level: - Changes become difficult to plan. - Teams must coordinate across many service boundaries. - Consumers often depend directly on multiple underlying services. - Airbnb argues that microservice architectures need stronger organizing principles and technical mechanisms for enforcing modularity. ## From Procedure-Oriented to Data-Oriented Design - Traditional procedural design groups procedures into modules with public APIs and hidden implementation details. - Data-oriented design instead organizes software around encapsulated data objects and the methods that operate on them. - Microservices have largely returned SOA to a procedural model: - Each service exposes collections of remote procedural endpoints. - Consumers must know which services provide the data they need. - Viaduct applies data-oriented principles to the service mesh itself. ## Viaduct’s GraphQL Data Mesh - Viaduct defines the mesh through a GraphQL schema containing: - Types and interfaces representing managed data. - Queries and subscriptions for reading data. - Mutations for updating data. - The schema forms a single graph spanning data owned by many microservices. - A consumer can navigate related data through one query, such as: - `productById { manufacturer }` - `productById { reviews }` - `productById { reviews { author } }` - Viaduct determines which services provide each requested field. - This hides service dependencies from consumers and prevents every client from building its own cross-service orchestration logic. ## The Central Schema - Unlike distributed GraphQL approaches that split schemas across modules or federated services, Viaduct treats the schema as one central artifact. - Airbnb uses schema-management primitives to let multiple teams collaborate while preserving a unified model. - Portions of the central schema can define individual microservice APIs. - Airbnb ultimately aims to use the same schema to define database structures. - This could improve “data agility”: - Database changes would no longer need manual translation through several API layers. - A single schema update could propagate changes from storage through services to clients. - Cross-team coordination and delivery times could be reduced. ## Serverless Derived Fields - Many SOAs contain stateless services that transform backend data for particular clients or presentation layers. - Viaduct supports derived fields computed by serverless cloud functions. - These functions operate on the graph without needing direct knowledge of the underlying microservices. - Moving transformation logic into stateless containers can: - Reduce the number of services. - Lower operational overhead. - Keep the core service graph simpler. ## Implementation and Operational Features - Viaduct is built on `graphql-java`. - It supports fine-grained field selection through GraphQL selection sets. - It uses data-loading techniques and an intra-request cache. - Reliability features include short-circuiting and soft dependencies. - Field-level observability shows which services consume particular data. - Its GraphQL interface enables use of established open-source tooling and interactive development tools. Viaduct’s practical recommendation is to place a unified data schema at the center of the architecture, allowing the mesh—not individual consumers—to manage service composition. This can make large SOAs more modular, easier to evolve, and better suited to serverless execution.

Read original(opens in new tab)

MicroservicesServerlessGraphQLService Mesh+3

datadog·Aug 11, 2016·3 min readCurated summary

Consul at Datadog

Consul has become a critical part of Datadog’s production infrastructure for distributing configuration and discovering services. After 18 months of use, the main lesson is that Consul requires careful capacity planning, controlled access, efficient query patterns, and continuous monitoring. The recommendations aim to keep clusters stable while supporting frequent configuration updates and high-volume service discovery. ## Consul Server Capacity and CPU Consul servers use Raft consensus to elect a leader and coordinate the cluster. - Followers trigger a leadership transition if they cannot hear from the leader for 500 milliseconds. - Frequent leadership transitions usually indicate insufficient CPU capacity. - Datadog’s approximate sizing guidance: - `m3.large`: about 300 agent nodes - `c3.xlarge`: about 500 agent nodes - `c3.2xlarge`: about 800 agent nodes - If transitions occur hourly or more often, increase server CPU capacity until they happen no more than daily. - Standard monitoring may miss brief 500-millisecond CPU spikes, so reducing CPU pressure is important even when dashboards look normal. ## Auditable Configuration Changes Consul’s key-value store is useful for distributing configuration throughout a cluster. - Configuration can be retrieved through HTTP or delivered through Consul watches. - Direct edits without an audit trail make it difficult to determine who changed a value and when. - `git2consul` distributes configuration from a Git repository, providing version control and accountability. - Datadog uses it for cluster-wide configuration updates roughly every 60 seconds, dozens of times per day. ## Access Control with ACLs Consul ACLs prevent unauthorized processes from modifying or deleting key-value data. - Tokens should be limited to the data and operations each process requires. - Scoped permissions reduce the impact of accidental changes. - ACLs provide an important safety boundary between services and configuration areas. ## Watches Instead of Excessive Polling Consul can handle substantial traffic, but it should not be queried hundreds of thousands of times per second like Redis or Memcached. - Watches notify clients when key-value data changes. - This reduces unnecessary polling and distributes updates efficiently. - Watches can sometimes trigger unexpectedly or too frequently. - Tools such as `sifter` can help protect systems from excessive watch activity. ## Using dnsmasq for Service Discovery Applications using Consul’s DNS interface can reduce load by placing `dnsmasq` between clients and Consul. - Use short DNS TTLs; Datadog commonly uses 10 seconds. - Query `dnsmasq` rather than Consul directly so repeated answers can be cached locally. - At very high request volumes, cache Consul services in an additional hosts file loaded by `dnsmasq`. - This setup served over 100,000 DNS requests per second while sending only about 400 requests per second directly to Consul. - `goshe` can collect `dnsmasq` statistics for monitoring. ## Monitoring Cluster Health Monitoring is essential for operating Consul reliably. - `consul.consul.leader.reconcile.count` should remain stable and indicate that a leader exists. - `consul.serf.events.consul_new_leader` shows leadership transitions; frequent events suggest instability. - `consul.raft.leader.lastContact` measures how recently nodes contacted the leader. - `consul.consul.dns.domain_query.count` reveals how many DNS requests are reaching Consul directly. - Also monitor CPU and network usage on Consul servers. Consul works best when server nodes have sufficient CPU, configuration changes are managed through version control, ACLs restrict access, watches replace aggressive polling, and `dnsmasq` absorbs service-discovery traffic. Continuous monitoring of leadership, Raft connectivity, DNS load, CPU, and networking helps identify failures before they affect production.

Read original(opens in new tab)

MicroservicesMonitoringConfiguration ManagementConsul+4