Performance Testing | Techlist.io

kakaoMay 21, 20263 min readCurated summary

Stress Testing Know-How for Messaging Servers and How AI Lightened the Load

Kakao’s messaging platform team uses a continuously available stress-testing environment to identify scalability limits, failure points, and recovery behavior before production incidents occur. The setup mirrors production hardware, generates realistic traffic patterns with Locust, and tests both routine and extreme scenarios. The central lesson is that performance testing must examine not only application throughput, but also observability, infrastructure, framework choices, and domain-specific traffic behavior. ## Continuous Stress-Testing Environment - The environment has two main components: - Target servers using the same JVM heap, CPU, memory, and network specifications as production. - Load-generating clients built primarily with Locust, with workers scaled to hundreds of pods when necessary. - Client capacity is deliberately oversized so that the load generators do not become the bottleneck. - JMH may also be used for focused benchmarking. - Traffic scenarios are maintained according to realistic production ratios rather than simply generating large volumes of identical requests. - Typical scenarios include: - Normal midday traffic. - New Year’s midnight bursts, when message sending increases sharply. - Scenarios are built from configurable settings, allowing new traffic patterns to be created without rewriting load-generation code. ## What the Team Stress-Tests ### Observability and Logging Infrastructure - New or modified components such as Logstash, Fluent Bit, OpenTelemetry, and Vector are tested under production-like load. - The team checks: - Application throughput and elapsed time. - CPU, memory, and network overhead. - Delays in metrics collection and alerting. - Previous increases in application load caused by metric collection intervals demonstrated why monitoring infrastructure must also be performance-tested. ### Protocol and Framework Benchmarks - Server protocols and frameworks are benchmarked before changing business logic. - Tests isolate I/O behavior and compare alternatives such as WebFlux or virtual threads using real worker-count changes and system metrics. - CPU-bound work and I/O wait are increased separately to understand how each affects: - Requests per second. - Latency. - CPU utilization and other system resources. - During the C++-to-Kotlin migration, stress tests exposed system-metric differences and supported additional garbage-collection tuning. ### Operating-System and Security Changes - Host OS migrations and the addition of antivirus, monitoring, or security agents are tested under high load. - Stress tests have revealed issues such as slab-memory leaks and resource spikes caused by security software. - Components that appear harmless under normal traffic can materially affect high-throughput applications. ### Domain-Specific User Scenarios - Messaging systems have distinctive worst-case patterns, including: - Many users writing simultaneously in one chat room. - Midnight message bursts. - Entering group chats with hundreds of members. - These cases are reproduced by adjusting configurable load settings. - New features are stress-tested to locate bottlenecks before launch. ## Interpreting Test Metrics ### Endpoint-Level Metrics - **RPS:** Increase workers gradually to find saturation, or hold worker count constant to verify that throughput remains stable. - Unexpectedly low saturation points or sharply fluctuating RPS indicate a problem requiring deeper investigation. - **Latency:** P50 represents typical user experience, while P95 and P99 expose worst-case behavior. - Sudden P95/P99 increases may indicate internal capacity limits. - A degraded P50 can signal broader performance regression. - **Error rate:** Analyze 5xx errors, timeouts, and business errors separately. - 5xx responses may indicate server capacity exhaustion. - Timeouts may result from insufficient client resources. - 400-level errors can indicate broken test data or business logic. - Nonlinear changes in RPS or latency, or any unexpected errors, are signals to investigate lower-level system metrics. ## Practical Recommendation Maintain a production-like, always-available stress-testing environment with configurable realistic scenarios. Validate every major application, framework, observability, infrastructure, and feature change under both normal and worst-case traffic, then diagnose problems from endpoint metrics down through system resources.

Read original

Performance Testing Observability Kotlin Open Telemetry+4

figmaAug 28, 20232 min readCurated summary

Keeping Figma Fast | Figma Blog

Figma’s original performance-testing system relied on a single MacBook running a few repeated scenarios, which worked when the product and team were smaller. As Figma expanded into plugins, FigJam, Dev Mode, and many other features, that approach became too limited and fragile—the laptop eventually overheated while the company was remote. Figma concluded it needed a scalable, proactive testing framework capable of catching regressions across its growing codebase. ## From One Laptop to a Growing Product - In 2018, one MacBook repeatedly ran performance scenarios and reported timing changes to a shared dashboard. - A major renderer restructuring and WebAssembly fixes made Figma three times faster. - Over five years, the product grew substantially, adding: - Plugins and Community features - FigJam - Dev Mode - Numerous ongoing product updates - The existing test files could no longer represent the product’s expanding feature set and edge cases. ## Limits of the Original Testing Process - Granular performance tests measure specific behavior at scale, such as panning through a file while 100 users edit layers and type simultaneously. - Creating feature-specific tests became impractical as Figma grew beyond 400 engineers and managers. - Increasing release velocity made it difficult for any individual or single machine to track every performance-affecting change. - During the shift to remote work in 2020, the office MacBook remained running unattended and eventually overheated. - Attempts to reproduce the setup on another laptop were unsuccessful, demonstrating that the system was not operationally scalable. ## Requirements for a New System Figma used the overhaul to define an ideal performance-testing framework: - **Test every proposed code change:** Performance checks should run against changes in the main monorepo, allowing regressions to be found during development rather than after users encounter them. - **Support proactive performance work:** Small delays can significantly disrupt users who spend hours working in Figma. - **Run tests in parallel:** Dozens of stress scenarios would need to execute simultaneously, similar to Figma’s existing cloud-based CI testing. - **Finish quickly:** Performance guardrail checks were required to complete in under 10 minutes. - **Scale across real hardware:** Running every pull request on physical machines could require roughly 100 identical runners at peak capacity. Figma’s experience shows that performance testing must evolve alongside product complexity. A lightweight single-machine setup can be effective initially, but larger teams and faster release cycles require automated, parallel, hardware-aware testing integrated directly into CI.

Read original

Performance Testing CI/CD Webassembly Monorepo+2