How we migrated our static analyzer from Java to Rust
Datadog migrated its static analyzer from Java to Rust after finding that ANTLR-based parsing was too slow and language support was incomplete. Rust’s strong integration with Tree-sitter enabled broader language coverage, faster scans, and lower memory usage. The migration preserved behavioral parity while tripling performance and reducing memory consumption tenfold. ## Why Performance Became a Priority - Datadog runs analysis directly in customers’ CI environments, often on resource-constrained runners. - On a two-core, 7 GB GitHub Actions runner, medium repositories took about five minutes to scan instead of the target of under three minutes. - Codiga’s previous hosted environment used large, tuned servers, which masked some performance problems. - Java also required customers to use JVM 17+, potentially conflicting with JVM versions already installed in their CI environments. - Improving Java offered limited upside, so the team considered a rewrite despite its cost and risk. ## Static Analyzer Architecture - The analyzer consists primarily of: - A parsing layer that builds an abstract syntax tree (AST). - An execution layer that analyzes the AST, reports violations, and offers fixes. - Tree-sitter generates the AST. - The existing Java binding lacked important functionality, including Tree-sitter pattern matching. - Tree-sitter’s core libraries are implemented in Rust, where support was more complete. - Analysis rules are written in JavaScript and were originally executed through GraalVM’s polyglot capabilities. - Fast parsing, pattern matching, and rule execution were central to meeting the desired CI performance. ## Migrating from Java to Rust - Rust was selected because it is a first-class part of the Tree-sitter ecosystem and provided better access to its features. - The migration required: - Feature parity with the Java implementation. - Identical analysis results and reported violations. - No execution-time regressions. - Migrating the parser was relatively straightforward because Rust support came directly from Tree-sitter. - The Rust implementation: - Tripled analyzer performance. - Reduced memory usage by a factor of ten. - JavaScript execution moved from GraalVM to `deno-core`, a Rust-based V8 integration. - Only the core JavaScript functionality was included. - Disk and network capabilities were excluded because analysis rules do not need them, improving security. ## Migration Strategy and Rust Adoption - The team treated automated equivalence and performance tests as requirements for a successful rewrite. - Rust allowed the analyzer to integrate more directly with its key dependencies rather than maintaining a separate Java binding. - The broader migration also required replacing supporting Java components with corresponding Rust libraries; the article indicates that these mappings were documented as part of the transition. Overall, the move to Rust was justified by the analyzer’s deployment model: faster execution and lower resource consumption directly improved the experience of customers running scans in constrained CI environments.
Read original(opens in new tab)