From single pull requests to full software packages: Detecting malicious code at scale

BewAIre evolved from a pull-request malware detector into a system for scanning dependency packages and upstream registries. Its core improvement is a two-stage pipeline: a cheap LLM filter handles routine changes, while a more capable agent investigates suspicious cases using external tools and repository context. This approach raised accuracy from 97.4% to 99.86%, eliminated false positives in a 690-diff sample, and reduced latency and cost through early exits.

Expanding Beyond Pull Requests

• Software supply-chain attacks increasingly compromise trusted dependencies such as axios, LiteLLM, and Mistral.
• BewAIre initially focused on detecting malicious pull requests, identifying security testing, bug-bounty activity, and real attacks such as the Hackerbot campaign.
• The team aimed to apply the same LLM-based detection to complete packages and package registries without sacrificing accuracy, latency, or predictable cost.

Limits of Single-Pass LLM Evaluation

• BewAIre began as a basic “LLM-as-judge” system that analyzed diffs through an inference API.
• More capable reasoning models improved detection but increased costs.
• Large diffs, especially dependency upgrades, challenged context-window limits.
• Two changes addressed these limitations:
  • A filter-then-review escalation path.
  • Tool-enabled investigation allowing models to gather additional evidence.

Two-Stage Filtering and Investigation

• The filter phase:
  • Runs on every change using a fast, inexpensive model.
  • Uses straightforward prompts and diff chunking for large changes.
  • Produces a binary suspicious/benign decision.
  • Ends processing immediately when a change appears benign.
• The investigation phase:
  • Runs only when the filter raises a concern.
  • Uses a stronger reasoning model in an agentic loop.
  • Can inspect commits, files, contributor histories, dependency metadata, and commit ranges through GitHub APIs.
  • Checks for reverted commits, typosquatting, suspicious contributor behavior, and dependency risks using sources such as osv.dev and Datadog SCA.

Detecting Obfuscated Attacks

• In the Hackerbot Claw example, the system identified a malicious filename containing shell command substitution.
• A base64-encoded payload decoded to a curl ... | bash command that downloaded and executed remote code.
• The investigation agent added useful context:
  • The contributor account was newly created, had no profile information, and had no followers.
  • The pull request had no reviews or approvals.
  • ${IFS} obfuscation was used to evade security filters.
• Combining code analysis with repository and author context made the final assessment more precise.

Combining LLMs with Static Checks

• The filter model could mistakenly treat Datadog-like typosquatting domains as legitimate without access to investigative tools.
• BewAIre added preprocessing that extracts domains and compares them against a static list of known typosquatting variants.
• This hybrid design improves reliability while avoiding the cost and nondeterminism of performing every check through a powerful LLM.

Measured Results

• Accuracy improved from 97.4% to 99.86% across 690 representative test diffs.
• False positives fell from 17 to zero.
• Most benign changes exit during the inexpensive filter stage.
• Suspicious changes still receive deeper analysis, preserving broad coverage while controlling latency and cost.

The practical recommendation is to combine inexpensive broad screening with selective, tool-driven investigation. Static security checks should complement LLM reasoning, especially for predictable threats such as domain typosquatting.