Better tools made Copilot code review worse. Here’s how we actually improved it. (opens in new tab)

Copilot code review became more expensive and less effective after GitHub replaced its specialized exploration tools with shared grep, glob, and view tools. The tools themselves worked correctly, but their general-purpose instructions encouraged broad repository browsing rather than focused pull request investigation. After rewriting the instructions around diff-first review workflows, GitHub achieved roughly 20% lower average review cost without reducing review quality.

Why the Tool Migration Regressed

• Copilot code review previously used specialized tools for:
  • Listing directories
  • Searching files and directories
  • Reading code
• These tools often returned matching lines along with surrounding context, which suited earlier models that made fewer tool calls and needed more context per request.
• GitHub migrated to the shared Unix-inspired tools used by Copilot CLI and other products:
  • glob replaced list_dir
  • grep replaced search_file and search_dir
  • view replaced read_code
• The migration aimed to reduce duplicated implementations and let improvements benefit multiple Copilot products.
• Offline benchmarks showed higher review costs and fewer useful comments after the migration.

Repository Browsing Instead of Pull Request Review

• Execution traces showed the agent:
  • Searching broadly
  • Guessing file paths
  • Reading large sections of code
  • Finding more things to search
  • Carrying unnecessary context into later reasoning
• This workflow is reasonable for a coding assistant asked to understand an unfamiliar repository.
• It is inefficient for code review, where the agent should begin with the pull request diff and investigate a specific potential problem.
• Excessive tool output increases token usage because returned file contents remain in the agent’s context window.
• Broad exploration can also make the review less focused by mixing relevant evidence with unrelated code.

The Difference Between Coding and Reviewing

• A coding assistant may need to map a large area of a repository before editing code safely.
• A reviewer typically asks targeted questions based on the diff, such as:
  • Where is the changed function called?
  • Is a modified configuration key used elsewhere?
  • Does a similar test or helper already exist?
  • What is the smallest code range needed to understand the behavior?
• Copilot code review has a narrower objective:
  • Start from the pull request diff
  • Determine whether the change introduced a real issue
  • Gather only the evidence needed to confirm or dismiss that issue
• The shared tools were designed for broader interactive workflows, so their instructions unintentionally encouraged the wrong behavior in the review agent.

Instructions Were the Real Fix

• GitHub concluded that changing tools was not enough; the agent’s workflow instructions also had to change.
• The revised instructions emphasized:
  • Diff-first investigation
  • Targeted searches
  • Minimal surrounding context
  • Narrow evidence gathering
  • Avoiding unnecessary repository-wide exploration
• With these workflow changes, the shared tools became more effective for review rather than merely reproducing their coding-assistant behavior.
• The result was approximately 20% lower average review cost while preserving review quality.

The practical lesson is that tool quality cannot be evaluated separately from the instructions and workflow guiding an agent. Shared tools can work well across products, but each use case needs instructions that match its task—in this case, focused, evidence-driven pull request review rather than broad repository exploration.