githubJul 10, 2026

Better tools made Copilot code review worse. Here’s how we actually improved it. (opens in new tab)

Copilot code review became more expensive and less effective after GitHub replaced its specialized exploration tools with shared `grep`, `glob`, and `view` tools. The tools themselves worked correctly, but their general-purpose instructions encouraged broad repository browsing rather than focused pull request investigation. After rewriting the instructions around diff-first review workflows, GitHub achieved roughly 20% lower average review cost without reducing review quality. ## Why the Tool Migration Regressed - Copilot code review previously used specialized tools for: - Listing directories - Searching files and directories - Reading code - These tools often returned matching lines along with surrounding context, which suited earlier models that made fewer tool calls and needed more context per request. - GitHub migrated to the shared Unix-inspired tools used by Copilot CLI and other products: - `glob` replaced `list_dir` - `grep` replaced `search_file` and `search_dir` - `view` replaced `read_code` - The migration aimed to reduce duplicated implementations and let improvements benefit multiple Copilot products. - Offline benchmarks showed higher review costs and fewer useful comments after the migration. ## Repository Browsing Instead of Pull Request Review - Execution traces showed the agent: - Searching broadly - Guessing file paths - Reading large sections of code - Finding more things to search - Carrying unnecessary context into later reasoning - This workflow is reasonable for a coding assistant asked to understand an unfamiliar repository. - It is inefficient for code review, where the agent should begin with the pull request diff and investigate a specific potential problem. - Excessive tool output increases token usage because returned file contents remain in the agent’s context window. - Broad exploration can also make the review less focused by mixing relevant evidence with unrelated code. ## The Difference Between Coding and Reviewing - A coding assistant may need to map a large area of a repository before editing code safely. - A reviewer typically asks targeted questions based on the diff, such as: - Where is the changed function called? - Is a modified configuration key used elsewhere? - Does a similar test or helper already exist? - What is the smallest code range needed to understand the behavior? - Copilot code review has a narrower objective: - Start from the pull request diff - Determine whether the change introduced a real issue - Gather only the evidence needed to confirm or dismiss that issue - The shared tools were designed for broader interactive workflows, so their instructions unintentionally encouraged the wrong behavior in the review agent. ## Instructions Were the Real Fix - GitHub concluded that changing tools was not enough; the agent’s workflow instructions also had to change. - The revised instructions emphasized: - Diff-first investigation - Targeted searches - Minimal surrounding context - Narrow evidence gathering - Avoiding unnecessary repository-wide exploration - With these workflow changes, the shared tools became more effective for review rather than merely reproducing their coding-assistant behavior. - The result was approximately 20% lower average review cost while preserving review quality. The practical lesson is that tool quality cannot be evaluated separately from the instructions and workflow guiding an agent. Shared tools can work well across products, but each use case needs instructions that match its task—in this case, focused, evidence-driven pull request review rather than broad repository exploration.

agentic-aigithub-copilotcode-reviewpull-requests+2

githubJun 25, 2026

Evaluating performance and efficiency of the GitHub Copilot agentic harness across models and tasks (opens in new tab)

GitHub argues that an agentic harness—the system coordinating tools, context, and workflow—strongly influences how effectively an AI model solves software tasks. Its shared Copilot harness aims to match model-vendor harnesses in task completion while using fewer tokens. Benchmark results across four models suggest near-parity overall, with performance varying by model and task. ## Benchmarking Approach - GitHub evaluates the harness using public and internal benchmarks, plus real-world metrics and online experiments. - Comparisons hold the following constant: - The same underlying model - The same task - Context-window size - Reasoning effort - Tool selection - MCP servers - Copilot CLI is compared with: - Claude Code for Claude Sonnet 4.6 and Claude Opus 4.7 - Codex CLI for GPT-5.4 and GPT-5.5 - Benchmarks include: - **SWE-bench Verified:** 500 human-validated Python bug fixes - **SWE-bench Pro:** More complex, multi-step engineering tasks - **SkillsBench:** Skill usage and triggering - **TerminalBench:** Command-line workflows - **Win-Hill:** Software tasks in Windows containers ## Token Efficiency - Copilot’s harness generally completes tasks at rates comparable to model-vendor harnesses while consuming fewer tokens. - It performed better across the reported tests for Sonnet 4.6 and Opus 4.7. - For GPT-5.4 and GPT-5.5, Copilot performed better on nearly every benchmark. - The main exception was SWE-bench Verified, where Copilot was 7% worse with GPT-5.4 and 4% worse with GPT-5.5. ## Task Resolution - GitHub emphasizes that lower token use matters only when the agent successfully completes the work. - Overall task-resolution rates were considered on par with vendor harnesses. - Results varied by benchmark: - Copilot generally did better on SWE-bench Pro, especially with GPT models. - It outperformed vendor harnesses on Win-Hill or matched them for every model. - It performed better on TerminalBench with Sonnet and Opus, matched GPT-5.5, and trailed GPT-5.4. - SkillsBench results favored GPT models on Copilot but favored vendor harnesses for Claude models. - Differences are described as statistically comparable because model behavior is stochastic and run-to-run variation can explain many gaps. ## TerminalBench Variance Analysis - GitHub uses TerminalBench 2.0 to study both cost and completion rate over repeated runs. - The preferred outcome is higher resolution with lower cost. - The analysis illustrates that benchmark results should account for variance rather than relying on a single run. - GitHub presents Copilot as equal to or better than the vendor harnesses on this cost-versus-success comparison. The practical conclusion is that harness design is a reusable performance multiplier across Copilot products. GitHub’s results support using its shared harness when developers need broad model choice, efficient token usage, and comparable task-completion performance, while recognizing that the best harness can still vary by model and workload.

agentic-aillmgithub-copilotmcp+3

githubJun 17, 2026

Getting more from each token: How Copilot improves context handling and model routing (opens in new tab)

GitHub is improving Copilot’s efficiency by optimizing both how context is handled and which model performs each task. Prompt caching and deferred tool loading reduce repeated overhead in long agentic sessions, while Auto selects models based on task complexity and real-time system conditions. The goal is to preserve quality while using tokens, compute, and cost more intelligently. ## Prompt Caching and Deferred Tools - Copilot sessions repeatedly include instructions, repository context, conversation history, tool definitions, and task state. - Prompt caching allows repeated prompt prefixes to reuse model state instead of being recomputed on every request. - Tool search loads tool definitions only when needed, rather than placing every full schema into context on every turn. - This is especially valuable for sessions involving MCP tools, terminal commands, file operations, workspace search, and other actions. - The approach reduces fixed per-turn overhead while preserving access to a broad toolset. ## Auto Model Selection - Copilot Auto chooses a model based on the task rather than requiring developers to select one manually. - Quick explanations, focused edits, and complex multi-file changes often benefit from different levels of reasoning. - Evaluations found that no single model consistently performs best across all tasks. - Auto uses more efficient models when they can achieve comparable results and selects stronger reasoning models when the task demands them. - The objective is not simply lower cost, but matching model capability to the work. ## How Auto Routes Tasks Auto combines two signals: - **Real-time model health** - Considers availability, utilization, response speed, error rates, and cost. - Avoids selecting a model that is technically capable but currently overloaded or unreliable. - **Task-aware routing with HyDRA** - Evaluates reasoning depth, code complexity, debugging difficulty, and tool-orchestration requirements. - Identifies models capable of meeting the task’s quality threshold, then chooses the best fit among them. - The system can be tuned toward maximum quality or greater cost savings. ## Cache-Aware Routing - Switching models during every turn can reduce efficiency by invalidating cached prompt prefixes. - Auto therefore keeps the same model during a conversation so the cache can continue building. - It changes models at natural cache boundaries: - The first turn, when no cache exists yet. - After context compaction, when older turns are summarized and the prompt prefix resets. ## Multilingual Routing - Copilot’s routing system was trained on conversations spanning 16 language families, including CJK and European languages. - Evaluations covered production VS Code chat data across 19 languages. - Routing accuracy remained within four points of the English baseline across language groups. - No statistically significant quality gap was observed between the language groups. Copilot’s efficiency strategy combines smarter context management with adaptive model selection. Caching and on-demand tools reduce repeated work, while cache-aware Auto routing chooses an appropriate model without unnecessarily sacrificing quality or session efficiency.

agentic-aigithub-copilotmcpvisual-studio-code+3

githubJun 11, 2026

Making secret scanning more trustworthy: Reducing false positives at scale (opens in new tab)

Mariko is a Principal Applied Scientist at Microsoft who leads the development of agentic AI workflows for cybersecurity operations. Her work centers on using large language models and agentic systems to bring frontier AI research into practical products and operational environments. ## Professional Role - Principal Applied Scientist at Microsoft. - Leads agentic AI workflow development for cybersecurity operations. ## Research and Technical Focus - LLM-powered systems. - Agentic workflows. - Applying frontier AI research to real-world products and operations. Overall, Mariko’s work connects advanced AI research with practical cybersecurity and enterprise applications.

agentic-aillmcybersecurityai-workflows+1

githubMay 6, 2026

Validating agentic behavior when “correct” isn’t deterministic (opens in new tab)

Agentic systems such as GitHub Copilot cloud agent can complete tasks through multiple valid action sequences, making traditional deterministic tests unreliable. Timing changes, loading screens, and UI differences often produce false negatives even when the agent achieves the correct result. The post proposes an independent “Trust Layer” that validates essential outcomes and convergent behavior rather than rigid step-by-step execution paths. ## Challenges of Agent-Driven Validation - An agent may adapt to network delays or changing UI conditions and still complete its task successfully. - Conventional CI tests can fail when execution no longer matches a recorded script or expected assertion timing. - This creates a trust gap: - **False negatives:** successful tasks are reported as failures. - **Fragile infrastructure:** rendering, timing, and environment noise affect test results. - **Compliance trap:** valid behavioral variation is mistaken for regression. - Agent correctness should focus on reliably achieving essential outcomes, not reproducing an identical sequence of actions. ## Why Traditional Testing Breaks Down - **Assertion-based tests** require manually specifying every expected check and often omit valid alternative paths. - **Record-and-replay tools** are highly sensitive to timing and rendering differences. - **Visual regression tests** compare screenshots without understanding semantic meaning or the broader workflow. - **ML-based oracles** need large training datasets and generally provide little explanation for their decisions. - All four approaches assume correctness means following a stable sequence of observable states, which does not fit autonomous agents. ## Essential, Optional, and Convergent Behavior The proposed approach distinguishes between behavior that determines success and behavior that merely reflects environmental variation: - **Essential states:** Required milestones, such as reaching a VS Code “Search Results” screen. - **Optional variations:** Incidental states, including loading spinners or decorative UI changes. - **Convergent paths:** Different action sequences—such as using a keyboard shortcut or a menu—that eventually reach the same result. - A loading screen may appear in one run and not another, but the appearance of search results is what establishes success. ## Dominator Analysis The post connects this model to **dominator relationships** from compiler theory: - In a control-flow graph, node A dominates node B when every path to B must pass through A. - Applying dominator analysis to agent execution traces can identify: - Mandatory states - Optional states - Points where different execution paths converge - This produces a minimal and explainable definition of correctness instead of relying on every recorded step. ## Graph-Based Execution Modeling - Agent behavior should be represented as a graph rather than a linear script. - Graphs capture branching paths, optional states, and convergence points. - This structure provides a foundation for lightweight, explainable validation in GitHub Actions and other CI environments. A reliable validation system for agents should test whether essential outcomes occurred and whether critical invariants held, while ignoring harmless differences in timing, rendering, and execution order. This outcome-oriented Trust Layer can reduce false failures and make agentic workflows more dependable in production CI pipelines.

agentic-aici-cdgithub-copilotgithub-actions+3

githubApr 14, 2026

Hack the AI agent: Build agentic AI security skills with the GitHub Secure Code Game (opens in new tab)

Agentic AI tools can automate powerful tasks, but their autonomy creates new security risks, including prompt injection, tool misuse, memory poisoning, and compromised multi-agent workflows. GitHub’s Season 4 Secure Code Game teaches developers to recognize these threats by attacking and hardening ProdBot, a deliberately vulnerable terminal-based AI assistant. Its five levels progressively add capabilities—and corresponding attack surfaces—mirroring how real-world AI systems evolve. ## The Secure Code Game’s Evolution - The free, open-source, in-editor course teaches security by having players exploit and fix intentionally vulnerable code. - Earlier seasons covered: - General secure coding across JavaScript, Python, Go, and GitHub Actions. - LLM security, including malicious prompts and defensive techniques. - More than 10,000 developers from industry, academia, and open source have participated. - Season 4 shifts focus from AI that generates content to AI that independently browses, uses tools, calls APIs, and acts for users. ## Why Agentic AI Security Is Urgent - Agentic systems are moving rapidly from research projects into production environments. - The OWASP Top 10 for Agentic Applications identifies threats such as: - Goal hijacking - Tool misuse - Identity abuse - Memory poisoning - A Dark Reading poll found that 48% of cybersecurity professionals expect agentic AI to be the leading attack vector by the end of 2026. - Cisco reported that although 83% of organizations planned to deploy agentic AI, only 29% felt prepared to secure it. - The article argues that learning to think like an attacker is essential for closing this readiness gap. ## ProdBot: A Deliberately Vulnerable AI Assistant - ProdBot is a terminal-based productivity and coding assistant inspired by tools such as OpenClaw and GitHub Copilot CLI. - It can: - Convert natural-language requests into bash commands. - Browse a simulated web. - Connect to MCP servers. - Run organization-approved skills. - Store persistent memory. - Coordinate multiple agents. - Players’ objective is to use natural language to make ProdBot reveal the contents of `password.txt`. - No prior AI or coding experience is required; all interaction takes place through the CLI. ## Five Progressive Attack Surfaces - **Level 1: Shell execution** - ProdBot runs generated bash commands in a sandbox. - The challenge is to determine whether the sandbox can be escaped. - **Level 2: Web browsing** - ProdBot reads simulated news, finance, sports, and shopping sites. - Untrusted web content introduces risks such as instruction hijacking and prompt injection. - **Level 3: MCP integrations** - ProdBot gains access to external tool providers for stock quotes, browsing, and cloud backup. - Additional tools increase both functionality and opportunities for abuse. - **Level 4: Skills and memory** - Organization-approved plugins and persistent memory create layered trust relationships. - The level tests whether trusted skills and stored information are actually safe. - **Level 5: Multi-agent orchestration** - ProdBot combines six specialized agents, three MCP servers, three skills, and a simulated open-source project. - Claims that agents are sandboxed and data is pre-verified become assumptions to test rather than guarantees. ## Real-World Relevance - The game’s vulnerabilities reflect active security concerns in deployed autonomous AI systems rather than purely theoretical exercises. - The article cites CVE-2026-25253, known as “ClawBleed,” an OpenClaw vulnerability rated CVSS 8.8. - The flaw allowed attackers to steal authentication tokens through a malicious link and gain full control of an OpenClaw instance. - Season 4’s broader goal is to develop instincts for identifying similar weaknesses during architecture reviews, tool-integration audits, and production deployments. Developers working with AI agents should treat every new capability—shell access, browsing, plugins, memory, or collaboration—as a potential attack surface. Practicing these failure modes in a controlled environment like the Secure Code Game can help teams design safer agentic systems before deploying them.

agentic-aimodel-context-protocolai-securityprompt-injection+2

githubMar 5, 2026

60 million Copilot code reviews and counting (opens in new tab)

Copilot code review has grown tenfold since launch, surpassing 60 million reviews and accounting for more than one in five GitHub code reviews. GitHub argues that effective AI review is not about maximum coverage or comment volume, but about accurate, actionable feedback delivered quickly enough to support development. Its newer agentic architecture, informed by user feedback and continuous evaluation, is designed to improve context, reduce noise, and help teams merge with greater confidence. ## Redefining a “Good” Code Review - GitHub’s focus has shifted from exhaustive review coverage to high-signal feedback that helps pull requests move forward. - The system evaluates reviews across three dimensions: - **Accuracy:** Identifying consequential logic and maintainability problems. - **Signal:** Prioritizing useful findings over a high number of comments. - **Speed:** Providing a timely first pass while accepting some latency for deeper analysis. ## Measuring Accuracy - Copilot combines internal tests against known code issues with production data from real pull requests. - Key production indicators include: - Developer thumbs-up and thumbs-down reactions. - Whether flagged issues are fixed before the pull request is merged. - GitHub says these measures help distinguish useful scrutiny from feedback that merely slows development. ## Prioritizing Signal Over Volume - Copilot produces actionable feedback in 71% of reviews and remains silent in the other 29% when it finds nothing worth reporting. - It now averages approximately 5.1 comments per review without increasing review churn or lowering quality standards. - Examples of high-signal findings include missing React hook dependencies and retry loops that could run indefinitely when an API returns HTTP 429 without a `Retry-After` header. ## Trading Some Speed for Better Reasoning - GitHub treats latency as a deliberate trade-off: deeper analysis is preferable to fast but noisy feedback. - A recent switch to a more advanced reasoning model increased positive feedback by 6% while increasing review latency by 16%. - The team continues to optimize speed, but not at the expense of findings developers can trust. ## Agentic Architecture and Repository Context - The redesigned system retrieves context, explores repositories, and reasons about architecture and invariants instead of examining changes in isolation. - This architectural shift produced an initial 8.1% increase in positive feedback. - Improvements include: - Identifying issues during analysis rather than waiting until the end, reducing forgotten findings. - Retaining memory across reviews to recognize recurring patterns. - Creating explicit plans for long or complex pull requests. - Reading linked issues and pull requests to compare code against project requirements. ## Making Reviews Easier to Navigate - Multi-line comments attach feedback to logical code ranges, making problems and suggested fixes easier to understand. - Related comments are clustered into a single unit instead of cluttering the pull request timeline. - Batch autofixes allow developers to resolve entire classes of bugs or style issues at once. - More than 12,000 organizations automatically run Copilot code review on every pull request. Copilot code review is most valuable when treated as a trusted first-pass reviewer rather than a replacement for human judgment. Teams should favor configurations and workflows that maximize actionable findings, preserve developer context, and accept modest delays when they produce materially better reviews.

agentic-aireactgithub-copilotcode-review+4

githubFeb 24, 2026

Multi-agent workflows often fail. Here’s how to engineer ones that don’t. (opens in new tab)

Multi-agent workflows often fail because agents make implicit assumptions about state, ordering, and intended actions. The post argues that these systems should be engineered like distributed software rather than treated as chat interfaces. Typed schemas, explicit action definitions, and MCP-enforced interfaces make agent behavior more predictable and failures easier to contain. ## Typed Schemas Prevent Data Drift - Natural-language exchanges and inconsistent JSON lead to changing field names, mismatched types, and ambiguous payloads. - Typed interfaces define machine-checkable contracts, such as a `UserProfile` with fixed fields and allowed plan values. - Schema violations can fail fast, triggering retries, repairs, or escalation before invalid state spreads. - Debugging becomes contract-based instead of dependent on inspecting logs and guessing. ## Action Schemas Clarify Intent - Agents cannot reliably infer what “take action” means; they may assign, close, escalate, or do nothing. - Action schemas restrict outcomes to explicit, valid choices such as: - Requesting more information - Assigning an issue - Closing an issue as a duplicate - Taking no action - A discriminated union or similar structure ensures every agent returns one recognized action. - Invalid or ambiguous actions can be rejected, retried, or escalated. ## MCP Enforces Agent Interfaces - Schemas and action definitions are only conventions unless consistently enforced. - Model Context Protocol (MCP) provides explicit input and output schemas for tools and resources. - Calls are validated before execution, preventing agents from inventing fields, omitting required inputs, or drifting between interfaces. - MCP therefore acts as the enforcement layer for both data structure and intended behavior. Reliable multi-agent systems require explicit contracts at every boundary. Engineers should treat agents like code components: define their data and actions precisely, enforce interfaces with mechanisms such as MCP, and prevent invalid state from propagating.

agentic-aimodel-context-protocolgithub-copilotdistributed-systems+2