githubJul 22, 2026

Copilot vs. raw API access: What are you actually paying for? (opens in new tab)

GitHub Copilot and direct model APIs serve different purposes rather than competing at the same layer. Copilot bundles model access with development workflows—repositories, editors, terminals, issues, pull requests, and organizational controls—while APIs give teams the primitives to build their own systems. The best choice depends on whether you want to own the surrounding infrastructure or use GitHub’s integrated tooling. ## Copilot as Development Tooling - Copilot supports workflows from GitHub Issues through code changes, testing, pull requests, and review. - Its value includes integration with: - Editors and repositories - Terminals and permitted commands - Repository instructions - Pull requests and organizational policies - Paid plans include code completions and Next Edit Suggestions, while more intensive chat and agentic tasks consume AI Credits. - Actual cost depends on context selection, input/output/cached tokens, tool calls, retries, and task complexity. - Organization plans pool credits and provide budgets and usage tracking through the billing dashboard. ## Raw APIs for Systems You Control - Direct API access is suited to product features, internal agent platforms, evaluation systems, and automation pipelines. - Teams control prompts, retrieval, model routing, retries, logging, security, credentials, and billing. - Production agents still require substantial engineering, including: - Selecting relevant repository or document context - Preserving instructions - Handling failed tool calls - Storing traces and audit records - Defining data boundaries and approval points - Agent SDKs can provide orchestration, tools, sessions, and streaming. GitHub’s Copilot SDK exposes the runtime used by Copilot CLI and can run with either a Copilot subscription or a provider key. ## BYOK: Keeping Copilot’s Workflow - Copilot’s public-preview Bring Your Own Key feature lets teams use supported external models in Copilot Chat, CLI, and VS Code. - Supported providers include Anthropic, AWS Bedrock, Google AI Studio, Microsoft Foundry, OpenAI, OpenAI-compatible services, and xAI. - GitHub continues to provide the Copilot harness and integrations, while the customer pays the model provider directly. - BYOK can preserve existing cloud contracts or provider commitments while maintaining a familiar Copilot workflow. - Administrators can control which GitHub-hosted or BYOK models teams may use. - Because BYOK is still in public preview, teams should consult the current documentation before making purchasing or architecture decisions. ## Choosing the Right Layer - Choose raw API access when you need custom integrations, behavior, security controls, auditing, or billing. - Choose Copilot when developers primarily need to work faster within existing repositories, editors, terminals, issues, pull requests, reviews, and security processes. - BYOK is a middle option for teams that want GitHub’s development workflow but prefer to pay for models through an existing provider relationship. The practical decision is not simply about token price. It is about whether your team needs to build and operate the surrounding AI system or wants an integrated development workflow managed through Copilot.

llmgithub-copilotapiagentic-workflows+4

githubJun 25, 2026

Evaluating performance and efficiency of the GitHub Copilot agentic harness across models and tasks (opens in new tab)

GitHub argues that an agentic harness—the system coordinating tools, context, and workflow—strongly influences how effectively an AI model solves software tasks. Its shared Copilot harness aims to match model-vendor harnesses in task completion while using fewer tokens. Benchmark results across four models suggest near-parity overall, with performance varying by model and task. ## Benchmarking Approach - GitHub evaluates the harness using public and internal benchmarks, plus real-world metrics and online experiments. - Comparisons hold the following constant: - The same underlying model - The same task - Context-window size - Reasoning effort - Tool selection - MCP servers - Copilot CLI is compared with: - Claude Code for Claude Sonnet 4.6 and Claude Opus 4.7 - Codex CLI for GPT-5.4 and GPT-5.5 - Benchmarks include: - **SWE-bench Verified:** 500 human-validated Python bug fixes - **SWE-bench Pro:** More complex, multi-step engineering tasks - **SkillsBench:** Skill usage and triggering - **TerminalBench:** Command-line workflows - **Win-Hill:** Software tasks in Windows containers ## Token Efficiency - Copilot’s harness generally completes tasks at rates comparable to model-vendor harnesses while consuming fewer tokens. - It performed better across the reported tests for Sonnet 4.6 and Opus 4.7. - For GPT-5.4 and GPT-5.5, Copilot performed better on nearly every benchmark. - The main exception was SWE-bench Verified, where Copilot was 7% worse with GPT-5.4 and 4% worse with GPT-5.5. ## Task Resolution - GitHub emphasizes that lower token use matters only when the agent successfully completes the work. - Overall task-resolution rates were considered on par with vendor harnesses. - Results varied by benchmark: - Copilot generally did better on SWE-bench Pro, especially with GPT models. - It outperformed vendor harnesses on Win-Hill or matched them for every model. - It performed better on TerminalBench with Sonnet and Opus, matched GPT-5.5, and trailed GPT-5.4. - SkillsBench results favored GPT models on Copilot but favored vendor harnesses for Claude models. - Differences are described as statistically comparable because model behavior is stochastic and run-to-run variation can explain many gaps. ## TerminalBench Variance Analysis - GitHub uses TerminalBench 2.0 to study both cost and completion rate over repeated runs. - The preferred outcome is higher resolution with lower cost. - The analysis illustrates that benchmark results should account for variance rather than relying on a single run. - GitHub presents Copilot as equal to or better than the vendor harnesses on this cost-versus-success comparison. The practical conclusion is that harness design is a reusable performance multiplier across Copilot products. GitHub’s results support using its shared harness when developers need broad model choice, efficient token usage, and comparable task-completion performance, while recognizing that the best harness can still vary by model and workload.

llmgithub-copilotagentic-aimcp+3

githubJun 15, 2026

GitHub Copilot CLI for Beginners: Overview of common slash commands (opens in new tab)

GitHub Copilot CLI’s slash commands provide a central way to control sessions directly from the terminal. They help users select models, manage context, inspect changes, switch projects, resume work, and control permissions. The article recommends typing `/` to explore the available commands and build a more deliberate CLI workflow. ## Slash Commands as Copilot CLI Controls - Slash commands are built-in terminal controls for: - Guiding Copilot’s behavior - Inspecting project changes - Managing conversation context - Moving between sessions and repositories - Resetting tool permissions - Typing `/` displays a scrollable list of supported commands. ## Selecting a Model - `/model` opens the list of available models. - Model information includes: - Capabilities for lightweight tasks or deeper reasoning - Availability based on the user’s plan or organization - Cost multipliers indicating relative usage cost - Choosing an appropriate model can balance speed, quality, and cost. ## Managing Context and Tokens - `/context` shows remaining tokens, system usage, and available buffer. - `/compact` summarizes the current conversation to free context space without starting over. - Copilot may compact automatically near the context limit, but users can trigger it manually when changing tasks. - `/clear` completely resets the current session. ## Resuming Previous Sessions - `/resume` lists earlier local and remote sessions. - Users can select a session to review its history and continue where they left off. ## Reviewing Changes - `/diff` displays recent modifications made during the session. - This helps users inspect and validate Copilot’s changes before proceeding. ## Switching Projects - `/cwd` changes Copilot’s current working directory. - It allows users to move between repositories or project directories without exiting the CLI. ## Resetting Permissions - `/reset-allowed-tools` removes previously granted permissions for actions such as editing files. - This is useful when switching to a repository that requires more cautious access controls. Users should start by typing `/` in Copilot CLI to discover available commands, then use commands such as `/model`, `/context`, `/compact`, `/diff`, and `/cwd` to maintain control over their coding sessions.

llmgithub-copilot-clislash-commandscommand-line-interface+2

githubJun 12, 2026

How we made GitHub Copilot CLI more selective about delegation (opens in new tab)

GitHub improved Copilot CLI by making subagent delegation more selective rather than treating delegation as inherently beneficial. The new orchestration policy keeps narrow tasks with the main agent, delegates broad or independent work, and encourages parallel execution instead of waiting. After full production rollout, it reduced tool failures by 23% and improved high-percentile wait times without reducing quality. ## The Cost of Unnecessary Delegation - Subagents help with complex investigations, large repositories, and parallel work, but every handoff adds tool calls, coordination, and latency. - Copilot sometimes delegated simple, well-scoped tasks that the main agent could complete directly. - Common problems included: - Repeated or overlapping repository searches. - Subagents re-discovering context already available to the main agent. - Sequential delegation that left the main agent idle. - Stale paths, incorrect relative paths, and workspace mismatches. - The result was slower execution and more tool failures for tasks that should have required only a few steps. ## How the Problem Was Identified - GitHub used LLMs to analyze complete agent trajectories rather than manually reviewing sessions. - The analysis found that delegation was frequently used for narrow, obvious, or fully described tasks. - This led to a clear target: - Keep focused discovery-and-edit work with the main agent. - Reserve subagents for broad exploration, cross-cutting tasks, or genuinely independent work. ## A More Selective Orchestration Policy - Copilot now starts with the narrowest effective workflow: - Find and read the relevant file. - Make the targeted change. - Verify the result. - Delegation becomes appropriate when additional context, uncertainty, or parallel execution creates real value. - Subagents are treated as a parallelism mechanism, not a reason for the main agent to pause. - Handoffs should clearly specify: - The user’s request. - What the main agent already knows. - Which work the subagent owns. - What result the subagent should return. ## Evaluation and Production Results - GitHub tested the change with generated regression cases and existing benchmarks before rollout. - Staff and public A/B tests measured reliability, responsiveness, subagent workload, and quality. - Production results showed: - 23% fewer tool failures per session. - 27% fewer search-tool failures. - 18% fewer edit-tool failures. - 5% lower P95 wait time. - 3% lower P75 wait time. - No quality regression. - The improvements came mainly from avoiding unnecessary subagent paths and reducing orchestration overhead, not from making individual model calls faster. Copilot CLI users can access the improvement by running `/update` and upgrading to version 1.0.42 or later. The broader recommendation is to delegate selectively: use the main agent for focused tasks and subagents only when independent context or parallel work provides meaningful leverage.

llmgithub-copilot-cliorchestrationab-testing+2

githubJun 11, 2026

Making secret scanning more trustworthy: Reducing false positives at scale (opens in new tab)

Mariko is a Principal Applied Scientist at Microsoft who leads the development of agentic AI workflows for cybersecurity operations. Her work centers on using large language models and agentic systems to bring frontier AI research into practical products and operational environments. ## Professional Role - Principal Applied Scientist at Microsoft. - Leads agentic AI workflow development for cybersecurity operations. ## Research and Technical Focus - LLM-powered systems. - Agentic workflows. - Applying frontier AI research to real-world products and operations. Overall, Mariko’s work connects advanced AI research with practical cybersecurity and enterprise applications.

llmagentic-aicybersecurityai-workflows+1

githubMay 15, 2026

Building a general-purpose accessibility agent—and what we learned in the process (opens in new tab)

GitHub is piloting a general-purpose accessibility agent that answers accessibility questions and automatically fixes straightforward issues in front-end code. The agent has reviewed 3,535 pull requests and resolved 68% of identified issues, especially problems involving structure, control names, status messages, text alternatives, and keyboard focus. GitHub’s experience shows that an accessibility agent is most effective as an augmentation of human expertise, supported by a strong foundation of manually documented accessibility work. ## Goals and Results - The agent serves two purposes: - Provide just-in-time accessibility guidance through GitHub Copilot CLI and VS Code. - Detect and automatically remediate simple, objective accessibility issues before production. - It evaluates pull requests that modify front-end code. - Its five most common issue categories are: - Making structure and relationships understandable to assistive technologies. - Giving interactive controls clear, concise names. - Ensuring users receive important status announcements. - Providing text alternatives for non-text content. - Maintaining a logical keyboard focus order. - Example fixes can identify mismatches between visual order and screen-reader reading order, then suggest code changes that developers can commit directly. ## An Augmenting, Not Universal, Tool - GitHub frames accessibility through the social model of disability: barriers are often created by how digital environments are designed and built. - The agent is intended to help engineers remove those barriers, not “solve” accessibility independently. - It is not a silver bullet capable of handling every accessibility scenario. - Clearly limiting its responsibility helped GitHub launch the experiment more quickly and gain broader internal support. ## Why Manual Accessibility Work Matters - New and upcoming regulations, including the European Accessibility Act and the Americans with Disabilities Act’s planned WCAG 2.1 AA requirements, increase the importance of accessibility investment. - Organizations without established processes for manually identifying and fixing accessibility problems will be at a disadvantage. - GitHub already had a mature issue-management process containing: - Structured problem reports. - Reproduction steps. - Severity, service-area, and WCAG metadata. - Links to fixing pull requests. - Acceptance criteria. - Centralizing these issues in one repository made the collection a valuable reference corpus for the agent. ## Using Historical Issues as Training Material - The agent examines past accessibility issues and related pull requests to find applicable code and language patterns. - LLMs’ fuzzy matching can be useful here because it helps connect new problems with similar historical examples. - Generic instructions such as “follow accessibility best practices” are insufficient. - LLMs often reproduce accessibility antipatterns because their training data contains decades of inaccessible code. - Manually cataloged issues and organization-specific fixes provide contextual examples that are more useful than short, generic accessibility checklists. GitHub’s experience suggests that teams should first build reliable human processes for reporting and remediating accessibility issues. Once that structured knowledge exists, an agent can help apply it consistently and efficiently—while remaining a complement to, rather than a replacement for, accessibility expertise.

llmaccessibilitygithub-copilotgithub-actions+3

githubMay 7, 2026

Improving token efficiency in GitHub Agentic Workflows (opens in new tab)

GitHub’s Agentic Workflows can quietly accumulate substantial token costs because they run automatically in CI. GitHub improved efficiency by instrumenting token usage, auditing workflows, pruning unused MCP tools, and replacing many MCP data-fetching calls with deterministic GitHub CLI commands. Early results show that reducing context and removing unnecessary LLM reasoning can save thousands of tokens per run, though measuring true efficiency requires accounting for model choice and workload quality. ## Logging Token Usage - GitHub runs hundreds of agentic workflows against real GitHub Actions limits. - Different agent frameworks produced incompatible usage logs, so GitHub used its API proxy to normalize data across Claude CLI, Copilot CLI, and Codex CLI. - Each workflow now emits a `token-usage.jsonl` artifact containing: - Input, output, cache-read, and cache-write tokens - Model and provider - Timestamps - One record per API call - These records make it possible to compare historical runs and identify recurring sources of waste. ## Automated Auditing and Optimization - A daily **Token Usage Auditor** aggregates recent usage by workflow and reports: - Significant increases in token consumption - The most expensive workflows - Anomalous runs, such as a workflow taking 18 LLM turns instead of its usual four - A daily **Token Optimizer** examines flagged workflows, their source YAML, and recent logs. - It creates GitHub Issues with concrete inefficiencies and recommended fixes. - The auditing workflows also consume tokens, creating a feedback loop in which their own costs are monitored. ## Removing Unused MCP Tools - MCP tool names and JSON schemas are typically included in every stateless LLM request. - A GitHub MCP server with roughly 40 tools can add 10–15 KB of schema to every turn. - If a workflow uses only two tools, the other 38 create repeated overhead without adding value. - GitHub compares configured tools with actual tool calls and recommends removing unused registrations. - In smoke tests, pruning tools reduced each call’s context by 8–12 KB and saved several thousand tokens per run without changing behavior. ## Replacing MCP Calls with GitHub CLI - GitHub found larger savings by replacing MCP calls for predictable data retrieval—such as pull request diffs, file contents, and review comments—with `gh` commands. - MCP calls require an additional reasoning cycle: the model chooses a tool, constructs arguments, and processes the response. - Commands such as `gh pr diff` make deterministic API requests without involving the LLM in the retrieval step. Two migration patterns were used: - **Pre-agentic downloads** - Workflow setup steps run `gh` commands before the agent starts. - Results such as diffs and changed-file lists are saved to workspace files. - The agent reads the files directly, eliminating MCP round trips. - **In-agent CLI proxy substitution** - When data must be selected dynamically, the agent runs commands such as `gh pr view --json`. - A transparent proxy routes CLI requests to GitHub’s API without exposing credentials. - This preserves the zero-secrets security model while avoiding MCP overhead. ## Measuring Efficiency - Lower token counts do not necessarily mean better workflows; a workflow may simply be doing less work. - Model selection also affects cost. Claude Haiku and Sonnet may use similar numbers of tokens, but Haiku is substantially cheaper. - GitHub therefore uses an **Effective Tokens (ET)** metric that weights usage by token type and model cost: ```text ET = m × (1.0 × I + 0.1 × C + 4.0 × O) ``` - `m` represents the model multiplier: Haiku `0.25×`, Sonnet `1.0×`, and Opus `5.0×`. - `I` is newly processed input, `C` is cache-read tokens, and `O` is output tokens. - Output tokens receive greater weight because they are typically the most expensive component. GitHub’s experience suggests that agentic workflow authors should measure usage continuously, remove tools that workflows do not actually use, and move routine API retrieval outside the LLM reasoning loop wherever possible.

llmmodel-context-protocolgithub-actionsyaml+3

githubMay 7, 2026

Agent pull requests are everywhere. Here’s how to review them. (opens in new tab)

Agent-generated pull requests are increasing rapidly, while human review capacity remains limited. Although these changes often look clean and pass CI, research suggests they can introduce more redundancy and technical debt—and reviewers may be more likely to approve them. The solution is not to review more slowly, but to focus human judgment on risks agents are least equipped to recognize. ## The Scale of Agent-Generated Pull Requests - GitHub Copilot code review has processed more than 60 million reviews and grown tenfold in under a year. - More than 20% of GitHub code reviews now involve an agent. - Developers can launch many agent sessions simultaneously, causing pull-request volume to grow faster than human review capacity. - Reviewers therefore need a deliberate method for identifying high-impact issues. ## Understanding the Agent’s Limitations - Coding agents are productive and literal, but lack: - Incident history - Team-specific edge-case knowledge - Operational constraints not documented in the repository - Agents can produce code that appears complete while quietly embedding incorrect assumptions. - Human reviewers provide the context and judgment that automated tools cannot fully replicate. ## CI Gaming Agents may weaken CI when their changes fail, for example by removing tests, skipping linting, or adding commands such as `|| true`. Reviewers should verify: - Coverage thresholds were not reduced. - Tests were not removed, renamed, or skipped. - Workflows still run for forks and pull requests. - CI steps were not placed behind new restrictive conditions. Any such change requires explicit justification before approval. ## Blindness to Existing Code Reuse Agents may copy patterns from nearby code without discovering equivalent utilities elsewhere in the repository. Warning signs include: - Duplicate helper or utility functions - Reimplemented validation logic - New middleware duplicating shared modules - “Almost identical” helpers with different names Reviewers should search for existing implementations and require consolidation rather than merely commenting on duplication. For larger agent pull requests, requiring justification for new utilities can prevent redundant code from becoming future “prior art.” ## Hallucinated Correctness The most dangerous agent errors are not obvious API or syntax failures. They are changes that compile, pass tests, and still behave incorrectly under conditions such as: - Pagination boundaries - Missing permission checks - Validation edge cases - Race conditions at scale Reviewers should trace a critical path from input to output, checking empty, zero, and maximum values, external input validation, permissions on every branch, and unusual conditionals. A claimed bug fix should include a test that fails before the change; otherwise, the fix or the agent’s understanding may be incomplete. ## Agentic Ghosting and Oversized Pull Requests Large, poorly structured agent pull requests are more likely to become abandoned or misaligned. Before conducting an in-depth review, check: - Whether the agent has responded usefully in earlier review rounds - Whether the pull request includes a clear implementation plan - Whether the changes can be divided into smaller, scoped units If no plan exists, request a breakdown or a clear explanation of each component before spending time on detailed comments. ## Untrusted Input in Agent Workflows Workflows that send pull-request bodies, issue content, or commit messages to an LLM can create prompt-injection risks—especially when model output is later executed with `GITHUB_TOKEN` permissions. Reviewers should block workflows that: - Interpolate untrusted content into prompts without sanitization - Grant write access when read-only permissions are sufficient - Execute model output as shell commands without validation - Expose secrets to agent steps or logs Safer designs should use least-privilege permissions such as `permissions: read-all`, sanitize and quote untrusted content, separate analysis from execution, and require human approval before actions affecting production. Agent pull requests should not automatically receive either extra trust or blanket suspicion. Reviewers should focus on CI integrity, reuse, behavior under edge cases, reviewability, and workflow security—the areas where contextual human judgment adds the most value.

llmci-cdgithub-copilotcode-review+3

githubMar 6, 2026

How to scan for vulnerabilities with GitHub Security Lab’s open source AI-powered framework (opens in new tab)

GitHub Security Lab’s open-source Taskflow Agent uses AI-driven, multi-step auditing workflows to find high-impact vulnerabilities in web applications and open-source projects. The authors report more than 80 vulnerabilities, including authorization bypasses and private-data disclosures, with about 20 already disclosed. They argue that carefully designed taskflows and prompts can give LLMs enough freedom to discover vulnerabilities while reducing hallucinations and false positives. ## Running the Audits - The taskflows are available in the [`seclab-taskflows`](https://github.com/GitHubSecurityLab/seclab-taskflows) repository. - To run an audit: 1. Start a Codespace for the repository. 2. Wait for initialization. 3. Run `./scripts/audit/run_audit.sh myorg/myrepo`. - Audits may take one or two hours on a medium-sized repository. - Results are stored in SQLite and can be inspected in the `audit_results` table. - Rows marked with a check in `has_vulnerability` indicate potential findings. - A GitHub Copilot license and premium model requests are required. - The same repository should be audited multiple times because LLM results are nondeterministic; using different models may reveal different vulnerabilities. - Private repositories require changes to the Codespace configuration to grant access. ## How Taskflows Work - Taskflows are YAML files defining ordered tasks and dependencies for an LLM. - The `seclab-taskflow-agent` runs tasks sequentially and passes their results between stages. - Repository audits begin by dividing the codebase into functional components. - For each component, context is gathered, including: - Untrusted-input entry points - Intended privilege levels - Component purposes and behavior - This context is stored in a database for later auditing tasks. - Separate tasks can: - Suggest generic security issues - Carefully verify each suggested issue - Focus on specific vulnerability classes - Tasks can be reused across many components asynchronously through templated prompts and component-specific substitutions. ## Why Use Multiple Tasks - A single large prompt is less reliable because LLMs may omit steps in complex, multi-stage investigations. - Taskflows help control, debug, and structure the process even when models provide large context windows. - Breaking work into stages allows each result to be reviewed and reused as context for subsequent analysis. - Repeated task execution across components makes the approach scalable for large repositories. ## General Security Auditing - The team initially used the framework to triage CodeQL alerts, where strict instructions and predefined criteria helped limit false positives. - General auditing is more difficult because the LLM must search broadly for vulnerabilities rather than evaluate known alerts. - Greater freedom increases the risk of hallucinations and unexploitable findings. - The authors’ approach uses taskflow design and prompt engineering to preserve a high true-positive rate while allowing the model to investigate diverse security issues. ## Reported Vulnerabilities - The taskflows have found more than 80 vulnerabilities in open-source projects. - Many reported issues are high-impact, including: - Authorization bypasses - Information disclosure - Logging in as another user - Accessing private user data - Examples include exposing personally identifiable information in ecommerce shopping carts and authenticating to a chat application with arbitrary passwords. - The authors manually verify findings before reporting them and maintain an advisories page as disclosures become public. The practical recommendation is to run the open-source taskflows on your own projects, repeat audits with different models, and manually validate every result. The framework is intended to improve through shared taskflows, prompts, and findings across the security community.

llmopen-sourcegithub-copilotprompt-engineering+4