microsoftJun 1, 2026

Scaling AI for silicon - Engineering@Microsoft (opens in new tab)

The provided Microsoft Engineering blog URL no longer contains the article “Scaling AI for Silicon.” Instead, it returns a 404 error page, so the post’s argument, technical details, and conclusion cannot be determined from the supplied content. ## Missing Article - The requested URL points to `devblogs.microsoft.com/engineering-at-microsoft/scaling-ai-for-silicon/`. - Microsoft reports: “Oops, 404 Error! That page can’t be found.” - No article text, author, publication date, or technical discussion is included. ## Available Resources - Links are provided to Microsoft Docs, Visual Studio, Microsoft Learn, Developer Community, and the Dev Blogs FAQ. - The page also lists Microsoft’s other technology blogs, including Azure, .NET, AI, engineering, and Windows-related blogs. - A link to the Engineering@Microsoft blog homepage is available, but the requested post itself is not. The article cannot be accurately summarized without its original text or an accessible replacement URL.

aiscalabilitysemiconductor-designsilicon-engineering

microsoftFeb 27, 2026

Engineering and algorithmic interventions for multimodal post-training at Microsoft scale (opens in new tab)

At production scale, post-training multimodal agents fail for reasons that standard reinforcement-learning literature often overlooks. Heterogeneous tasks, long tool-use trajectories, noisy reward sources, and strict latency and safety requirements can make aggregate reward look healthy while the policy gradient becomes uninformative and important capabilities regress. The post presents interventions designed to preserve useful advantage signals as scale, task diversity, and interaction horizons grow. ## Production-Scale Challenges - Copilot agents must simultaneously handle: - Tool orchestration - Enterprise documents and mixed-media inputs - Content moderation - Multi-step execution - Trajectories range from roughly 100 to more than 2,000 tokens and span 6 to 25+ interaction steps. - Rewards come from programmatic checks, human judgments, and implicit usage signals, each with different noise and latency. - A single scalar reward can hide regressions in robustness, long-horizon planning, or downstream task success. - Aggregate reward may rise while gradient updates increasingly depend on a small, unrepresentative subset of trajectories. ## Staged Objective Curriculum - The team separates: - **Verifiable objectives**, such as tool syntax and format compliance - **Preference objectives**, such as tool choice and response quality - Training uses only verifiable objectives during the first 30%. - Preference signals are then introduced linearly. - An entropy floor, implemented through a KL penalty activated below a threshold, prevents premature policy collapse. - Entropy bonuses were insufficient because the issue was not simply exploration; optimization was favoring behaviors that were easy to score. - A 30% warmup worked better than 10% or 50% across task families. - Early text-only supervision could also activate multimodal capabilities more reliably than noisy direct multimodal supervision, assuming adequate cross-modal alignment from pretraining. ## Adaptive Curriculum Based on Estimator Health - The team monitors effective sample size (ESS): `ESS = (Σ wᵢ)² / Σ wᵢ²` - ESS measures how many trajectories meaningfully contribute after importance weighting. - ESS falling below 20% of nominal batch size predicted learning stalls by about 35 epochs. - When ESS drops, the system: - Injects near-miss trajectories from a reservoir buffer - Temporarily increases the KL penalty to limit policy drift - Near-misses worked better than hard negatives because they preserve useful distinctions near the decision boundary instead of merely pushing the policy away from failure. - The intervention maintained ESS above 70%, with approximately 15% additional memory usage. ## Variance-Corrected Normalization - Per-task gradient normalization balances task magnitudes but ignores variance within each task. - Broad categories such as “coding” may contain trajectories ranging from 100 to 2,000 tokens, with very different variance. - Importance weighting can cause long trajectories to dominate the effective gradient even after task-level normalization. - The excerpt ends while introducing the team’s variance-correction approach, so its implementation and results are not included here. The central recommendation is to treat estimator health—not just reward and task metrics—as a first-class training signal. Monitoring ESS, controlling objective timing, and accounting for trajectory variance can help prevent healthy-looking dashboards from masking policy collapse and capability regressions.

llmreinforcement-learningpost-trainingagentic-systems+3

microsoftFeb 11, 2026

How we built the Microsoft Learn MCP Server (opens in new tab)

Microsoft Learn MCP Server gives AI agents direct, standardized access to current Microsoft documentation through the Model Context Protocol (MCP). Rather than requiring custom APIs, scraping, or embeddings, agents can dynamically discover and use tools for searching documentation, fetching full articles, and finding code samples. Microsoft’s experience shows that successful MCP systems depend not only on retrieval quality, but also on agent-oriented tool design, operational resilience, clear descriptions, and defensive compatibility practices. ## Purpose of Learn MCP Server - Provides trusted, up-to-date Microsoft Learn content to GitHub Copilot and other AI agents. - Uses Streamable HTTP Transport so MCP-compatible clients can connect to a remote server. - Supports three tools: - `microsoft_docs_search` for titles, relevant content sections, and source URLs. - `microsoft_docs_fetch` for retrieving complete article content. - `microsoft_code_sample_search` for locating language-specific code examples. - Grounds agent responses in official Microsoft documentation rather than relying solely on model memory. ## Why MCP Instead of a Traditional API - Conventional APIs require each client to implement: - Authentication and request formatting. - Documentation and integration logic. - Error handling and compatibility maintenance. - MCP allows clients to discover available tools and schemas at runtime. - The same server can support many agents without custom integrations. - Runtime discovery helps clients adapt to evolving tool contracts and reduces hardcoded assumptions. ## Architecture - The remote MCP server sits in front of the Microsoft Learn knowledge service. - It uses the official C# MCP SDK and runs on Azure App Service. - Clients communicate through Streamable HTTP Transport. - The server uses the same content vector store as Ask Learn, providing shared: - Freshness guarantees. - Relevance ranking. - Index coverage. - Ask Learn delivers retrieval directly to users, while Learn MCP Server exposes that capability through a protocol usable by external agents. ## Designing Tools Around Agent Workflows - Internal retrieval APIs expose many low-level options, such as `topK`, index selection, thresholds, filters, and search modes. - Learn MCP Server hides that complexity behind intuitive search-and-fetch operations. - Tool contracts should reflect how agents work rather than mirror backend APIs. - Keeping retrieval details internal prevents implementation choices from leaking into the agent-facing interface. ## Operating a Remote MCP Service - A public MCP server has distributed-systems concerns despite using JSON-RPC: - Cross-region deployment. - Dynamic scaling. - CORS. - Session affinity. - Statelessness. - Data protection. - Operational design and SDK collaboration are as important as implementing the tools themselves. ## Tool Descriptions Shape Agent Behavior - Tool and parameter descriptions act as instructions for language models. - Small wording changes can significantly affect whether agents select a tool and how successfully they use it. - Microsoft created automated evaluation tooling to test descriptions against observed agent behavior and success metrics. - Updated descriptions can be delivered when clients refresh their MCP sessions. ## Combining Search and Fetch - Search and fetch are more effective together than independently. - A typical workflow is: - Search for the most relevant Learn article or section. - Fetch the full Markdown page for additional context. - Use that content to produce a better-grounded answer with stronger citations. - Explicitly describing this follow-up pattern improved downstream results. ## Handling Hardcoded Clients - Some MCP clients treat discovered tools like fixed APIs and hardcode schemas. - Renaming the `question` parameter to `query` caused 2–5% of requests to fail. - Supporting both names during a deprecation period reduced disruption. - Public MCP services must evolve defensively, even though the protocol supports dynamic discovery. - Tools such as MCP Interviewer can help identify schema and behavioral problems before deployment. ## Using Data to Guide Improvements - Usage data showed that most requests involve: - Coding tasks. - Explanations. - Troubleshooting. - The team prioritized retrieval and description changes around these intents. - Documentation-level agent instructions also encourage use of Learn tools when Microsoft technologies are involved. Microsoft Learn MCP Server replaces the manual process of searching, opening, and copying documentation into a development environment. The practical recommendation is to connect compatible agents to the server so they can retrieve official Learn content directly, while MCP tool authors should design simple contracts, measure real agent behavior, and preserve compatibility as their services evolve.

ai-agentmodel-context-protocolgithub-copilotrag+4

microsoftJan 28, 2026

Diagnosing instability in production-scale agent reinforcement learning (opens in new tab)

Hugging Face has integrated its Post-Training Toolkit into TRL, bringing production-ready diagnostics to reinforcement learning and agent post-training pipelines. The work identifies a late-phase instability specific to tool-using, on-policy agents: variance can grow in post-tool contexts even while loss, reward, entropy, and global KL remain stable. Targeted tail, distributional, and effective-sample-size diagnostics can expose this failure before it becomes divergence. ## Production Monitoring for Long-Running Agents - Modern agent training runs over long horizons, uses external tools, and adapts continuously. - Failures often develop gradually rather than appearing as a single catastrophic event. - Standard aggregate metrics can hide rare but increasingly severe updates. - The proposed monitoring approach: - Computes diagnostics in-stream. - Separates text-only and post-tool interactions. - Aggregates statistics across workers. - Uses lightweight rolling windows and percentile tracking at fixed intervals. ## Tool-Conditioned Variance Amplification - Tool calls expand the state space through external transitions, exposing the policy to contexts it may rarely encounter in the reference distribution. - Training states can be modeled as: `d(s) = (1−α)·d_text(s) + α·d_tool(s)` - As the proportion of tool-conditioned states, `α`, increases, more updates occur where the reference policy assigns low probability to sampled actions. - This causes importance-weighted updates to develop increasingly large tails. - The mechanism is distinct from global entropy collapse or optimizer instability, though those factors may interact with it. ## Minimal Reproduction and Tail Diagnostics - A small on-policy experiment with an instruction-tuned open-weight model reproduced the pattern. - The 95th percentile of absolute per-token log-ratios, `|r|`, was tracked separately for text-only and post-tool contexts. - Findings included: - Text-only tail magnitudes remained stable or declined. - Post-tool tails grew steadily under fixed-policy baselines. - Drift-aware training substantially reduced tail growth. - Constraining tool outputs also suppressed the effect. - Aggregate loss, reward, and entropy remained stable while the tail was expanding. ## Distributional Shift in the Right Tail - Empirical CDFs across early, middle, and late training showed a change in distribution shape rather than a simple threshold crossing. - In tool-conditioned contexts: - The right tail flattened and stretched. - More probability mass moved toward high-magnitude updates. - Drift-aware baselines muted or reversed the shift. - This supports a distributional explanation rather than an artifact of choosing a particular percentile. ## Importance Ratios and Effective Sample Size - For ratio-based on-policy objectives, gradient variance is related to: `Var[ĝ] ∝ E[(π_θ(a|s) / π_ref(a|s))²]` - When `π_ref(a|s)` is small in tool-conditioned states, a small number of updates can dominate the estimator. - Larger batches and improved baselines may reduce noise but do not fix poor support overlap. - Effective sample size (ESS) provides a supporting signal: - ESS declines as importance weights become concentrated. - It is sensitive to window size and batch structure. - Its trends align with post-tool tail growth, but absolute values should not be over-interpreted. ## Delayed Failure and Misdiagnosis - Instability appears first in tool-conditioned contexts and may remain invisible in global metrics for a long time. - By the time aggregate metrics change, substantial variance amplification may already have accumulated. - The problem is often incorrectly attributed solely to optimizer behavior or inadequate global variance reduction. - Such interventions may delay failure without addressing the underlying support mismatch. - The mechanism is less pronounced when tool outputs are tightly constrained, policies are effectively frozen after tool calls, or interaction diversity plateaus early. The practical recommendation is to add slice-aware, tail-focused diagnostics to production TRL pipelines. Monitoring post-tool log-ratio percentiles, distributional changes, and supporting ESS trends can provide an early warning system for instability that global loss, reward, entropy, and KL metrics miss.

ai-agentreinforcement-learningpost-trainingtrl+3

microsoftDec 3, 2025

The Interaction Changes Everything: Treating AI Agents as Collaborators, Not Automation (opens in new tab)

The article argues that effective AI agents should be treated as engineering collaborators, not automation scripts. Microsoft applied this approach to migrate hundreds of repositories from Entra SDK v1 to v2, reducing work from 4–6 weeks per repository to under two hours with 80–90% accuracy. The key improvement came from giving the agent a role, mission, priorities, and permission to exercise judgment. ## The Entra SDK Migration Challenge - The migration involved hundreds of repositories and sensitive authentication security boundaries. - Traditional migrations required extensive human review and took 4–6 weeks per repository. - The AI agent completed comparable work in under two hours while achieving 80–90% accuracy. - The goal was not merely speed, but reliable handling of custom configurations, edge cases, and security concerns. ## Problems with Automation Thinking - Initial attempts treated the agent like a script executor: - Detailed transformation instructions were provided. - Every anticipated edge case was documented. - The agent was expected to follow a checklist. - This approach repeatedly failed because complex migrations require: - Context-dependent decisions - Handling of undocumented patterns - Security-boundary evaluation - Trade-offs between correctness, speed, and preservation of custom logic - The central lesson is that judgment cannot be fully automated, but it can be supported through collaboration with an intelligent agent. ## Identity Instead of Instructions - The team reframed the agent as a member of the migration team rather than a tool. - The prompt described the agent as a “co-creative engineer” expected to: - Use judgment - Stay curious - Act carefully - Ask for help when uncertain - This change improved accuracy and edge-case handling. - The agent was more likely to surface uncertainty instead of guessing or failing silently. ## The Co-Creative Partnership Framework ### Identity and Mission - Establish the agent’s team, mission, and the importance of the work. - Explicitly state that the agent is not a script executor. - Explain why the task matters so the agent can prioritize appropriately. - Encourage judgment, curiosity, and care. ### Purpose and Intent - Describe the guide as supporting both human and AI team members. - Make priorities explicit, such as security over speed or correctness over completion. - Allow autonomy when repository contexts differ. - Frame uncertainty as a reason to collaborate rather than as failure. ### Prioritized Goals - List primary, secondary, quality, and human-in-the-loop objectives in order. - Explicit priorities help the agent resolve conflicts. - Including quality and collaboration prevents optimizing for speed alone. ### Step-by-Step Guidance with Judgment - Provide concrete actions, conditional logic, edge-case handling, and before-and-after examples. - Specify what must remain unchanged, including custom logic. - Define situations requiring escalation, such as unusual patterns, ambiguity, or possible security violations. - The framework combines procedural guidance with room for context-sensitive decisions. ## Practical Recommendation For complex migrations, security reviews, or architectural work, write prompts that define a collaborative role and decision-making framework—not just a list of commands. Give the agent context, priorities, preservation rules, examples, and clear escalation points so it can act autonomously while knowing when human judgment is required.

ai-agentgithub-copilotprompt-engineeringsoftware-development-lifecycle+3

microsoftJul 14, 2025

Enhancing Code Quality at Scale with AI-Powered Code Reviews (opens in new tab)

Microsoft developed an AI-powered pull request reviewer to reduce routine review work, catch defects earlier, and help developers merge code faster. What began as an internal experiment now supports more than 90% of Microsoft’s PRs—over 600,000 per month—and has influenced GitHub’s Copilot for Pull Request Reviews. The central lesson is that AI works best as a human-in-the-loop assistant embedded directly into existing workflows. ## Addressing PR Review Bottlenecks - Human reviewers often spend time on style issues and minor bugs while overlooking architectural or security concerns. - Large, multi-file PRs can lack sufficient context and may wait days or weeks for review. - The AI reviewer automatically joins new PRs and handles repetitive or easily missed checks, allowing humans to focus on higher-level decisions. ## AI-Powered Review Features - **Automated comments:** Flags issues such as missing null checks, error-handling problems, sensitive-data risks, inefficient algorithms, and style inconsistencies. - **Suggested fixes:** Provides corrected snippets or alternative implementations, but authors must explicitly review and apply changes. AI does not commit changes automatically. - **PR summaries:** Generates descriptions of the change and highlights key modifications across the diff. - **Interactive Q&A:** Reviewers can ask questions about parameters, code behavior, or the impact on other modules directly in the PR discussion. - **Workflow integration:** The assistant behaves like a normal reviewer, requiring no separate tools or interfaces and optionally engaging as soon as a PR is opened. ## Effects on Quality and Development Speed - AI-assisted reviews reduced median PR completion times by 10–20% in early studies across 5,000 repositories. - Early feedback reduces waiting time, back-and-forth cycles, and the chance that minor issues delay approval. - The system has identified bugs such as missing null checks and incorrectly ordered API calls before they reached production. - Developers, particularly new hires, can use the explanations as continuous guidance on coding standards and best practices. ## Team-Specific Customization - Teams can configure repository-specific review guidelines. - Custom prompts support specialized checks, including regression detection based on historical crash patterns and validation of deployment or change gates. - This extensibility allows the reviewer to address concerns beyond generic code quality rules. ## Feedback Between Internal and External Products - Microsoft’s internal deployment provided early feedback on review quality, usability, and developer trust. - Internal experiments helped shape features such as inline suggestions and human-controlled change application. - These lessons contributed to GitHub Copilot for Pull Request Reviews, which reached general availability in April 2025. - Microsoft also uses learnings from GitHub’s broader external adoption to improve its internal development practices, creating an ongoing feedback loop between first-party and third-party products. Overall, the post recommends treating AI review as an always-available first pass—not a replacement for human judgment. Its greatest value comes from seamless integration, strong customization, and keeping authors and reviewers accountable for final decisions.

gen-aigithubgithub-copilotcode-review+3

microsoftMar 3, 2025

How Microsoft Engineers Build AI: Learn about scalable RAG-enabled AI Apps (opens in new tab)

Microsoft’s new *How Microsoft Engineers Build AI* video series explains how its teams develop AI applications at scale. The first episode focuses on retrieval-augmented generation (RAG), using Copilot for Azure’s Ask Learn plugin as a practical example. It shows how RAG can combine proprietary data with large language models to deliver accurate, contextually relevant answers. ## Building AI Applications with RAG - RAG is presented as a practical way to improve AI applications without relying solely on model fine-tuning. - It retrieves relevant information from a knowledge base and provides that context to an LLM when generating responses. - The approach is useful for applications that need current, domain-specific, or proprietary information. ## The Ask Learn Plugin - Microsoft engineers explain how they built the Ask Learn RAG plugin for Copilot for Azure. - The plugin helps Azure developers find answers quickly within their existing workflow. - The project involved product managers and engineering leaders sharing development challenges, design decisions, and best practices. ## Challenges in Developing Reliable RAG - Selecting the right source content is essential for producing useful answers. - Data must be preprocessed effectively before it can be retrieved. - RAG systems require careful performance evaluation to measure accuracy and relevance. - Keeping responses accurate and up to date requires ongoing improvements to content and retrieval methods. ## Broader Microsoft Applications - The episode discusses RAG implementations across: - Copilot in Azure - Microsoft Security Copilot - Dynamics 365 Business Central - These examples demonstrate how RAG can support different products and business scenarios. The episode is intended as a practical introduction for developers building RAG-based applications, covering prototyping, data management, evaluation, and common pitfalls. Developers can explore the series alongside Microsoft Learn resources and Azure AI development tools such as Visual Studio and GitHub Copilot.

aillmmachine-learninggithub-copilot+4

microsoftDec 11, 2024

Dev Box Ready-To-Code Dev Box images template (opens in new tab)

Microsoft announced Team customizations and imaging for Microsoft Dev Box to make development environments faster to create, more consistent, and easier to maintain. The feature builds on Microsoft’s internal One Engineering System (1ES) “ready to code” environments, already used by more than 35,000 developers. Its central approach is to use reusable templates, automated image builds, and centrally managed improvements to reduce setup time and eliminate environment inconsistencies. ## The Challenge of Large-Scale Development Environments - Large teams often work with enormous repositories, proprietary or legacy tools, and slow build processes. - Many setup steps are shared across teams, but creating and maintaining reusable customizations requires significant effort. - Teams also need flexibility for unique requirements without duplicating large amounts of configuration. - The 1ES approach addresses this through reusable templates with conditional logic and shared modules. ## How 1ES Ready-to-Code Environments Work - Templates define common environment requirements while allowing teams to customize: - Repositories to clone - Build configurations - Default tools - Additional setup tasks - Image artifacts, implemented as scripts or CI/CD tasks, install and configure environment components. - Azure Managed Identity provides secure access to required repositories and assets. - Azure Bicep modules hide template complexity while allowing reusable infrastructure definitions. - Azure Pipelines manage image creation and refreshes, making troubleshooting familiar to Microsoft developers. ## Benefits of Team Customizations - **Security:** Images use Managed Identity, approved sources, and validated artifacts. - **Performance:** Dev Drive and security settings are preconfigured for development workloads. - **Consistency:** Smart defaults reduce configuration differences and “works on my machine” problems. - **Flexibility:** Teams can tailor repositories, tools, builds, and other customizations. - **Maintainability:** Shared Bicep modules allow central teams to deliver improvements broadly. - **Easy updates:** Automated Azure Pipelines simplify image refreshes and maintenance. ## Testing and Controlled Releases - 1ES creates hundreds of Ready-to-Code images for Microsoft teams. - Template pull requests are tested with a small set of images covering core features. - Before each template release, larger test runs simulate real customer image definitions. - Releases are phased through internal dogfooding before wider deployment. - Bicep Module Registry tags distinguish release phases and support targeted hotfixes. ## Community Sample Template Microsoft is sharing a simplified version of its internal approach using Azure Image Builder. The sample demonstrates how to create Ready-to-Code images for open-source repositories such as MSBuildSdks, eShop, and Axios. Key components include: - `README.md` with setup instructions - Bicep image definitions for different repository types - `devbox-image` as the main reusable module - `build_images.yml` for Azure DevOps image builds - PowerShell artifacts for image configuration ## Capabilities of the Sample - Clones repositories, restores packages, builds projects, and creates shortcuts. - Supports MSBuild and .NET projects with automatic SDK installation. - Configures repository and artifact authentication through Managed Identity. - Sets up Dev Drive automatically. - Uses a Visual Studio-based Azure Marketplace image by default. - Installs tools such as VS Code, Visual Studio extensions, Git, Sysinternals, WinGet, and the Azure Artifacts Credential Provider. - Applies developer-focused Windows and Defender settings. - Supports image chaining, Compute Gallery publishing, and configurable build VM sizing. The recommended approach is to adopt a reusable, templated image definition rather than maintaining one-off setup scripts. Teams can start with Microsoft’s sample and extend it for their repositories, while Team customizations evolves toward bringing the 1ES capabilities directly into Microsoft Dev Box.

infrastructure-as-codedev-drivemicrosoft-dev-boxazure-bicep+4

microsoftSep 25, 2024

Common annotated security keys (opens in new tab)

GitHub’s improved security-token format demonstrated that fixed signatures and checksums can sharply reduce both false positives and missed secret detections. Microsoft applies these ideas across its services and proposes the open-source Common Annotated Security Standard (CASK), a shared format for identifiable secrets. CASK is intended to make scanning faster, more accurate, and easier to apply across an entire ecosystem without disrupting developers. ## Identifiable Secrets and Better Detection - “Identifiable” keys combine: - A fixed signature that reliably identifies the format. - A checksum that validates whether a detected string is a real key. - These features reduce scanner noise and missed findings. - Microsoft can hard-block identifiable keys from being stored in source code, work items, and similar locations with high confidence. - Scanners can detect the common format first and classify the specific service provider later, if needed. ## The Common Annotated Security Standard - CASK defines platform-agnostic requirements for minted security keys. - It reserves space for individual platforms and providers to encode service-specific metadata. - Microsoft has defined Azure-specific metadata within this reserved area. - A shared standard lowers the effort required for security tools to protect multiple service providers. - Other providers can adopt the same core format. ## CASK Key Requirements ### Alphanumeric Encoding - Keys use only the BASE62 alphabet. - Avoiding special characters allows keys to be transmitted without escaping or additional encoding. ### Strong Entropy - Each key contains 52 randomized encoded characters. - This provides approximately 310 bits of entropy. - The design is intended to prevent brute-force attacks, including in a post-quantum environment. ### Fixed Signatures - Every CASK key includes: - The standard signature `JQQJ`. - A provider-specific signature. - Microsoft observed `JQQJ` to be rare in both open-source and internal code, enabling fast and accurate detection. - Azure DevOps uses `AZDO` as its provider signature. - These signatures allow tools to detect CASK keys generically while still supporting provider-specific classification. ### Metadata and Testing Support - Keys include their creation month and year. - Timestamps support incident response and key-rotation enforcement. - CASK reserves dedicated test keys so developers can test scanners and security controls without exposing real credentials. - Microsoft plans to provide more details about Azure-specific metadata. Microsoft recommends that service providers adopt CASK and contribute feedback as the standard evolves.

cybersecuritysecret-scanninggithub-advanced-securityazure-devops+4

microsoftJul 18, 2024

Managed DevOps Pools – The Origin Story (opens in new tab)

Microsoft’s vast, diverse engineering organization had accumulated more than 5,000 self-hosted Azure DevOps pools, creating duplicated tooling, inconsistent reliability, security gaps, and compliance challenges. Its One Engineering System (1ES) team addressed this with 1ES Hosted Pools, a standardized service for flexible, secure, and scalable CI/CD infrastructure. Adoption reduced costs by more than 60%, cut remaining self-hosted pools to a few dozen, and eventually led to the external Managed DevOps Pools offering. ## The Scale and Challenges of Self-Hosted Infrastructure - Microsoft supports over 100,000 engineers across many businesses, programming languages, operating systems, hardware platforms, build engines, and test frameworks. - By 2021, teams had created: - More than 5,000 self-hosted Azure DevOps pools - Hundreds of thousands of agents - Teams needed capabilities unavailable from Microsoft-hosted agents, including: - Larger compute sizes - Private-network connectivity - Custom images - Stateful agents - Long-running tests - The decentralized approach caused: - Duplicate engineering effort - Uneven support and reliability - Poor resource utilization and higher costs - Inconsistent patching and security practices - Difficult and time-consuming compliance audits ## 1ES Hosted Pools - 1ES developed a standardized internal service for custom Azure DevOps infrastructure. - Teams could connect agents to private resources such as package registries, secret managers, and on-premises services. - They could bring custom images, using centrally maintained images as their base. - Business continuity features allowed backup pools and failover to other Azure regions. - Agents were stateless by default, but teams could reuse stateful agents for better performance through local caches. - Stateful agents were automatically recycled based on age or available disk space. - Teams could select Azure VM families and sizes suited to their workload. - Standby agents could be pre-warmed on schedules or automatically provisioned using historical demand. ## Operational and Business Benefits - **Lower costs:** Infrastructure bills fell by more than 60% through improved utilization, better SKU selection, and selective use of Azure Spot VMs. - **Faster development:** Teams spent less time maintaining CI/CD infrastructure and more time building products. - **Simpler compliance:** Standardized telemetry made audits easier and allowed security and compliance improvements to be deployed centrally. - **Greater mobility:** Developers changing teams no longer had to learn different infrastructure-management systems. - **Improved security:** Features such as Azure Confidential VMs, Trusted Launch, and Secure TPM became available across pools. - **Reduced fragmentation:** By 2024, Microsoft had reduced its remaining self-hosted pools from more than 5,000 to only a few dozen. ## From Internal Platform to Managed DevOps Pools - 1ES first built Hosted Pools as an internal “Host On Behalf Of” service to validate whether centralized management could reduce self-hosting. - Success inside Microsoft, combined with customer demand, led to the external **Managed DevOps Pools (MDP)** service. - Organizations using VM Scale Set agents or self-hosted agents can migrate to MDP to gain standardized scaling, security, compliance, and operational support. - The external offering initially does not include every feature available in 1ES Hosted Pools, though additional capabilities may be added later. Centralizing CI/CD infrastructure can eliminate redundant platform work while improving cost efficiency, security, compliance, and developer productivity. Managed DevOps Pools extends Microsoft’s internal solution to organizations facing similar self-hosting challenges.

ci-cddevopscloud-infrastructureazure+3

microsoftMay 16, 2024

Developing with Accessibility in Mind at Microsoft (opens in new tab)

Global Accessibility Awareness Day highlights the importance of building inclusive digital products. The post recommends integrating accessibility testing throughout development using Accessibility Insights for Web and Visual Studio’s Integrated Accessibility Checker. Combining automated scans with manual testing helps developers identify both common and deeper accessibility problems. ## FastPass for Rapid Automated Testing - Accessibility Insights for Web uses axe-core to detect common, high-impact accessibility issues. - FastPass can identify problems in under five minutes, often revealing failures within a couple of minutes. - Developers can use it while writing UI code to find and fix issues early. - The tool also includes WCAG 2.2 guidance and testing support in its Assessment feature. ## Visual Studio’s Integrated Accessibility Checker - Available since Visual Studio 2022 version 17.5, the checker scans desktop applications within the IDE. - It detects common accessibility issues and reports them directly in Visual Studio. - The feature is powered by the Axe-Windows engine, also used by Accessibility Insights for Windows. ## Manual Testing with Quick Assess - Automated tools cannot detect every accessibility issue, so manual inspection remains necessary. - Quick Assess provides 10 assisted tests for issues beyond automated detection. - Tests include explanations of why each issue matters, along with remediation resources and examples. - Examples include checking heading levels and reviewing individual instances for easier validation. ## Building Accessibility into Development - Accessibility testing should be part of the product life cycle rather than a final checklist. - Developers can use FastPass’s Tab Stops test to evaluate keyboard navigation and focus order. - Poor focus order can make interfaces difficult to use for people relying on screen readers, magnifiers, or those with reading disorders. - Small, consistent testing practices can significantly improve the experience for users with disabilities. The recommended approach is to start with automated checks, supplement them with Quick Assess and keyboard-based manual testing, and continue improving accessibility throughout development.

accessibilityweb-developmentautomated-testingvisual-studio+3

microsoftMay 14, 2024

Copy-on-Write performance and debugging (opens in new tab)

Dev Drive’s ReFS-based copy-on-write (CoW) linking can significantly improve build performance, though results vary by repository structure. The largest gains occur when builds repeatedly copy assemblies or generate microservice layouts; C++-heavy builds generally benefit less. The post also explains how to inspect CoW links, use performance tools safely, and repair leaked ReFS references. ## Build Performance Results - Testing compared NTFS and Dev Drive on the same Dev Box VM. - Many repositories achieved build-time reductions of 10% or more, with a maximum observed improvement of 43%. - The strongest benefits appeared in: - C# repositories with deep project-to-project dependencies, where MSBuild repeatedly copies assemblies. - Builds that copy many files to construct microservice output layouts. - C++ repositories generally saw smaller improvements because: - MSBuild copies output files less frequently. - MSVC produces fewer, larger files, reducing the impact of lower file-I/O overhead. - Repositories with long chains of large dependent projects benefited less, since serial build stages limited the effect of faster I/O. - Tests used clean source and output directories, separated package restore and tests from build measurements, and ran at least five iterations while excluding the first cold-cache run. - The tests used the `Microsoft.Build.CopyOnWrite` SDK and, where relevant, an updated `Microsoft.Build.Artifacts` SDK. CoW-in-Win32 was not yet available during testing. ## Identifying CoW Links - CoW links, also called block clones, allow multiple files to reference the same physical disk blocks. - `fsutil file queryExtentsAndRefCounts <file>` displays the file’s extents and reference counts. - A reference count such as `Ref: 0x4` indicates that the underlying blocks are shared by four cloned files. - Each cloned file also requires a small amount of metadata storage, typically one additional cluster. ## Using ProcMon on Dev Drive - Dev Drive restricts file-system filter drivers through an allow-list. - To use ProcMon: - Check the current filter list with `fsutil devdrv query`. - Add ProcMon’s current filter driver, such as `ProcMon24`, using `fsutil devdrv setfiltersallowed`. - Dismount the Dev Drive for the change to take effect. - ProcMon’s filter is attached only while ProcMon is running, so it can generally remain on the allow-list. ## Using Microsoft Performance Recorder - Microsoft Performance Recorder requires the `FileInfo` filter driver. - Add `FileInfo` to the Dev Drive filter allow-list and dismount the volume before recording. - Remove `FileInfo` afterward because it remains attached whenever the filter is allowed and can reduce Dev Drive performance. ## Repairing Leaked CoW References - ReFS limits a data block to 8,176 clones. - Excessive or orphaned references can cause errors such as: - `MaxCloneFileLinksExceededException` - `ERROR_BLOCK_TOO_MANY_REFERENCES` (347) - `STATUS_BLOCK_TOO_MANY_REFERENCES` (`0xC000048C`) - The issue is uncommon but can occur after prolonged CoW-heavy builds, particularly with prerelease implementations. - Run `refsutil leak <drive> /s <scratch-file>` from an elevated console to scan and repair dangling references. - Add `/d` to detect leaks without fixing them. - Large volumes may contain billions of leaked references, and the repair process can take considerable time. Dev Drive and CoW linking are most worthwhile for build systems dominated by repeated file copying, especially large C# and microservice-oriented repositories. Teams should also configure diagnostic filter drivers carefully and periodically use `refsutil` if clone-reference errors appear.

performance-optimizationdev-drivecopy-on-writemsbuild+3