microsoftDec 11, 2024

Dev Box Ready-To-Code Dev Box images template (opens in new tab)

Microsoft announced Team customizations and imaging for Microsoft Dev Box to make development environments faster to create, more consistent, and easier to maintain. The feature builds on Microsoft’s internal One Engineering System (1ES) “ready to code” environments, already used by more than 35,000 developers. Its central approach is to use reusable templates, automated image builds, and centrally managed improvements to reduce setup time and eliminate environment inconsistencies. ## The Challenge of Large-Scale Development Environments - Large teams often work with enormous repositories, proprietary or legacy tools, and slow build processes. - Many setup steps are shared across teams, but creating and maintaining reusable customizations requires significant effort. - Teams also need flexibility for unique requirements without duplicating large amounts of configuration. - The 1ES approach addresses this through reusable templates with conditional logic and shared modules. ## How 1ES Ready-to-Code Environments Work - Templates define common environment requirements while allowing teams to customize: - Repositories to clone - Build configurations - Default tools - Additional setup tasks - Image artifacts, implemented as scripts or CI/CD tasks, install and configure environment components. - Azure Managed Identity provides secure access to required repositories and assets. - Azure Bicep modules hide template complexity while allowing reusable infrastructure definitions. - Azure Pipelines manage image creation and refreshes, making troubleshooting familiar to Microsoft developers. ## Benefits of Team Customizations - **Security:** Images use Managed Identity, approved sources, and validated artifacts. - **Performance:** Dev Drive and security settings are preconfigured for development workloads. - **Consistency:** Smart defaults reduce configuration differences and “works on my machine” problems. - **Flexibility:** Teams can tailor repositories, tools, builds, and other customizations. - **Maintainability:** Shared Bicep modules allow central teams to deliver improvements broadly. - **Easy updates:** Automated Azure Pipelines simplify image refreshes and maintenance. ## Testing and Controlled Releases - 1ES creates hundreds of Ready-to-Code images for Microsoft teams. - Template pull requests are tested with a small set of images covering core features. - Before each template release, larger test runs simulate real customer image definitions. - Releases are phased through internal dogfooding before wider deployment. - Bicep Module Registry tags distinguish release phases and support targeted hotfixes. ## Community Sample Template Microsoft is sharing a simplified version of its internal approach using Azure Image Builder. The sample demonstrates how to create Ready-to-Code images for open-source repositories such as MSBuildSdks, eShop, and Axios. Key components include: - `README.md` with setup instructions - Bicep image definitions for different repository types - `devbox-image` as the main reusable module - `build_images.yml` for Azure DevOps image builds - PowerShell artifacts for image configuration ## Capabilities of the Sample - Clones repositories, restores packages, builds projects, and creates shortcuts. - Supports MSBuild and .NET projects with automatic SDK installation. - Configures repository and artifact authentication through Managed Identity. - Sets up Dev Drive automatically. - Uses a Visual Studio-based Azure Marketplace image by default. - Installs tools such as VS Code, Visual Studio extensions, Git, Sysinternals, WinGet, and the Azure Artifacts Credential Provider. - Applies developer-focused Windows and Defender settings. - Supports image chaining, Compute Gallery publishing, and configurable build VM sizing. The recommended approach is to adopt a reusable, templated image definition rather than maintaining one-off setup scripts. Teams can start with Microsoft’s sample and extend it for their repositories, while Team customizations evolves toward bringing the 1ES capabilities directly into Microsoft Dev Box.

dev-driveinfrastructure-as-codemicrosoft-dev-boxazure-bicep+4

microsoftMay 14, 2024

Copy-on-Write performance and debugging (opens in new tab)

Dev Drive’s ReFS-based copy-on-write (CoW) linking can significantly improve build performance, though results vary by repository structure. The largest gains occur when builds repeatedly copy assemblies or generate microservice layouts; C++-heavy builds generally benefit less. The post also explains how to inspect CoW links, use performance tools safely, and repair leaked ReFS references. ## Build Performance Results - Testing compared NTFS and Dev Drive on the same Dev Box VM. - Many repositories achieved build-time reductions of 10% or more, with a maximum observed improvement of 43%. - The strongest benefits appeared in: - C# repositories with deep project-to-project dependencies, where MSBuild repeatedly copies assemblies. - Builds that copy many files to construct microservice output layouts. - C++ repositories generally saw smaller improvements because: - MSBuild copies output files less frequently. - MSVC produces fewer, larger files, reducing the impact of lower file-I/O overhead. - Repositories with long chains of large dependent projects benefited less, since serial build stages limited the effect of faster I/O. - Tests used clean source and output directories, separated package restore and tests from build measurements, and ran at least five iterations while excluding the first cold-cache run. - The tests used the `Microsoft.Build.CopyOnWrite` SDK and, where relevant, an updated `Microsoft.Build.Artifacts` SDK. CoW-in-Win32 was not yet available during testing. ## Identifying CoW Links - CoW links, also called block clones, allow multiple files to reference the same physical disk blocks. - `fsutil file queryExtentsAndRefCounts <file>` displays the file’s extents and reference counts. - A reference count such as `Ref: 0x4` indicates that the underlying blocks are shared by four cloned files. - Each cloned file also requires a small amount of metadata storage, typically one additional cluster. ## Using ProcMon on Dev Drive - Dev Drive restricts file-system filter drivers through an allow-list. - To use ProcMon: - Check the current filter list with `fsutil devdrv query`. - Add ProcMon’s current filter driver, such as `ProcMon24`, using `fsutil devdrv setfiltersallowed`. - Dismount the Dev Drive for the change to take effect. - ProcMon’s filter is attached only while ProcMon is running, so it can generally remain on the allow-list. ## Using Microsoft Performance Recorder - Microsoft Performance Recorder requires the `FileInfo` filter driver. - Add `FileInfo` to the Dev Drive filter allow-list and dismount the volume before recording. - Remove `FileInfo` afterward because it remains attached whenever the filter is allowed and can reduce Dev Drive performance. ## Repairing Leaked CoW References - ReFS limits a data block to 8,176 clones. - Excessive or orphaned references can cause errors such as: - `MaxCloneFileLinksExceededException` - `ERROR_BLOCK_TOO_MANY_REFERENCES` (347) - `STATUS_BLOCK_TOO_MANY_REFERENCES` (`0xC000048C`) - The issue is uncommon but can occur after prolonged CoW-heavy builds, particularly with prerelease implementations. - Run `refsutil leak <drive> /s <scratch-file>` from an elevated console to scan and repair dangling references. - Add `/d` to detect leaks without fixing them. - Large volumes may contain billions of leaked references, and the repair process can take considerable time. Dev Drive and CoW linking are most worthwhile for build systems dominated by repeated file copying, especially large C# and microservice-oriented repositories. Teams should also configure diagnostic filter drivers carefully and periodically use `refsutil` if clone-reference errors appear.

dev-driveperformance-optimizationcopy-on-writemsbuild+3