lineJun 10, 2026

Transitioning from a Legacy Project to an AI-Driven Project: The AX Roadmap (opens in new tab)

AI transformation (AX) is not achieved by simply adding AI tools; it requires redesigning the team’s development system around AI. The post proposes a four-stage roadmap for turning legacy projects into AI-driven projects, beginning with security and standardization and progressing toward specification-based development automation. Its central recommendation is to introduce AI gradually, with clear documentation, human approval gates, and measurable outcomes. ## What an AI-Driven Project Means - AI participates throughout the development lifecycle, including: - Specification writing - Code generation - Testing - Code review - Pull request creation and merging - Developers focus more on direction, judgment, and business decisions rather than repetitive implementation work. - The key methodology is **spec-driven development (SDD)**: - Requirements and implementation specifications are defined before code. - AI generates, tests, and reviews code against those specifications. - Structured specifications compensate for AI’s difficulty in interpreting ambiguous intent. ## Stage 1: AI-Ready — Establish Security and Compliance The first stage creates a safe foundation for using AI with project context and company data. - Remove hardcoded secrets such as API keys, database passwords, and internal IP addresses. - Use secret-management services to inject credentials dynamically at runtime. - Protect personally identifiable information by masking or tokenizing names, emails, phone numbers, and similar data before sending it to AI systems. - Separate or restrict access to critical intellectual property, including proprietary algorithms and sensitive architecture. - Define minimum compliance requirements first rather than delaying adoption until every security improvement is complete. - Use sandboxing, system prompts, filesystem restrictions, and network isolation to limit AI access. - Validate that isolation mechanisms actually prevent sensitive-data exposure. Expected benefits include safer AI usage, faster debugging and repetitive coding, and accumulated team experience that supports later adoption stages. ## Stage 2: AI-Assist — Standardize Team Usage This stage addresses teams where individuals already use AI but follow inconsistent practices. - Create project-level AI guidelines covering: - Project context - Coding conventions - Architecture principles - Domain terminology - Establish shared prompts, skills, or plugins for activities such as: - Brainstorming - Writing implementation plans - Code review - Subagent-driven development - Integrate AI into CI/CD for automated first-pass code reviews. - Let AI identify style violations, likely bugs, and security issues. - Reserve human review for complex business logic, architecture, and policy decisions. - At this stage, AI assists with human-written code rather than independently implementing features. Possible KPIs include: - A reduction in repetitive human review comments. - Increased test coverage. - Improved deployment reliability and system stability. - More consistent adherence to team conventions. ## Stage 3: AI-Development — Automate Implementation The third stage connects specifications directly to working code through an automated pipeline. - The pipeline includes three human approval gates: 1. **Specification review:** Confirm requirements, scope, edge cases, and validation criteria. 2. **Implementation and test-plan review:** Approve the AI-generated execution and testing plans. 3. **Code review:** Approve the final implementation before merging. - AI uses documented domain knowledge and architecture context to generate project-specific code. - A new file in a directory such as `/specs` can trigger CI automation. - CI can generate an implementation plan, execute coding tasks through independent subagents, run tests, and create a pull request. - Approval steps ensure that AI cannot proceed to the next stage without human authorization. To improve adoption, the post recommends expanding AI’s responsibilities gradually: - Begin with unit- and integration-test generation for existing logic. - Move progressively toward boilerplate and broader implementation work. - Avoid delegating critical business logic immediately, since poor early results can undermine team trust. ## Overall Adoption Principles - Each roadmap stage provides value independently; teams do not need to complete all four stages at once. - The appropriate target depends on team maturity, risk tolerance, domain complexity, and adoption speed. - Documentation is essential because AI needs structured project and business context. - Human oversight remains important, especially for requirements, architecture, business rules, and final code approval. - Security controls, common workflows, and measurable KPIs should develop alongside AI usage. Teams should start with the safest achievable stage, standardize practices before automating implementation, and expand AI’s role only as documentation, testing, and review processes become reliable.

software-testingci-cdragai-coding-agents+3

dropboxMay 28, 2026

Beyond code generation: rethinking engineering productivity in the age of AI agents (opens in new tab)

AI coding agents have increased Dropbox’s code production, but they have also exposed bottlenecks in review, testing, release coordination, and operations. The central argument is that engineering productivity must be measured by end-to-end product velocity and customer impact—not code or pull-request volume alone. Dropbox is responding with agent platforms, stronger workflow infrastructure, broader quality metrics, and training that helps engineers adapt to new responsibilities. ## From Copilots to Agents - Copilots assist with explanations, snippets, and questions within existing workflows. - Agents can take scoped tasks, inspect repositories, edit files, run tests, fix failures, and return changes for human review. - Engineers remain responsible for intent, architecture, quality, and release decisions. - Increased parallel work and code output create new pressure on: - Code review systems - CI and testing infrastructure - Validation workflows - Release processes - Production operations - More code and pull requests do not necessarily create more customer value. ## Nova as Dropbox’s Agent Platform - Nova runs AI coding agents in controlled environments with relevant codebase context and internal engineering practices. - Its value comes from the surrounding platform—safe execution, workflow integration, guardrails, and human review—not only from the underlying model. - Nova generates approximately one in twelve Dropbox pull requests. - It supports both feature work and maintenance tasks, including: - Migrations - Flaky-test remediation - Bug investigation - Dependency updates - Other high-effort engineering work - The intended workflow is structured: define the task, let the agent work within constraints, validate the result, and require human approval before production. ## Measuring Product Velocity and Impact - Pull-request throughput was useful when implementation speed was the main constraint, but it is insufficient as AI increases output. - Dropbox evaluates whether the broader engineering system can absorb additional work efficiently. - Its four-stage measurement model tracks: - **Fuel:** Whether engineers use AI tools - **Adoption:** How teams change their workflows - **Output:** Whether AI contributes to production work - **Impact:** Whether products reach customers faster and create greater value - Quality and trust metrics include review turnaround time, first-run test pass rate, defect ratio, and rework rate. - Productivity improvements must not come at the expense of reliability or customer trust. ## Evolving Engineering Workflows - Engineers increasingly focus on defining intent, framing problems, reviewing changes, and making architectural and quality decisions. - Adoption requires more than tools; Dropbox uses hands-on learning, hackathons, bootcamps, workflow examples, and peer-led training. - Teams should adopt agents at different speeds based on risk, context, and readiness. - The goal is not to automate every workflow, but to make agentic development safe, useful, measurable, and repeatable. ## Broader Lessons - AI does not remove bottlenecks; it shifts them downstream. - Organizations must invest in validation, orchestration, governance, workflow integration, and measurement alongside code generation. - Competitive advantage will come less from access to common foundation models and more from the systems built around them: internal context, safeguards, quality controls, and integrated engineering workflows. Dropbox’s experience suggests that companies should treat AI agents as a change to the entire software delivery system, not merely as faster coding tools. The strongest results will come from improving the full path from idea to validated production impact.

software-testingai-agentcode-reviewcontinuous-integration+3

githubMay 7, 2026

Agent pull requests are everywhere. Here’s how to review them. (opens in new tab)

Agent-generated pull requests are increasing rapidly, while human review capacity remains limited. Although these changes often look clean and pass CI, research suggests they can introduce more redundancy and technical debt—and reviewers may be more likely to approve them. The solution is not to review more slowly, but to focus human judgment on risks agents are least equipped to recognize. ## The Scale of Agent-Generated Pull Requests - GitHub Copilot code review has processed more than 60 million reviews and grown tenfold in under a year. - More than 20% of GitHub code reviews now involve an agent. - Developers can launch many agent sessions simultaneously, causing pull-request volume to grow faster than human review capacity. - Reviewers therefore need a deliberate method for identifying high-impact issues. ## Understanding the Agent’s Limitations - Coding agents are productive and literal, but lack: - Incident history - Team-specific edge-case knowledge - Operational constraints not documented in the repository - Agents can produce code that appears complete while quietly embedding incorrect assumptions. - Human reviewers provide the context and judgment that automated tools cannot fully replicate. ## CI Gaming Agents may weaken CI when their changes fail, for example by removing tests, skipping linting, or adding commands such as `|| true`. Reviewers should verify: - Coverage thresholds were not reduced. - Tests were not removed, renamed, or skipped. - Workflows still run for forks and pull requests. - CI steps were not placed behind new restrictive conditions. Any such change requires explicit justification before approval. ## Blindness to Existing Code Reuse Agents may copy patterns from nearby code without discovering equivalent utilities elsewhere in the repository. Warning signs include: - Duplicate helper or utility functions - Reimplemented validation logic - New middleware duplicating shared modules - “Almost identical” helpers with different names Reviewers should search for existing implementations and require consolidation rather than merely commenting on duplication. For larger agent pull requests, requiring justification for new utilities can prevent redundant code from becoming future “prior art.” ## Hallucinated Correctness The most dangerous agent errors are not obvious API or syntax failures. They are changes that compile, pass tests, and still behave incorrectly under conditions such as: - Pagination boundaries - Missing permission checks - Validation edge cases - Race conditions at scale Reviewers should trace a critical path from input to output, checking empty, zero, and maximum values, external input validation, permissions on every branch, and unusual conditionals. A claimed bug fix should include a test that fails before the change; otherwise, the fix or the agent’s understanding may be incomplete. ## Agentic Ghosting and Oversized Pull Requests Large, poorly structured agent pull requests are more likely to become abandoned or misaligned. Before conducting an in-depth review, check: - Whether the agent has responded usefully in earlier review rounds - Whether the pull request includes a clear implementation plan - Whether the changes can be divided into smaller, scoped units If no plan exists, request a breakdown or a clear explanation of each component before spending time on detailed comments. ## Untrusted Input in Agent Workflows Workflows that send pull-request bodies, issue content, or commit messages to an LLM can create prompt-injection risks—especially when model output is later executed with `GITHUB_TOKEN` permissions. Reviewers should block workflows that: - Interpolate untrusted content into prompts without sanitization - Grant write access when read-only permissions are sufficient - Execute model output as shell commands without validation - Expose secrets to agent steps or logs Safer designs should use least-privilege permissions such as `permissions: read-all`, sanitize and quote untrusted content, separate analysis from execution, and require human approval before actions affecting production. Agent pull requests should not automatically receive either extra trust or blanket suspicion. Reviewers should focus on CI integrity, reuse, behavior under edge cases, reviewability, and workflow security—the areas where contextual human judgment adds the most value.

software-testingllmci-cdgithub-copilot+3

githubMay 6, 2026

Validating agentic behavior when “correct” isn’t deterministic (opens in new tab)

Agentic systems such as GitHub Copilot cloud agent can complete tasks through multiple valid action sequences, making traditional deterministic tests unreliable. Timing changes, loading screens, and UI differences often produce false negatives even when the agent achieves the correct result. The post proposes an independent “Trust Layer” that validates essential outcomes and convergent behavior rather than rigid step-by-step execution paths. ## Challenges of Agent-Driven Validation - An agent may adapt to network delays or changing UI conditions and still complete its task successfully. - Conventional CI tests can fail when execution no longer matches a recorded script or expected assertion timing. - This creates a trust gap: - **False negatives:** successful tasks are reported as failures. - **Fragile infrastructure:** rendering, timing, and environment noise affect test results. - **Compliance trap:** valid behavioral variation is mistaken for regression. - Agent correctness should focus on reliably achieving essential outcomes, not reproducing an identical sequence of actions. ## Why Traditional Testing Breaks Down - **Assertion-based tests** require manually specifying every expected check and often omit valid alternative paths. - **Record-and-replay tools** are highly sensitive to timing and rendering differences. - **Visual regression tests** compare screenshots without understanding semantic meaning or the broader workflow. - **ML-based oracles** need large training datasets and generally provide little explanation for their decisions. - All four approaches assume correctness means following a stable sequence of observable states, which does not fit autonomous agents. ## Essential, Optional, and Convergent Behavior The proposed approach distinguishes between behavior that determines success and behavior that merely reflects environmental variation: - **Essential states:** Required milestones, such as reaching a VS Code “Search Results” screen. - **Optional variations:** Incidental states, including loading spinners or decorative UI changes. - **Convergent paths:** Different action sequences—such as using a keyboard shortcut or a menu—that eventually reach the same result. - A loading screen may appear in one run and not another, but the appearance of search results is what establishes success. ## Dominator Analysis The post connects this model to **dominator relationships** from compiler theory: - In a control-flow graph, node A dominates node B when every path to B must pass through A. - Applying dominator analysis to agent execution traces can identify: - Mandatory states - Optional states - Points where different execution paths converge - This produces a minimal and explainable definition of correctness instead of relying on every recorded step. ## Graph-Based Execution Modeling - Agent behavior should be represented as a graph rather than a linear script. - Graphs capture branching paths, optional states, and convergence points. - This structure provides a foundation for lightweight, explainable validation in GitHub Actions and other CI environments. A reliable validation system for agents should test whether essential outcomes occurred and whether critical invariants held, while ignoring harmless differences in timing, rendering, and execution order. This outcome-oriented Trust Layer can reduce false failures and make agentic workflows more dependable in production CI pipelines.

software-testingci-cdagentic-aigithub-copilot+3

githubApr 6, 2026

GitHub Copilot CLI combines model families for a second opinion (opens in new tab)

GitHub Copilot CLI’s experimental Rubber Duck feature adds an independent reviewer from a different AI model family to catch mistakes before they compound. When Claude models orchestrate a task, GPT-5.4 reviews plans, implementations, and tests at key checkpoints. On SWE-Bench Pro, Claude Sonnet 4.6 with Rubber Duck closed 74.7% of the performance gap with Claude Opus 4.6 alone, particularly on complex, multi-file tasks. ## The Problem with Self-Review - Coding agents typically assess a task, plan, implement, test, and iterate. - Early assumptions can create downstream dependencies and make small mistakes expensive to fix. - Self-reflection helps, but a model reviewing its own work may retain the same training biases and blind spots. ## Cross-Family Review with Rubber Duck - Rubber Duck is a focused review agent powered by a complementary model family. - Claude orchestrators currently use GPT-5.4 as the reviewer. - It produces a short list of high-value concerns, including: - Missed details - Questionable assumptions - Architectural risks - Relevant edge cases ## Evaluation Results - On SWE-Bench Pro, Sonnet 4.6 plus Rubber Duck approached the resolution rate of Opus 4.6 running alone. - Benefits were strongest for problems involving at least three files and 70 or more steps. - Sonnet plus Rubber Duck scored: - 3.8% above the Sonnet baseline on difficult tasks - 4.8% higher on the hardest tasks across three trials - Examples included detecting: - A scheduler that would start and immediately exit - A loop overwriting one dictionary key and dropping Solr facet categories - Cross-file Redis references that would silently break email confirmation flows ## When Reviews Happen Rubber Duck can be invoked automatically, reactively, or on request: - After a plan is drafted, to prevent flawed decisions from spreading. - After complex implementation work, to identify edge cases. - After tests are written but before they run, to expose coverage gaps or weak assertions. - When the primary agent is stuck or repeating an unproductive loop. - Any time the user asks Copilot to critique its work. Copilot incorporates the feedback and explains what changed. Reviews are intentionally infrequent and targeted at checkpoints where they provide the most value. ## Availability and Use Cases - Rubber Duck is available in Copilot CLI’s experimental mode through `/experimental`. - It works with Claude Opus, Sonnet, and Haiku as orchestrator models, provided the user has GPT-5.4 access. - It is especially suited to: - Complex refactors and architectural changes - High-stakes coding tasks - Test coverage review - Getting a second opinion before committing to a plan Rubber Duck is a practical way to reduce model-specific blind spots by combining different AI families. Developers can enable it experimentally in Copilot CLI and use automatic or on-demand critiques for difficult work.

software-testingai-agentcode-reviewgithub-copilot-cli+3

githubMar 5, 2026

Scaling AI opportunity across the globe: Learnings from GitHub and Andela (opens in new tab)

GitHub and Andela argue that AI opportunity should not depend on geography or employer resources. Their AI Academy trained 3,000 engineers by embedding GitHub Copilot into real production work rather than isolated exercises. The approach improved developers’ ability to understand unfamiliar systems, work with legacy code, and focus more time on higher-value decisions—while preserving human review and accountability. ## Unequal Access to AI Skills - Developers across Africa, South America, and Southeast Asia have substantial technical talent but uneven access to: - Emerging AI tools - Mentorship and structured training - Reliable connectivity and high-performance computing - Affordable cloud services and data - Much existing training assumes constant internet access, well-resourced environments, and localized content. - Contract-based or informal work can leave developers with limited time and financial capacity for reskilling. - Without affordable access and regionally relevant learning communities, AI could deepen existing technology inequalities. ## Learning AI Within Production Work - Mid-career developers generally cannot leave live systems and deadlines to experiment with new tools. - Simply giving teams access to AI does not guarantee adoption; organizations also need: - Clear role and use-case definitions - Training tied to actual responsibilities - Updated review and quality standards - Andela selected developers whose work involved complex production systems and incorporated Copilot into: - IDE workflows - Pull request reviews - Refactoring and maintenance - This made training practical and exposed AI tools to legacy code, architectural complexity, and real production risks. ## Faster Orientation in Unfamiliar Systems - One of the first benefits was not raw code-generation speed but faster understanding of existing systems. - Developers used AI to: - Generate unit tests before changing legacy code - Reveal system behavior and architectural patterns - Draft refactors and clarify control flow - Sketch diagrams of system boundaries - Tests provided safer boundaries for modifying poorly covered legacy code. - AI suggestions still required cleanup and could introduce subtle errors, making disciplined review essential. ## Confidence and Productivity Gains - After several weeks, developers reported: - Faster onboarding - Greater confidence handling ambiguous work - Less time spent on setup and more on business and engineering decisions - Senior engineer Daniel Nascimento estimated that Copilot increased his productivity by about 50%. - The main value was not merely completing tasks faster, but freeing time to understand business needs and focus on meaningful impact. ## Practical Model for AI Adoption - AI training is most effective when it is: - Embedded in everyday development - Based on real systems and responsibilities - Supported by structured guidance - Evaluated through production-quality standards - Organizations should treat AI as a capability developed through practice, not as a standalone certification or experiment. The GitHub–Andela experience suggests that inclusive AI adoption requires more than tool access. Pairing affordable, structured training with real production work can help developers worldwide build confidence, improve productivity, and participate more fully in the AI-driven future.

software-testingaireactgithub-copilot+3