The Death of Traditional Testing: Agentic Development Broke a 50-Year-Old Field, JiTTesting Can Revive It (opens in new tab)
Just-in-Time Tests (JiTTests) are an LLM-driven testing approach designed for fast, agentic software development. Instead of maintaining static test suites, the system generates tests for each code change, simulates likely faults, and reports only meaningful regressions. The goal is to reduce test maintenance and false positives while catching serious bugs before production. ## Limitations of Traditional Testing - Tests are manually written as code changes enter the system. - They must account for both current behavior and unknown future changes. - This often leads to: - Tests that fail to detect relevant bugs. - False positives when intended changes break outdated assumptions. - Ongoing maintenance and review costs. - Agentic development increases the volume and speed of changes, making these problems harder and more expensive to manage. ## How Catching JiTTests Work - A new code change or pull request is submitted. - An LLM infers the likely intent of the change. - The system creates mutants—versions of the code containing deliberately introduced faults. - It generates and runs tests designed to expose those faults. - Rule-based and LLM-based assessors evaluate failures and filter out likely false positives. - Engineers receive a focused report when the system identifies an unexpected behavior change. Because the tests are tailored to a specific change, they can reason about intended behavior and distinguish legitimate updates from regressions. ## Benefits for Agentic Development - Tests are generated on demand and do not remain in the codebase. - There is no ongoing test maintenance or test-code review. - Each test is specific to the change being evaluated. - Tests automatically adapt as the code evolves. - Human attention is required mainly when an actual bug is detected. - Testing shifts from measuring generic code quality to determining whether a specific change introduces a real fault. Catching JiTTests are presented as a way to make testing scale with AI-assisted development by moving routine test creation and maintenance from engineers to automated systems.