How Meta Used AI to Map Tribal Knowledge in Large-Scale Data Pipelines
AI coding assistants struggle when they lack a map of a large, proprietary codebase. To address this, the team built a pre-compute system using 50+ specialized agents that analyzed over 4,100 files across four repositories and three languages, producing 59 concise context files. The approach gave agents complete module coverage, captured previously undocumented tribal knowledge, reduced tool calls by about 40%, and made complex development tasks much faster. ## The Problem: Powerful Tools Without Codebase Context - The pipeline combines Python configuration, C++ services, and Hack automation across multiple repositories. - A seemingly simple change, such as adding a data field, can affect: - Configuration registries - Routing logic - DAG composition - Validation rules - C++ code generation - Automation scripts - AI agents often explored repeatedly, guessed at conventions, and produced code that compiled but was subtly incorrect. - Important examples of missing context included: - Different field names for the same operation in separate configuration modes - “Deprecated” enum values that must remain for serialization compatibility - Hidden intermediate field names used between pipeline stages ## The Pre-Compute Approach The team used a large-context model and orchestrated specialized agents in several phases: - Two agents explored and mapped the codebase. - Eleven analysts read every file and answered five questions: - What does the module configure? - How is it commonly modified? - What non-obvious patterns can cause failures? - What are its cross-module dependencies? - What tribal knowledge is hidden in comments? - Writers generated context files. - More than ten critic passes reviewed quality across three rounds. - Fixers, upgraders, gap-fillers, prompt testers, and final critics corrected and validated the results. - In total, more than 50 specialized tasks were coordinated in one session. This process uncovered over 50 non-obvious design patterns, including naming conventions and append-only identifier rules that were not documented elsewhere. ## Context Files: “A Compass, Not an Encyclopedia” Each of the 59 context files is intentionally short—about 25–35 lines or roughly 1,000 tokens—and contains: - Quick Commands for common operations - Key Files limited to the most relevant three to five files - Non-Obvious Patterns - See Also references to related modules Together, the files use less than 0.1% of a modern model’s context window. They are designed for targeted, opt-in use rather than being loaded into every task. ## Routing and Dependency Navigation - An orchestration layer routes natural-language requests to the appropriate tool. - Operational questions can trigger dashboard scans and matching against more than 85 historical incident patterns. - Development requests can launch configuration generation and multi-phase validation. - A cross-repository dependency index and data-flow maps show how changes propagate. - Dependency questions that previously required about 6,000 tokens of exploration can be answered through a graph lookup using roughly 200 tokens. ## Results and Quality Controls - Preliminary tests across six tasks showed approximately 40% fewer tool calls and tokens. - Work that previously required around two days of research and engineer consultation took about 30 minutes. - Critic reviews raised quality scores from 3.65 to 4.20 out of 5. - Every referenced file path was verified, with no hallucinated paths. - Coverage expanded from navigation guidance for roughly 5% of modules to all 4,100+ files across three repositories. ## Why This Differs from Generic Context Files Research has found that AI-generated context files can reduce agent performance on familiar open-source projects. The team argues that this result does not directly apply to proprietary systems whose conventions and tribal knowledge are absent from model training data. Their approach addresses common problems by making context: - Concise rather than encyclopedic - Opt-in rather than always loaded - Quality-gated through independent critics - Continuously refreshed to prevent stale information Without this context, agents typically spend 15–25 tool calls exploring and remain vulnerable to subtle domain-specific errors. ## Keeping the Knowledge Fresh Automated jobs refresh the system every few weeks by: - Validating file paths - Detecting coverage gaps - Re-running critic reviews - Finding and repairing stale references - Updating routing and dependency information The system treats AI not merely as a consumer of documentation, but as the engine that creates and maintains it. ## Applying the Method Elsewhere Teams can adapt the approach by: - Identifying where agents most often fail due to undocumented conventions or dependencies - Applying the five-question analysis framework to each module - Keeping context files short and action-oriented - Using independent quality critics before publishing generated guidance - Automating freshness checks and self-repair The practical recommendation is to build a small, targeted, continuously maintained knowledge layer for proprietary codebases. Concise navigation and dependency context can reduce exploration costs while preventing the subtle errors that arise when agents lack domain-specific understanding.
Read original(opens in new tab)
