How we built an internal data analytics agent (opens in new tab)
Qubot is GitHub’s internal, Copilot-powered analytics agent, designed to make warehouse data accessible without requiring an analyst. Employees ask natural-language questions through Slack, VS Code, or the Copilot CLI, while Qubot uses curated context and selects between Kusto and Trino to produce answers. GitHub’s experience shows that structured context is central to improving both accuracy and response speed. ## The Problem Qubot Addresses - Product teams often struggle to identify the right data model, grain, filters, and query. - Limited analyst availability leaves many teams to investigate telemetry independently. - Qubot targets exploratory questions rather than replacing dashboards or formal reporting. - It enables employees to investigate unfamiliar datasets with minimal setup and maintenance. ## Multiple Ways to Ask Questions - **Slack:** Users ask questions in a shared channel, receive answers in threads, and collaborate on follow-up questions. - Each result is saved as a Markdown report in a pull request, allowing users to refine queries or reuse them in dashboards. - **VS Code and Copilot CLI:** Qubot is installed as a plugin and operates alongside other agents, skills, and tools. - Offering both zero-configuration and developer-focused interfaces increased adoption among different user groups. ## A Federated Context Layer Qubot’s context is tailored to GitHub’s warehouse layers: - **Bronze:** Product teams provide telemetry schemas and metadata for raw events. - **Silver:** Data teams maintain query examples, usage guidance, and required filters for conformed data. - **Gold:** Dataset owners contribute business rules and metric definitions. - ETL pipelines add derived metadata and other signals automatically. - Context is fetched at runtime through the GitHub MCP Server. A dedicated context agent standardizes contributions from different teams. It ingests Markdown documentation and repository references, then organizes and normalizes them into a structure that Qubot can use effectively. ## Evaluation Before Deployment Every change to Qubot’s instructions or context layer is tested through an offline evaluation framework. - Test cases contain prompts, expected answers, ground-truth SQL, domains, and difficulty levels. - Automated orchestration launches multiple parallel agent trials using `gh agent-task create`. - Results are saved as JSON and aggregated by completion rate, accuracy, and duration. - Teams compare configurations and detect regressions before releasing changes. ## Choosing Between Kusto and Trino Qubot connects to both query engines through MCP servers: - **Kusto** is used for fast exploration of recent event data. - **Trino** handles complex joins and deeper historical analysis. - Qubot defaults to Kusto and switches to Trino when the question requires more advanced or historical querying. - This hides query-engine selection from users. ## Adoption and Lessons Learned - Hundreds of GitHub employees have run thousands of queries through Qubot. - Routine questions in analytics support channels declined as teams gained more autonomy. - The agent helped employees who previously avoided the warehouse access data for decision-making. - GitHub found that well-structured, carefully maintained context improved accuracy and made Qubot approximately three times faster at finding the right answer. The main recommendation is to treat analytics context as a maintained engineering asset. A capable agent depends not only on model intelligence, but also on accurate metadata, documented business rules, evaluation-driven iteration, and interfaces that fit users’ existing workflows.