A Human-Augmenting Agentic Workflow for Causal Inference
The post presents a human-augmenting agentic workflow for observational causal inference (OCI), designed to automate repetitive analysis while preserving expert oversight. It combines an actor agent that executes analyses with a critic agent that evaluates assumptions, diagnostics, and credibility. The authors argue that transparent artifacts and process audits are essential because observational analyses rarely have definitive ground truth. ## Why Causal Inference Requires Oversight - Data agents can quickly query data and run regressions, but may overlook confounding, selection bias, or differences between average users and specific subgroups. - OCI requires substantial domain judgment, particularly when estimating effects from observational data under an unconfoundedness assumption. - Automation is best used to reduce repetitive work—such as repeated balance checks, sensitivity analyses, and tracking iterations—so practitioners can focus on framing questions and scrutinizing assumptions. ## Target Trial Emulation and Design Diagnostics Netflix’s OCI toolkit frames each analysis around the ideal randomized controlled trial that would answer the question. - This “target trial” clarifies the treatment, outcome, population, timing, and assumptions required for a credible estimate. - The workflow evaluates: - **Covariate balance:** weighted standardized mean differences should generally be below 0.2. - **Overlap:** propensity scores should remain between 0.1 and 0.9. - **Placebo outcomes:** treatment should not appear to affect variables measured before treatment. - **Sensitivity to hidden confounding:** estimated effects should be assessed against hypothetical omitted variables. - These diagnostics help identify whether treated and untreated groups are sufficiently comparable. ## Human-Augmenting Agent Design The workflow uses three personas: - **Principal:** The human data scientist who defines the research question, context, threats to validity, tools, and data. - **Actor:** The software agent that turns the plan into an analysis specification, executes the analysis, runs diagnostics, and produces reproducible artifacts. - **Critic:** The software agent that reviews the plan and results, identifies omissions, assesses credibility, and recommends improvements. The actor and critic operate in an iterative loop. Actors must use only approved tools, create human- and machine-checkable outputs, and report how they address failed diagnostics. Critics check for missing confounders, inconsistencies between the plan and execution, differences between the estimated estimand and the ATE, and gaps relative to the ideal randomized trial. ## Transparent Evaluation Through Artifacts Because observational data generally lacks ground truth, evaluating an agent solely by comparing its answer to a known result is insufficient. - Agents produce plans, specifications, plots, reports, and executed notebooks. - Reports are version-controlled and notebooks are stored so principals can download and re-run them. - Human reviewers can inspect every analytical step rather than trusting only the final estimate. - The workflow also supports conventional evaluations using simulated datasets, while emphasizing process audits for real-world analyses. ## Empowering Practitioners The system provides a templated notebook built on Netflix’s vetted, non-agentic OCI toolkit. - The toolkit uses doubly robust learning for causal effect estimation. - Humans remain responsible for writing the initial analysis plan and reviewing the executed notebook and critic’s report. - The design is intended to extend beyond unconfoundedness-based OCI to methods with different assumptions, such as panel methods requiring parallel trends. ## Case Study: New Entertainment Types Netflix applies the workflow to questions about whether newer entertainment offerings affect member satisfaction and subscription retention. - The case study focuses on an entertainment category referred to as **Type X**. - The broader goal is to estimate effects that could inform business strategy and understanding of member behavior. - The workflow is positioned as a way to combine automated analysis with human judgment in this setting. The recommended approach is not to let an agent make an unaudited causal claim. Instead, practitioners should use agents to execute standardized analyses and diagnostics, then inspect reproducible artifacts and critically assess the assumptions behind the result.
Read original(opens in new tab)