Autonomous Agents | Techlist.io

googleJul 30, 20263 min readCurated summary

Science One Framework: A verifiable autonomous research framework via Chain-of-Evidence

The Science One Framework addresses a central weakness in autonomous AI research: polished papers can contain fabricated citations, unreproducible results, or methods that do not match the underlying code. Its Chain-of-Evidence (CoE) framework requires every claim to be linked to concrete evidence and introduces CoE Audit to test those links automatically. In evaluations, Science One produced fully verifiable papers while matching or exceeding baseline research agents and human performance on several benchmarks. ## Chain-of-Evidence for Verifiable Research - CoE defines trustworthy research artifacts through two requirements: - **Completeness:** Every claim has a recorded evidence chain. - **Correctness:** The evidence genuinely supports the claim. - Claims may include: - Bibliographic references - Reported scores - Method descriptions - Conclusions - Evidence can include peer-reviewed papers, experiment logs, executed code, or result tables. - Hallucinated citations, unreproducible scores, and discrepancies between described and implemented methods are treated as broken evidence chains. ## The Science One Framework The framework builds evidence into the research process instead of attempting to verify a paper after it has been written. - **Problem Investigator** - Uses the Semantic Scholar API to construct a citation graph. - Reads up to 100 full-text PDFs per topic. - Produces a structured research brief. - Restricts final-paper references to sources retrieved through the grounded API, avoiding citations generated from model memory. - **Discovery Engine** - Explores ideas through parallel explore-exploit branches. - Each isolated cycle includes a Solver agent and a task-specific evaluator. - High-performing solutions are iteratively refined. - Raw evaluator outputs are preserved in strict, read-only records. - **Paper Writer and Claim Verifier** - Creates a structured inventory of factual claims. - Attaches inline evidence tags linking claims to workspace artifacts. - Checks each claim against its declared source. - Rewrites unsupported claims conservatively rather than allowing them to exceed the evidence. ## CoE Audit Integrity Checks CoE Audit is an automated, post-hoc forensic review of a paper, solution, code, and references. - **Score verification:** Re-runs the submitted code independently and compares the result with the paper’s reported score. - **Specification violation:** Checks whether the code solves the intended task without exploiting the evaluator or accessing ground-truth answers. - **Reference verification:** Validates every bibliography entry against academic APIs. - **Method-code alignment:** Compares the paper’s method description with the actual implementation using LLM-based judges. ## Evaluation Results - The audit evaluated 75 papers across five systems-optimization tasks: Prism, Cloudcast, EPLB, LLM-SQL, and transaction scheduling. - Science One led the evaluated systems on all four integrity checks. - It had: - Zero phantom references - Perfect score verification - The strongest method-code alignment - Baseline systems hallucinated up to 21% of references and sometimes described advanced algorithms that were implemented as simple deterministic heuristics. - Strong verification did not reduce performance: - Science One matched or exceeded human experts on all five ADRS tasks. - It achieved the best overall result on Cloudcast and EPLB. - On additional MLE-Bench and Parameter Golf evaluations, the framework also demonstrated competitive performance, including two Gold Medals across five difficult Kaggle competitions. The main recommendation is to design autonomous research systems around evidence generation from the beginning. Grounded retrieval, immutable experiment records, claim-level verification, and independent auditing can substantially improve reliability without necessarily sacrificing research performance.

Read original

Autonomous Agents Large Language Models Machine Learning Semantic Scholar Api+3

metaMar 17, 20264 min readCurated summary

Ranking Engineer Agent (REA): The Autonomous AI Agent Accelerating Meta’s Ads Ranking Innovation

Meta’s Ranking Engineer Agent (REA) autonomously manages much of the ads-ranking ML experimentation lifecycle, from generating hypotheses and launching training jobs to debugging failures and analyzing results. Unlike session-based AI assistants, REA maintains context across workflows lasting days or weeks, while engineers retain oversight at strategic checkpoints. In its first production rollout, REA doubled average model accuracy across six models and helped three engineers produce launch proposals for eight models—about five times the historical engineering output. ## The Bottleneck in Traditional ML Experimentation - Meta’s advertising systems rely on large, complex ML models serving billions of users across Facebook, Instagram, Messenger, and WhatsApp. - Improving these models traditionally requires engineers to: - Form hypotheses - Design experiments - Launch training jobs - Debug failures - Analyze results - Iterate on promising approaches - Each cycle can take days or weeks, and mature models make meaningful improvements increasingly difficult to find. - The sequential, hands-on process became a bottleneck to experimentation and innovation. ## REA as an Autonomous ML Agent - Existing ML AI tools generally assist with isolated tasks such as drafting hypotheses, writing configurations, or interpreting logs. - REA instead coordinates the full experimentation process and advances it without continuous prompting. - Its design addresses three central challenges: - **Long-running workflows:** Persistent state and memory allow REA to manage multiday or multiweek experiments. - **Hypothesis quality:** It combines historical experiment data with current ML research. - **Operational resilience:** It handles failures and compute limits within engineer-approved safeguards. ## Hibernate-and-Wake Workflow Management - Training jobs may run for hours or days, so REA delegates waiting to a background system. - It hibernates to conserve resources and automatically wakes when jobs finish. - This lets it preserve context and continue experiments without constant human supervision. - REA is built on Meta’s Confucius agent framework, which provides: - Code-generation capabilities - Integration with job schedulers - Experiment tracking - Codebase navigation tools ## Dual-Source Hypothesis Generation - REA draws ideas from two systems: - **Historical Insights Database:** A repository of previous experiments, successes, and failures used for pattern recognition and in-context learning. - **ML Research Agent:** A research component that examines baseline configurations and proposes new optimization strategies. - Combining these sources produces configurations that may not emerge from either source alone. - Some of REA’s strongest improvements resulted from combining model architecture changes with training-efficiency techniques. ## Three-Phase Experiment Planning - Before running experiments, REA proposes an exploration plan, estimates GPU costs, and obtains engineer confirmation. - Its typical strategy includes: - **Validation:** Test individual hypotheses in parallel to establish baselines. - **Combination:** Combine promising ideas to identify synergistic effects. - **Exploitation:** Intensively optimize the strongest candidates within the approved compute budget. ## Autonomous Failure Handling and Safeguards - REA adapts to infrastructure problems, unexpected errors, poor results, and compute constraints without escalating every issue to an engineer. - It uses runbooks and diagnostic reasoning to: - Exclude jobs with clear out-of-memory failures - Detect training instability, such as exploding losses - Debug preliminary infrastructure failures - Reprioritize experiments when results are weak - Its autonomy is constrained by: - Access limited to Meta’s ads-ranking codebase - Explicit engineer approval through preflight reviews - Confirmed GPU budgets - Automatic pausing or stopping when thresholds are reached ## Production Results - Across six models, REA-driven iterations achieved approximately **2× the average model accuracy** compared with baseline. - Three engineers produced proposals to launch improvements for eight models. - Historically, that volume of work would have required roughly two engineers per model, resulting in about **5× greater engineering output** with REA. REA demonstrates that autonomous agents can improve ML experimentation by handling long-running execution, generating broader hypotheses, and recovering from routine failures. The most effective deployment model combines substantial agent autonomy with explicit compute limits, codebase restrictions, and human review at major strategic decisions.

Read original

Autonomous Agents Machine Learning Deep Learning Model Optimization+4

googleNov 5, 2025Original article

DS-STAR: A state-of-the-art versatile data science agent (opens in new tab)

DS-STAR is an advanced autonomous data science agent developed to handle the complexity and heterogeneity of real-world data tasks, ranging from statistical analysis to visualization. By integrating a specialized file analysis module with an iterative planning and verification loop, the system can interpret unstructured data and refine its reasoning steps dynamically based on execution feedback. This architecture allows DS-STAR to achieve state-of-the-art performance on major industry benchmarks, effectively bridging the gap between natural language queries and executable, verified code. ## Comprehensive Data File Analysis The framework addresses a major limitation of current agents—the over-reliance on structured CSV files—by implementing a dedicated analysis stage for diverse data formats. * The system automatically scans a directory to extract context from heterogeneous formats, including JSON, unstructured text, and markdown files. * A Python-based analysis script generates a textual summary of the data structure and content, which serves as the foundational context for the planning phase. * This module ensures the agent can navigate complex, multi-file environments where critical information is often spread across non-relational sources. ## Iterative Planning and Verification Architecture DS-STAR utilizes a sophisticated loop involving four specialized roles to mimic the workflow of a human expert conducting sequential analysis. * **Planner and Coder:** A Planner agent establishes high-level objectives, which a Coder agent سپس translates into executable Python scripts. * **LLM-based Verification:** A Verifier agent acts as a judge, assessing whether the generated code and its output are sufficient to solve the problem or if the reasoning is flawed. * **Dynamic Routing:** If the Verifier identifies gaps, a Router agent guides the refinement process by adding new steps or correcting errors, allowing the cycle to repeat for up to 10 rounds. * **Intermediate Review:** The agent reviews intermediate results before proceeding to the next step, similar to how data scientists use interactive environments like Google Colab. ## Benchmarking and State-of-the-Art Performance The effectiveness of the DS-STAR framework was validated through rigorous testing against existing agents like AutoGen and DA-Agent. * The agent secured the top rank on the public DABStep leaderboard, raising accuracy from 41.0% to 45.2% compared to previous best-performing models. * Performance gains were consistent across other benchmarks, including KramaBench (39.8% to 44.7%) and DA-Code (37.0% to 38.5%). * DS-STAR showed a significant advantage in "hard" tasks—those requiring the synthesis of information from multiple, varied data sources—demonstrating its superior versatility in complex environments. By automating the time-intensive tasks of data wrangling and verification, DS-STAR provides a robust template for the next generation of AI assistants. Organizations looking to scale their data science capabilities should consider adopting iterative agentic workflows that prioritize multi-format data understanding and self-correcting execution loops.

Autonomous Agents AI Large Language Models Machine Learning+4