Designing synthetic datasets for the real world: Mechanism design and reasoning from first principles

Synthetic data generation should be treated as dataset-level mechanism design rather than one-sample-at-a-time prompting. Google’s Simula framework uses reasoning models to control coverage, diversity, complexity, and quality independently, enabling seedless, reproducible datasets for scarce or privacy-sensitive domains. Experiments across five domains show that thoughtfully designed data can outperform larger datasets, but the best configuration depends on the target task and model.

Why Real-World Data Is Insufficient

• Specialized AI applications often lack accessible data because domains are uncommon, expensive to label, or privacy-sensitive.
• Manually creating datasets is costly, slow, and error-prone.
• Real-world datasets are static, limiting rapid iteration and making it difficult to proactively generate safety edge cases.
• Synthetic-first workflows can make data programmable, versioned, reproducible, and inspectable.

Limitations of Existing Synthetic Data Methods

• Many approaches depend on manual prompts, evolutionary algorithms, or large amounts of seed data.
• These dependencies reduce scalability and explainability.
• Generation parameters are often entangled, making it difficult to independently control diversity, difficulty, and correctness.
• Most methods optimize individual samples instead of designing the dataset’s overall distribution.

Simula’s Reasoning-First Design

• Simula constructs datasets from first principles using reasoning models rather than opaque generation processes.
• The framework is seedless and agentic, allowing improvements as the underlying models become better at reasoning.
• It separates generation into independently controllable axes.

Global Diversification

• Reasoning models map a domain into deep, hierarchical taxonomies that serve as sampling scaffolds.
• A recursive propose-and-refine process generates candidate categories, then evaluates, merges, and filters them with a critic model.
• These taxonomies help datasets cover long-tail concepts instead of concentrating on common examples.
• The approach was demonstrated with structures such as a Cyber Threat Intelligence taxonomy.

Local Diversification

• Taxonomy nodes are converted into “meta-prompts” representing scenarios.
• Multiple distinct instantiations are generated for each scenario.
• This reduces mode collapse—for example, representing SQL injection through varied contexts rather than repeating nearly identical questions.

Complexification

• A configurable portion of scenarios is made more elaborate or difficult.
• Complexity can therefore be adjusted without changing the dataset’s semantic coverage.
• The appropriate difficulty level depends on the capabilities of the model that will consume the data.

Quality Checks

• A dual-critic loop independently assesses whether outputs and answers are correct.
• Independent verification helps reduce sycophancy and improves label reliability.
• This enables quality control without requiring human review of every example.

Reasoning-Based Evaluation

• Conventional metrics such as embedding cosine distance offer only broad signals and limited practical guidance.
• Simula introduces:
  • Taxonomic Coverage, which measures how thoroughly the conceptual space is represented.
  • Calibrated Complexity Scoring, which uses LLM-based batch comparisons and chess-style Elo ratings to estimate the difficulty of individual examples.
• These metrics aim to evaluate diversity and difficulty in ways that better reflect downstream usefulness.

Results Across Domains

• Simula was evaluated using Gemini 2.5 Flash as a teacher and Gemma 3 4B as a student.
• The experiments covered cybersecurity, legal reasoning, grade-school mathematics, and multilingual academic knowledge, with datasets reaching 512,000 examples per domain.
• The full combination of global coverage, local diversity, and quality critique consistently outperformed simpler baselines.
• High complexity improved math reasoning accuracy by 10% in GSM8k but harmed legal reasoning performance, where the teacher model was weaker.
• Simula often achieved stronger downstream results with fewer examples, suggesting that data properties matter as much as volume.
• The findings show that there is no universally optimal synthetic-data recipe; effective dataset design must be tailored to the domain and target model.

Synthetic data is most effective when treated as an engineered system rather than a source of additional volume. Practitioners should separately tune coverage, variation, difficulty, and verification, then validate those choices against the downstream task.