Things I learned using 2

Karrot’s Taxonomy team built an LLM-powered system to classify marketplace posts, group activities, and local businesses into a shared category and attribute structure. After finding that manually managed taxonomies and event-only pipelines were difficult to scale, they created a configurable Taxonomy Management System using Dataflow/Beam, BigQuery, Kafka, and multiple LLM strategies. The system emphasizes scalable inference, rapid evaluation, multilingual support, and continuous taxonomy expansion.

What a Taxonomy Is and Why It Matters

• A taxonomy is a hierarchical category system, such as Outerwear > Padding/Down > Long Padding.
• It can also include attributes that describe an item’s characteristics:
  • Category: long padding
  • Attributes: brand=Nike, color=black, material=polyester
• A consistent taxonomy acts as a shared language across:
  • Search, including parent and child-category expansion
  • Recommendations and diversity controls
  • Advertising and targeting segments
  • Analytics and machine-learning features

Karrot’s Taxonomy Challenges

• Karrot manages roughly 1,400 marketplace categories across up to three levels.
• Users are not required to manually select highly detailed categories because that would increase posting friction and produce unreliable labels.
• Earlier systems used a Golang Kafka consumer to receive posting events and extract categories with an LLM.
• This approach had several limitations:
  • Taxonomy definitions were managed separately by different teams.
  • Categories alone could not express useful properties such as season or material.
  • Batch processing and backfilling were difficult.
  • Expanding to data sources outside Kafka was inconvenient.
  • Quality monitoring and failure handling were insufficient.
  • Changes to prompts or models required slow offline and online experiments.

The Taxonomy Management System

• The new system centrally manages taxonomies, performs LLM-based classification, delivers category and attribute results, and monitors quality.
• Dataflow with Apache Beam was selected because it supports:
  • Parallel, high-throughput LLM inference
  • Both streaming and large-scale batch processing
  • Existing team expertise compared with alternatives such as Spark or Flink
• BigQuery serves as the source of truth for inference results.
  • Analysts and data scientists can query results directly.
  • Online consumers can receive results through Kafka sinks into the internal feature platform.

Configuration-Driven and Extensible Design

• Taxonomy definitions are stored in YAML, allowing different services and category trees to use the same framework.
• Pipeline settings, worker sizing, Kafka topics, and BigQuery destinations are also configured through YAML.
• LLM models and inference strategies can be selected through configuration, including:
  • Primary and evaluation models
  • Single-shot or two-stage categorization
  • Attribute extraction modes
  • Evaluation sampling ratios
• The system is designed for multilingual taxonomies.
  • Large translation jobs are divided into chunks.
  • One LLM generates translations and another validates consistency and naturalness.
  • A depth-first traversal carries parent-category translations into child-category prompts to maintain terminology consistency.

Creating and Expanding Taxonomies with LLMs

• New taxonomies are developed by researching established taxonomies and generating candidate trees from real data.
• Existing taxonomies are expanded by:
  • Classifying sampled data against the current taxonomy
  • Asking the LLM to suggest categories for unsuitable examples
  • Merging similar suggestions using LLM similarity judgments
  • Promoting sufficiently strong candidates for review
• Candidates undergo two evaluations:
  • Whether the originating examples are correctly assigned to the new category
  • Regression testing comparing classifications under the old and new taxonomies
• This process enabled the team to move beyond the existing 1,400 three-level categories and create taxonomies with more than 10,000 categories and six or more levels.

LLM Categorization Strategies

The team supports multiple strategies because the best approach depends on the model and taxonomy size:

• Single shot: Provide all categories and ask the model to choose one.
• Hierarchical classification: Select the best category at each depth, then continue through the chosen branch.
• Two-stage tournament: Split categories into chunks, select candidates from each chunk, and run a second selection among those candidates.
• Categorization and attribute assignment are separate modular Beam DoFn stages:
  • Article → Category inference → Attribute inference
• New approaches can be added as interchangeable strategies without redesigning the whole pipeline.

Evaluation with LLM-as-a-Judge

• A sample of production data is processed by multiple different models.
• Their labels are combined into a ground-truth label, generally through majority voting.
• Each model’s output is compared against that ground truth.
• Accuracy changes are tracked whenever the team modifies:
  • The LLM model
  • Prompts
  • Pipeline structure
  • Categorization or attribute strategies
• The ground-truth method varies depending on whether the task involves:
  • A single category
  • Multiple categories
  • Multi-label attributes
• Category quality is measured as a precision-at-one-style accuracy: the primary model’s category must match the ground-truth category.
• Attributes are evaluated with precision and recall because a post can legitimately contain multiple attribute-value pairs.

The main recommendation is to treat LLM classification as a production data pipeline rather than a one-off prompt: centralize taxonomy management, support both batch and streaming execution, make inference strategies configurable, and build automated evaluation and monitoring into the system from the beginning.