Things I learned using 2
Karrot’s Taxonomy team built an LLM-powered system to classify marketplace posts, group activities, and local businesses into a shared category and attribute structure. After finding that manually managed taxonomies and event-only pipelines were difficult to scale, they created a configurable Taxonomy Management System using Dataflow/Beam, BigQuery, Kafka, and multiple LLM strategies. The system emphasizes scalable inference, rapid evaluation, multilingual support, and continuous taxonomy expansion. ## What a Taxonomy Is and Why It Matters - A taxonomy is a hierarchical category system, such as `Outerwear > Padding/Down > Long Padding`. - It can also include attributes that describe an item’s characteristics: - Category: long padding - Attributes: brand=Nike, color=black, material=polyester - A consistent taxonomy acts as a shared language across: - Search, including parent and child-category expansion - Recommendations and diversity controls - Advertising and targeting segments - Analytics and machine-learning features ## Karrot’s Taxonomy Challenges - Karrot manages roughly 1,400 marketplace categories across up to three levels. - Users are not required to manually select highly detailed categories because that would increase posting friction and produce unreliable labels. - Earlier systems used a Golang Kafka consumer to receive posting events and extract categories with an LLM. - This approach had several limitations: - Taxonomy definitions were managed separately by different teams. - Categories alone could not express useful properties such as season or material. - Batch processing and backfilling were difficult. - Expanding to data sources outside Kafka was inconvenient. - Quality monitoring and failure handling were insufficient. - Changes to prompts or models required slow offline and online experiments. ## The Taxonomy Management System - The new system centrally manages taxonomies, performs LLM-based classification, delivers category and attribute results, and monitors quality. - Dataflow with Apache Beam was selected because it supports: - Parallel, high-throughput LLM inference - Both streaming and large-scale batch processing - Existing team expertise compared with alternatives such as Spark or Flink - BigQuery serves as the source of truth for inference results. - Analysts and data scientists can query results directly. - Online consumers can receive results through Kafka sinks into the internal feature platform. ## Configuration-Driven and Extensible Design - Taxonomy definitions are stored in YAML, allowing different services and category trees to use the same framework. - Pipeline settings, worker sizing, Kafka topics, and BigQuery destinations are also configured through YAML. - LLM models and inference strategies can be selected through configuration, including: - Primary and evaluation models - Single-shot or two-stage categorization - Attribute extraction modes - Evaluation sampling ratios - The system is designed for multilingual taxonomies. - Large translation jobs are divided into chunks. - One LLM generates translations and another validates consistency and naturalness. - A depth-first traversal carries parent-category translations into child-category prompts to maintain terminology consistency. ## Creating and Expanding Taxonomies with LLMs - New taxonomies are developed by researching established taxonomies and generating candidate trees from real data. - Existing taxonomies are expanded by: - Classifying sampled data against the current taxonomy - Asking the LLM to suggest categories for unsuitable examples - Merging similar suggestions using LLM similarity judgments - Promoting sufficiently strong candidates for review - Candidates undergo two evaluations: - Whether the originating examples are correctly assigned to the new category - Regression testing comparing classifications under the old and new taxonomies - This process enabled the team to move beyond the existing 1,400 three-level categories and create taxonomies with more than 10,000 categories and six or more levels. ## LLM Categorization Strategies The team supports multiple strategies because the best approach depends on the model and taxonomy size: - **Single shot:** Provide all categories and ask the model to choose one. - **Hierarchical classification:** Select the best category at each depth, then continue through the chosen branch. - **Two-stage tournament:** Split categories into chunks, select candidates from each chunk, and run a second selection among those candidates. - Categorization and attribute assignment are separate modular Beam `DoFn` stages: - `Article → Category inference → Attribute inference` - New approaches can be added as interchangeable strategies without redesigning the whole pipeline. ## Evaluation with LLM-as-a-Judge - A sample of production data is processed by multiple different models. - Their labels are combined into a ground-truth label, generally through majority voting. - Each model’s output is compared against that ground truth. - Accuracy changes are tracked whenever the team modifies: - The LLM model - Prompts - Pipeline structure - Categorization or attribute strategies - The ground-truth method varies depending on whether the task involves: - A single category - Multiple categories - Multi-label attributes - Category quality is measured as a precision-at-one-style accuracy: the primary model’s category must match the ground-truth category. - Attributes are evaluated with precision and recall because a post can legitimately contain multiple attribute-value pairs. The main recommendation is to treat LLM classification as a production data pipeline rather than a one-off prompt: centralize taxonomy management, support both batch and streaming execution, make inference strategies configurable, and build automated evaluation and monitoring into the system from the beginning.
Read original(opens in new tab)