Xgboost | Techlist.io

googleJun 30, 20263 min readCurated summary

Introducing TabFM: A zero-shot foundation model for tabular data

TabFM is a foundation model for tabular classification and regression that uses in-context learning instead of dataset-specific training. It processes an entire table in one forward pass, eliminating much of the usual hyperparameter tuning and feature engineering. The authors report that it performs competitively with or better than tuned tree-based models, with planned integration into BigQuery. ## The Challenge of Traditional Tabular ML - Tabular prediction supports applications such as churn prediction and fraud detection. - Algorithms including XGBoost, AdaBoost, and random forests remain widely used. - Building reliable models typically requires: - Hyperparameter optimization - Cross-validation - Domain-specific feature engineering - Repeated model training for each dataset ## Tabular Prediction Through In-Context Learning - TabFM treats tabular prediction as an in-context learning problem. - Users provide historical training rows and target rows together as a single input. - The model learns relationships between rows and columns during inference without updating model weights. - This enables zero-shot predictions on previously unseen datasets. ## Hybrid Architecture TabFM combines ideas from TabPFN and TabICL to address the two-dimensional, order-independent nature of tables. - **Alternating row and column attention** - Attends across both features and examples. - Learns feature interactions and dependencies directly from the table. - Reduces the need for manually engineered features. - **Row compression** - Converts the contextualized information for each row into a dense vector. - **In-context Transformer** - Operates on compressed row representations rather than the full table grid. - Reduces computation and supports larger datasets efficiently. ## Training with Synthetic Data - Large, diverse real-world tables are difficult to obtain because they often contain proprietary or sensitive information. - TabFM is trained entirely on hundreds of millions of synthetic datasets. - Structural causal models generate tables using varied random functions and relationships. - The goal is to reproduce the broad range of distributions and feature interactions found in real-world data. ## Benchmark Results and Model Variants - Evaluation used TabArena, an Elo-based benchmark covering: - 38 classification datasets - 13 regression datasets - Dataset sizes from 700 to 150,000 samples - **TabFM** - Works out of the box. - Produces predictions in one forward pass without tuning or cross-validation. - **TabFM-Ensemble** - Adds cross features and SVD-derived features. - Uses a 32-model ensemble with non-negative least squares to optimize weights. - Applies Platt scaling for classification calibration. - The authors state that TabFM consistently outperforms heavily tuned conventional supervised algorithms. ## Availability and BigQuery Integration - TabFM is available through Google’s Hugging Face and GitHub repositories. - Google plans to integrate it into BigQuery. - Users will be able to run regression and classification through an `AI.PREDICT` SQL command without specialized machine-learning expertise. TabFM’s main promise is to make high-quality tabular prediction as simple as supplying a dataset and requesting predictions. Its zero-shot approach could reduce the engineering burden of traditional workflows, though practical adoption will depend on benchmark reproducibility, inference costs, and performance across specific production datasets.

Read original

Xgboost Transformer Synthetic Data Hugging Face+4

dropboxFeb 26, 20263 min readCurated summary

Using LLMs to amplify human labeling and improve Dash search relevance

Dropbox Dash improves AI answers through retrieval-augmented generation (RAG): enterprise search retrieves relevant company documents, and an LLM uses a small subset of them to generate grounded responses. Because ranking determines which documents reach the LLM, search relevance depends heavily on high-quality query–document labels. Dash combines a small set of human judgments with large-scale LLM-generated labels to produce training data efficiently while retaining human oversight. ## How Dash search ranking works - Dash uses a trained ranking model, such as XGBoost, rather than manually configured rules. - The model learns from query–document pairs labeled on a 1–5 relevance scale: - **5:** Closely matches the user’s intent. - **1:** Not useful enough to display. - Relevance depends on the query, user context, and timing; it is not an intrinsic property of a document. - Ranking quality is especially important because enterprises may have millions or billions of indexed documents, while only a small selection can be sent to the answer-generating LLM. ## Sources of relevance labels - Labels can come from: - User behavior, such as clicks or skipped results. - Human evaluators assigning relevance scores. - LLMs directly judging query–document relevance. - Behavioral signals are useful but often sparse, biased by existing rankings, and unevenly distributed, so they work best as a supplement. - Human evaluators can provide comprehensive judgments across result sets, but labeling is expensive, difficult to scale, and vulnerable to inconsistency. - Humans also cannot directly review sensitive or proprietary customer data in this process, and different content types—such as Slack messages, Jira tickets, and Salesforce records—require different contextual expertise. ## LLM-assisted relevance evaluation - LLMs can evaluate far larger candidate sets at lower cost and with greater consistency than human annotators. - They can operate across languages and analyze customer content within established compliance boundaries. - Their judgments still depend on the model’s quality and the clarity of the evaluation prompt. - LLM-generated labels therefore require calibration and validation before being used for model training. ## Combining human review with LLM scale - Dropbox first creates a relatively small, high-quality dataset using human evaluators and limited, non-sensitive internal data. - These human labels are used to tune LLM prompts and model parameters. - Once the LLM meets quality thresholds, it generates hundreds of thousands or millions of relevance labels. - This approach multiplies human labeling effort by roughly 100 times, enabling broader and more representative training data. - LLMs are used offline rather than directly at query time because production-time use would introduce excessive latency and context-window limitations. - The LLM acts as a teacher for smaller, faster ranking models that can serve searches at scale. ## Evaluation as the foundation - Dash follows an iterative process: measure performance, change the model or instructions, and measure again. - The article compares this to chess engines, where the quality of the evaluation function determines which possible moves are preserved or discarded. - The same principle applies to ranking: poor relevance judgments can cause useful search-result patterns to be eliminated, while accurate judgments guide the model toward better rankings. Dash’s approach uses humans for quality control and contextual grounding, then uses LLMs to expand that expertise into large-scale training data. This hybrid strategy offers a practical way to improve enterprise search relevance without exposing customer data to human reviewers or imposing LLM latency on every search.

Read original

Xgboost Large Language Models Machine Learning RAG+3

tossDec 1, 2025Original article

Toss Next ML Challenge (opens in new tab)

Toss recently hosted the "Toss Next ML Challenge," a large-scale competition focused on predicting advertisement Click-Through Rates (CTR) using real-world, anonymized data from the Toss app. By tasking over 2,600 participants with developing high-performance models under real-time serving constraints, the event successfully identified innovative technical approaches to feature engineering and model ensembling. ### Designing a Real-World CTR Prediction Task * The competition required participants to predict the probability of a user clicking a display ad based on a dataset of 10.7 million training samples. * Data included anonymized features such as age, gender, ad inventory IDs, and historical user behavior. * A primary technical requirement was "real-time navigability," meaning models had to be optimized for fast inference to function within a live service environment. ### Overcoming Anonymization with Sequence Engineering * To maintain data privacy while allowing external access, Toss provided anonymized features in a single flattened table, which limited the ability of participants to perform traditional data joins. * A complex, raw "Sequence" feature was intentionally left unprocessed to serve as a differentiator for high-performing teams. * Top-tier participants demonstrated extreme persistence by deriving up to 37 unique variables from this single sequence, including transition probabilities, unique token counts, and sequence lengths. ### Winning Strategies and Technical Trends * All of the top 30 teams utilized Boosting Tree-based models (such as XGBoost or LightGBM), while Deep Learning was used only by a subset of participants. * One standout solution utilized a massive ensemble of 260 different models, providing a fresh perspective on the limits of ensemble learning for predictive accuracy. * Performance was largely driven by the ability to extract meaningful signals from anonymized data through rigorous cross-validation and creative feature interactions. The results of the Toss Next ML Challenge suggest that even in the absence of domain-specific context due to anonymization, meticulous feature engineering and robust tree-based architectures remains the gold standard for tabular data. For ML engineers, the competition underscores that the key to production-ready models lies in balancing complex ensembling with the strict latency requirements of real-time serving.

Xgboost AI Machine Learning Feature Engineering+4

discordOct 8, 20253 min readCurated summary

From Single-Node to Multi-GPU Clusters: How Discord Made Distributed Compute Easy for ML Engineers

Discord argues that distributed machine learning becomes practical when developer experience is treated as a first-class engineering problem. Ray provided the distributed-computing foundation, while Discord built a platform around it with a CLI, Dagster and KubeRay orchestration, and the X-Ray observability interface. This transformed GPU-intensive ML from manual experimentation into reproducible production pipelines, enabling Ads Ranking to move to multi-GPU neural networks and produce major business gains. ## Scaling Beyond Single-Node ML - Discord’s ML systems grew from simple classifiers to complex models serving hundreds of millions of users. - Teams needed: - Multiple GPUs for training - Datasets larger than a single machine - More compute than existing infrastructure could provide - Ray addressed the distributed-computing challenge, but Discord still needed a standardized internal platform to make it easy to use. ## Problems with Ad-Hoc Ray Clusters - Early ML engineers manually created Ray clusters using open-source documentation. - This led to: - Inconsistent cluster configurations - Uneven resource management - No centralized scheduling - Limited monitoring - Multiple teams independently rebuilding infrastructure solutions - Discord concluded that Ray needed an internal platform layer rather than direct, manual use. ## A Parameterized CLI for Cluster Creation - Discord replaced numerous GPU-specific YAML templates with one parameterized template. - Engineers specify requirements such as: - GPU type - Worker count - Memory - The CLI generates Kubernetes configuration, security settings, and hardware-specific resource requests. - It manages the full cluster lifecycle, including creation and deletion. - This made multi-GPU environments available through a single command and standardized deployments across teams. ## Automated Orchestration with Dagster, KubeRay, and Ray - Discord combined three systems: - **Dagster** defines workflows, dependencies, schedules, and validated configuration. - **KubeRay** dynamically provisions Ray clusters on Kubernetes with the appropriate namespace, service account, and GPU node pool. - **Ray** executes distributed training, evaluation, and batch inference. - The workflow is: 1. An engineer launches or schedules a Dagster pipeline. 2. Dagster submits the job specification. 3. KubeRay creates the required Ray cluster. 4. Ray distributes the workload across GPUs. 5. Logs and metrics flow back to Dagster and monitoring systems. - The approach provides predictable, reproducible jobs with centralized visibility. - Discord’s ad relevance model now trains daily without engineers manually editing cluster configurations. ## Centralized Observability with X-Ray - Discord built X-Ray as a web UI for monitoring Ray infrastructure. - It displays: - Active clusters - Cluster ownership - Machine types - Current status - Engineers can inspect dashboards and launch interactive notebooks for experimentation from one place. ## Ads Ranking as a Production Test - Ads Ranking determines which Quest advertisements are most relevant to individual users. - Before Ray, the system relied on XGBoost and lacked: - Model sharding - Multi-GPU support - Scalable, frequent retraining - Ray enabled sharded neural networks trained on multi-GPU clusters. - Reported results included: - Twice as many players joining Quests - Ad coverage increasing from roughly 40% to nearly 100% - A production pipeline that retrains daily and continuously delivers new model versions Discord’s experience suggests that distributed ML succeeds when powerful infrastructure is paired with simple interfaces, automated orchestration, and strong observability. Organizations adopting Ray should build comparable platform tooling around it rather than expecting ML engineers to manage clusters, scheduling, and monitoring themselves.

Read original

Xgboost Kubernetes Ray Gpu Computing+3