Supervised Fine Tuning

googleMar 4, 20263 min readCurated summary

Teaching LLMs to reason like Bayesians

LLMs often struggle to update their beliefs as new evidence arrives, relying instead on simplistic heuristics. Google Research tested whether training models to imitate an optimal Bayesian assistant could improve this capability. The results show that Bayesian teaching substantially improves recommendation accuracy, adaptation across interactions, and generalization to other tasks—more effectively than training on always-correct answers. ## Testing Bayesian Reasoning in LLMs - Researchers created a five-round flight recommendation task involving three options with different: - Departure times - Flight durations - Number of stops - Costs - Simulated users had hidden preferences, such as strong, weak, or no preference for high or low values of each feature. - After every recommendation, the user revealed the correct choice, giving the assistant new evidence. - The benchmark compared: - Off-the-shelf LLMs - Human participants - An optimal Bayesian assistant - The Bayesian assistant maintained a probability distribution over possible user preferences and updated it using Bayes’ rule. - Most LLMs performed substantially worse and often stopped improving after the first interaction, showing limited ability to incorporate information over time. - Humans improved more than most LLMs but still failed to match the Bayesian assistant. ## Bayesian Teaching Framework - Bayesian reasoning requires an agent to: - Start with a prior belief about the world - Incorporate new evidence - Produce a posterior belief - Use that posterior as the prior for future reasoning - For LLMs, the “world state” includes facts, relationships, concepts, and inferred user preferences. - Researchers used supervised fine-tuning on many simulated user interactions to teach models this update process. ## Oracle Teaching vs. Bayesian Teaching - **Oracle teaching** trained models on interactions with an assistant that knew the user’s preferences perfectly and always selected the correct option. - **Bayesian teaching** trained models to imitate an assistant that estimated preferences probabilistically and sometimes made mistakes, especially during early uncertain rounds. - The researchers argued that Bayesian examples better preserve uncertainty and demonstrate how beliefs should change as evidence accumulates. - This approach resembles knowledge distillation: the LLM learns to reproduce the predictions of a more principled teacher rather than memorizing only correct outcomes. ## Results and Generalization - Both fine-tuning strategies improved performance compared with the original LLMs. - Bayesian teaching consistently outperformed oracle teaching. - Models trained on Bayesian predictions more often agreed with the optimal Bayesian assistant. - Improvements extended beyond the original flight recommendation task, suggesting the models learned a broader approximation of probabilistic reasoning rather than merely memorizing task-specific patterns. - The findings indicate that LLMs can acquire reasoning strategies from examples and apply them in new domains. The practical implication is that training models on the behavior of an optimal probabilistic reasoner may be more effective than supplying only correct answers. For agents that must learn user preferences or update beliefs over time, examples that explicitly preserve uncertainty and demonstrate evidence-based belief revision could produce more reliable behavior.

Read original

Supervised Fine Tuning Large Language Models Bayesian Inference Probabilistic Reasoning+3

kakaoJan 15, 2026Original article

Kanana-2 Development Log ( (opens in new tab)

Kakao’s development of the Kanana-2 model family represents a strategic shift toward Agentic AI, prioritizing complex reasoning and execution capabilities over simple conversational fluency. By implementing a sophisticated post-training pipeline—including a specialized Mid-training stage and refined reinforcement learning—the team successfully enhanced the model's instruction-following and tool-calling performance. This methodology ensures that the 30B parameter models excel in logical tasks and real-world agentic environments while maintaining high linguistic stability in both English and Korean. ## Mid-training and Catastrophic Forgetting Prevention * A 250B token Mid-training stage was introduced between Pre-training and Post-training to bridge the gap in reasoning, coding, and tool-calling capabilities. * The dataset comprised 200B tokens of high-quality reasoning data (Chain-of-Thought math and code) and 50B tokens of "replay" data from the original pre-training set. * This replay strategy specifically targeted "Catastrophic Forgetting," preventing the model from losing its Korean linguistic nuances and performance on benchmarks like KoMT-bench while it gained English-heavy reasoning skills. * Experimental results indicated that Mid-training serves as a foundational "force multiplier," leading to faster convergence and higher performance ceilings during subsequent Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) stages. ## Enhanced Instruction Following and Tool Calling * To optimize for Agentic AI, the developers focused on Instruction Following (IFEval) by synthesizing high-quality, long-form responses that strictly adhere to complex constraints. * Tool-calling capabilities were improved using "Rejection Sampling" (Iterative SFT), where model-generated trajectories are validated in a real execution environment; only successful outcomes are retained for training. * The training data was categorized into distinct buckets—such as Chat, Math, Code, and Tool Calling—allowing for a more balanced recipe compared to previous Kanana versions. * This approach specifically addressed multi-turn and multi-tool scenarios, ensuring the model can handle the recursive logic required for autonomous agents. ## Parallel Reinforcement Learning and Calibration Tuning * A "Parallel RL" framework was adopted to optimize different capabilities simultaneously: the "Chat" track focused on helpfulness and safety, while the "Logic" track focused on accuracy in math and programming. * The pipeline moved beyond standard SFT to include Reinforcement Learning from Human Feedback (RLHF), utilizing DPO and PPO-style methods to align the model with human preferences. * A final "Calibration Tuning" step was implemented to ensure the model’s internal confidence levels match its actual accuracy, effectively reducing hallucinations and improving reliability in technical tasks. * Comparative benchmarks show that the Kanana-2 Instruct and Thinking models significantly outperform earlier versions and rival larger open-source models in reasoning and coding benchmarks like HumanEval and GSM8K. The Kanana-2 development cycle demonstrates that achieving "Agentic" performance requires more than just scaling data; it requires a structured transition from general language understanding to execution-verified reasoning. For organizations building AI agents, the Kanana-2 post-training recipe suggests that integrating environment-validated feedback and balancing reasoning data with foundational language "replays" is critical for creating reliable, multi-functional models.

Supervised Fine Tuning AI Large Language Models AI Agents+5

netflixOct 25, 2025Original article

Post-Training Generative Recommenders with Advantage-Weighted Supervised Finetuning | by Netflix Technology Blog | Netflix TechBlog (opens in new tab)

Netflix is evolving its recommendation systems by moving beyond simple behavior imitation toward generative recommenders that better align with true user preferences. While generative models like HSTU and OneRec effectively capture sequential user patterns, they often struggle to distinguish between habitual clicks and genuine satisfaction. To bridge this gap, Netflix developed Advantage-Weighted Supervised Fine-tuning (A-SFT), a post-training method that leverages noisy reward signals to refine model performance without the need for complex counterfactual data. ### The Shift to Generative Recommenders * Modern generative recommenders (GRs), such as HSTU and OneRec, utilize transformer architectures to treat recommendation as a sequential transduction task. * The models are typically trained using next-item prediction, where the system learns to imitate the chronological sequence of a user’s activities. * A significant drawback of this "behavior cloning" approach is that it captures external trends and noise rather than long-term user satisfaction, potentially recommending content the user finished but did not actually enjoy. ### Barriers to Reinforcement Learning in RecSys * Traditional post-training methods used in Large Language Models, such as Proximal Policy Optimization (PPO) or Direct Preference Optimization (DPO), require counterfactual feedback that is difficult to obtain in recommendation contexts. * Because user sequences span weeks or years, it is impractical to generate and test hypothetical, counterfactual experiences for real-time user validation. * Reward signals in recommendation systems are inherently noisy; for instance, high watch time might indicate interest, but it can also be a result of external circumstances, making it an unreliable metric for optimization. ### Advantage-Weighted Supervised Fine-tuning (A-SFT) * A-SFT is a hybrid approach that sits between offline reinforcement learning and standard supervised fine-tuning. * The algorithm incorporates an advantage function to weight training examples, allowing the model to prioritize actions that lead to higher rewards while filtering out noise from the reward model. * This method is specifically designed to handle high-variance reward signals, using them as directional guides rather than absolute truth, which prevents the model from over-exploiting inaccurate data. * Benchmarks against other representative methods show that A-SFT achieves superior alignment between the generative recommendation policy and the underlying reward model. For organizations managing large-scale recommendation engines, A-SFT offers a practical path to implementing post-training improvements. By focusing on advantage-weighted signals, developers can improve recommendation quality using existing implicit feedback—like watch time and clicks—without the infrastructure hurdles of online reinforcement learning.

Supervised Fine Tuning AI Machine Learning Transformer+4

googleJun 2, 2025Original article

Learning to clarify: Multi-turn conversations with Action-Based Contrastive Self-Training (opens in new tab)

Action-Based Contrastive Self-Training (ACT) is a novel approach designed to enhance the multi-turn conversational capabilities of large language models, specifically their ability to ask clarifying questions when faced with ambiguity. While standard models often default to guessing a user's intent or overhedging, ACT optimizes conversational action planning as an implicit subtask of response generation. This method demonstrates that data-efficient tuning can significantly improve dialogue policy learning and reasoning in complex, mixed-initiative interactive scenarios. ## Implicit Action Planning * Traditional conversational agents use separate modules for dialogue planning (deciding when to clarify) and response generation. * ACT introduces "implicit action planning," which integrates these steps by teaching the model to perform planning as an inherent part of the end-to-end generation process. * This approach addresses the limitations of standard Direct Preference Optimization (DPO), which often fails to account for the long-term, multi-turn consequences of specific dialogue actions. ## Action-Based Contrastive Data Generation * The first phase involves building a preference dataset by identifying "winning" and "losing" actions for specific conversation turns. * Using an existing dataset, the system identifies a successful turn (e.g., a clarifying question) as the winning response. * A synthetic "rejected" response is then generated to represent a converse, less-optimal action (e.g., attempting to answer despite ambiguity). * This creates a pairwise dataset that contrastively defines successful versus unsuccessful conversational strategies. ## Quasi-Online Contrastive Self-Training * Instead of relying solely on static, offline pairs, ACT employs on-policy sampling to simulate the multi-turn trajectory of a response. * The model evaluates whether a sampled response (such as a clarifying question) leads to a successful final outcome based on the user's original intent. * If the simulated trajectory is successful, it replaces the winning response in the DPO update; if it fails, it is used to refine the losing response. * This quasi-online feedback loop ensures the model is optimized based on the actual outcomes of its conversational decisions rather than just single-turn labels. ## Evaluation and the AmbigSQL Benchmark * The researchers introduced AmbigSQL, a new benchmark task focusing on disambiguating information-seeking requests for complex SQL code generation. * ACT was also tested on real-world tasks including tabular-grounded question-answering and machine reading comprehension. * Experimental results show that ACT substantially outperforms standard Supervised Fine-Tuning (SFT) and standard DPO in multi-turn conversation modeling. By focusing on the downstream consequences of dialogue actions, ACT provides a practical framework for developers to build more "mixed-initiative" agents that know when to stop and ask for clarification, ultimately leading to higher accuracy in complex data-seeking tasks.

Supervised Fine Tuning Large Language Models Reinforcement Learning Action Based Contrastive Self Training+4