netflix·Jul 30, 2026·4 min readCurated summary

GenRec: Towards LLM-Native Recommendation at Netflix

GenRec is Netflix’s LLM-backed recommendation ranker, designed to reduce dependence on thousands of hand-engineered features. It verbalizes user history, item metadata, and context, then post-trains a Netflix-adapted foundation model for catalog-aware ranking and long-term member value. In large-scale A/B testing, it reportedly improved both short- and long-term metrics while using far fewer labeled examples and input signals than an established production ranker. ## Motivation for an LLM-Native Recommender - Netflix’s existing recommendation stack supports many content types and product surfaces but is costly to extend. - New use cases can require substantial feature engineering, architectural changes, infrastructure work, and experimentation. - LLMs offer: - Shared semantic representations for users, items, and interactions - Natural-language steering through prompts - Rich understanding of content and user histories - General-purpose LLMs are not production-ready on their own because they may: - Over-recommend globally popular titles - Hallucinate items outside the catalog - Ignore business constraints - Provide limited personalization ## Ranking Problem and Long-Term Utility - GenRec ranks the full Netflix catalog, or a provided candidate set. - It uses the user, interaction history, current context, and time to produce a personalized ranking. - The optimization target is expected long-term member utility—a proxy for satisfaction and retention—rather than only immediate clicks or plays. ## Two-Phase Training ### Netflix-Adapted Foundation Model - Netflix first adapts an open-source LLM using proprietary Netflix data. - The model learns: - Netflix content and metadata - Member behavior and preference patterns - General language understanding and generation - This relatively stable foundation can support multiple Netflix applications. ### GenRec Post-Training - A second training phase specializes the foundation model for recommendation. - It focuses on ranking quality, steering, reward alignment, and serving-cost constraints. - This phase is refreshed more frequently to reflect new content and changing member preferences. ## Interaction Data as Conversations - Hundreds of billions of events—including views, play duration, feedback, add-to-list actions, and abandons—are converted into single- or multi-turn recommender conversations. - Each user message includes verbalized: - Context and profile - Interaction history - Item metadata - The recommendation task - Assistant messages represent actual member behavior, such as titles played, viewing duration, or feedback. - During inference, GenRec uses the verbalized request and a catalog-aware scoring head; it does not generate conversational responses. - The conversational format mainly supports language-model training and comprehension of rich textual inputs. ## Context Engineering Instead of Feature Engineering - GenRec represents histories and context as natural language rather than relying primarily on dense, manually designed features. - The token budget becomes the new feature budget, so histories are selectively compressed: - Retain detailed, high-signal events such as long plays and thumbs-up - Remove low-signal events such as brief plays and hovers - Summarize repetitive behavior such as binge-watching - Add detail for important or cold-start items, including new releases - Recent and high-value interactions receive priority, while older information is compressed or dropped. - Prompt structure is also optimized for shared prefixes and prefix caching, reducing serving costs. ## Ranking, Language, and Reward Objectives - GenRec combines several training objectives: - **Catalog-aware ranking:** Uses high-value engagements as positive labels, applies denoising and thresholds, and trains the model with cross-entropy over the catalog or candidate set. - **Language modeling:** Preserves understanding of natural-language histories and metadata and supports potential future features such as recommendation explanations. - **Reward-weighted alignment:** Incorporates business requirements and longer-term satisfaction into training. - Reward signals can help balance content categories such as movies, series, games, live content, and podcasts instead of optimizing only immediate engagement. ## Serving and Results - GenRec runs in prefill-only mode on Netflix’s LLM serving infrastructure, avoiding the cost of decoding generated text. - A catalog-aware scoring head converts the model’s representations into item rankings. - Compared with a mature production ranker, GenRec achieved statistically significant improvements in short- and long-term online metrics. - It did so with a small fraction of the labeled data and input signals used by the existing system. GenRec suggests that recommendation systems can shift from extensive manual feature construction toward careful context engineering, LLM post-training, and reward alignment. The approach is most promising when paired with catalog constraints, efficient serving, and objectives that reflect long-term member value rather than raw engagement alone.

Read original(opens in new tab)

Reinforcement LearningLarge Language ModelsNatural Language ProcessingRecommendation Systems+2

google·Jul 22, 2026·3 min readCurated summary

Towards a quantum computer that learns from its errors

Quantum computers require constant recalibration because analog control signals drift during computation. Google Quantum AI combined reinforcement learning (RL) with quantum error correction (QEC), allowing a system to learn from detected errors and adjust thousands of control parameters while computation continues. Tests on the Willow processor showed improved logical stability, suggesting this approach could support much longer quantum computations. ## The Challenge of Quantum Errors - Quantum systems are highly sensitive to drift in signal frequencies, amplitudes, and phases. - Conventional recalibration requires stopping the entire computation, limiting algorithms that may need to run for days or months. - QEC uses many physical qubits to form logical qubits and converts analog noise into binary error-detection events. - These events indicate that an error occurred within a spacetime region but do not identify its precise cause or location. - Decoders such as AlphaQubit and Tesseract infer corrections, but they do not explain whether errors arose from environmental decoherence or preventable calibration drift. ## Moving Beyond Physics-Based Calibration - Traditional calibration depends on manually designed physical models. - Such models can reach performance limits when hardware behavior involves complex, poorly understood interactions. - Google argues that quantum control may benefit from the same shift toward data-driven learning seen in computer vision, robotics, and protein-folding research. - As quantum hardware improves, remaining errors increasingly reflect subtle phenomena that are difficult to model analytically. ## Using Error Detection as a Learning Signal - An RL agent experiments with control strategies and improves based on the resulting error data. - QEC detection events serve two purposes: - Decoders use them to infer logical corrections. - The RL system uses them to identify drift and refine control parameters. - This enables continuous calibration without interrupting the quantum computation. - The approach can steer thousands of analog control parameters dynamically. ## Results on the Willow Processor - Researchers deliberately introduced control-parameter drift into Google’s Willow superconducting processor. - RL steering improved the logical stability of the error-correcting code by 3.5 times. - After expert, human-guided calibration, RL fine-tuning reduced the logical error rate by an additional 20%. - Combined improvements produced fewer than one logical error per 1,000 surface-code correction cycles and fewer than one per 100 color-code cycles. - The processor therefore operated as a more reliable quantum memory for longer periods. ## Scaling to Larger Systems - Simulations included hundreds of qubits and tens of thousands of control parameters. - The RL agent reduced initially high physical error rates by learning better control settings. - QEC suppressed the logical error rate exponentially as the number of physical qubits increased. - The simulations indicated that the number of RL training iterations needed to reduce physical errors did not depend on system size, supporting potential scalability. The results suggest that future quantum computers could use QEC not only to correct errors but also to learn their causes and continuously adapt to hardware drift. RL-based calibration could reduce dependence on manual tuning and help make long-running, fault-tolerant quantum computation practical.

Read original(opens in new tab)

Reinforcement LearningMachine LearningQuantum ComputingSuperconducting Qubits+4

netflix·Jun 29, 2026·3 min readCurated summary

GenPage: Towards End-to-End Generative Homepage Construction at Netflix

GenPage is Netflix’s end-to-end generative approach to building personalized homepages. Instead of separately ranking rows and items, one transformer autoregressively generates the entire page—including rows, entities, and layout—from user and request context. In production, it outperformed Netflix’s mature multi-stage recommender on a core engagement metric while reducing serving latency by 20%. ## Reframing Homepage Recommendation - Netflix’s homepage is a personalized two-dimensional structure, not a single ranked list. - Traditional systems use separate candidate-generation and ranking stages for rows and entities. - GenPage treats homepage construction as a prompt-response task: - The prompt contains user history, profile information, and request context. - The response is the complete homepage generated autoregressively. - The approach aims to: - Replace complex multi-stage pipelines with one end-to-end model. - Optimize the whole page using reinforcement learning. - Capture interactions such as diversity and the trade-off between high-value rows and continued browsing. - Scale more predictably with additional data, compute, and model capacity. - Support new content types, layouts, UI components, and personalized artwork with fewer architectural changes. ## Production Challenges and Results - Real-time generation makes serving latency a major constraint. - The system must address: - Cold-start entities in a constantly changing catalog. - Shifting user interests and cultural trends. - Product and business rules that constrain generated pages. - An online A/B test against Netflix’s optimized production recommender produced: - Statistically significant improvement on Netflix’s primary launch engagement metric. - A 20% reduction in end-to-end serving latency. - Offline experiments found that: - Improving the prompt helped more than increasing model capacity in the tested regime. - Reinforcement-learning post-training improved homepage diversity, even though diversity was not an explicit objective. ## Tokenizing Context and Pages - Each training example contains: - **Context:** user history, profile attributes, and request information. - **Page:** displayed rows and entities in layout order. - **Feedback:** interactions such as plays, thumbs-up, and abandonment. - Context and page are tokenized as model inputs and outputs. - Feedback is used to derive reward and supervision signals rather than being directly generated. ## Domain-Specific Tokenization - GenPage uses a custom recommender-system tokenizer instead of a general-purpose text tokenizer. - This reduces sequence length and improves inference cost and latency. - For example, an action such as watching *Orange Is the New Black* can be represented with four tokens: - Entity ID - Action type - Time bucket - Duration bucket - Direct token mappings to product concepts, such as rows and entities, also make it easier to enforce generation rules and business constraints. ## Context Representation - User-history tokens encode: - Action type - Entity ID - Timestamp - Duration - The history includes explicit signals, such as playback, adding titles to My List, and thumbs-up, as well as implicit signals such as trailer views and detail-page visits. - Profile tokens represent attributes including language and profile type. - Request-context tokens include time of day, day of week, and device. - Long data sources, such as complete impression histories, are summarized to control sequence length and cost. - These summaries improve practicality but introduce handcrafted prompt engineering; learning to compress such information end to end remains a future direction. - Special segment markers help the model distinguish between different context sources.

Read original(opens in new tab)

Reinforcement LearningGenerative AITransformerPersonalization+2

netflix·Jun 19, 2026·3 min readCurated summary

Thinking Fast & Slow for a Personalized Notification System

Netflix’s personalized notification system separates long-term messaging strategy from real-time content selection. A “slow” policy sets each member’s personalized weekly pacing plan, while a “fast” policy chooses the best message when an opportunity arises. This hierarchy addresses the limits of short-term optimization by balancing immediate engagement with fatigue, opt-outs, and long-term member experience. ## Limitations of the Previous System - The earlier system used a causal model to estimate the short-term incremental effect of sending a single notification. - It optimized immediate actions, but could not capture cumulative effects such as: - Notification fatigue - Declining responsiveness over time - Sustained viewing behavior - Gradual opt-out risk - Send frequency and message ranking were coupled: - A relevance threshold implicitly controlled overall send volume. - Changing the threshold affected both frequency and the quality or distribution of selected messages. - Frequency could not be explicitly personalized according to each member’s engagement patterns. ## The Hierarchical Slow-Fast Architecture - The **Slow policy** makes strategic decisions over a longer horizon, such as a week. - It selects a personalized “Pacing Plan Action” that defines intended push and email frequencies. - The action space contains roughly 100 combinations of cross-channel pacing strategies. - The **Fast policy** operates in real time, selecting the most relevant message within the limits established by the slow policy. ## Utility-Based Strategic Planning The Slow policy chooses the action that maximizes a personalized utility function: `U(member, action) = Σ wₖ · Rewardₖ(member, action) — Cost(action)` - Positive signals estimate whether a member will value and engage with notifications. - Negative signals estimate fatigue and the likelihood of opting out of a channel. - Explicit negative feedback is sparse, so predicted messaging costs alone are too small to prevent excessive sending. - Netflix adds a universal cost to every message, ensuring that the utility remains well-behaved and discourages “always send” strategies. - This cost is tuned through online experiments and offline evaluation. ## Pacing Messages Over Time - A basic pacing strategy converts the target frequency into a per-opportunity probability. - At each eligible opportunity, the system uses weighted randomization to decide whether to send. - This produces a naturally varied schedule while maintaining the desired expected frequency. - The architecture can also support structured patterns, including: - Day-of-week preferences - User-activity-based pacing - Bursts aligned with product launches ## Communication Between Policies - The Slow policy calculates a member’s plan and stores it in a low-latency feature store. - The Fast policy retrieves that plan whenever a notification opportunity occurs. - This asynchronous event-and-state design lets the planner focus on long-term member health while the executor focuses on immediate relevance. The main recommendation is to decouple notification frequency and pacing from message ranking. A hierarchical system can explicitly manage long-term communication strategy while preserving the responsiveness and personalization of real-time selection.

Read original(opens in new tab)

Reinforcement LearningFeature StorePersonalizationNotification Systems+3

microsoft·Feb 27, 2026·3 min readCurated summary

Engineering and algorithmic interventions for multimodal post-training at Microsoft scale

At production scale, post-training multimodal agents fail for reasons that standard reinforcement-learning literature often overlooks. Heterogeneous tasks, long tool-use trajectories, noisy reward sources, and strict latency and safety requirements can make aggregate reward look healthy while the policy gradient becomes uninformative and important capabilities regress. The post presents interventions designed to preserve useful advantage signals as scale, task diversity, and interaction horizons grow. ## Production-Scale Challenges - Copilot agents must simultaneously handle: - Tool orchestration - Enterprise documents and mixed-media inputs - Content moderation - Multi-step execution - Trajectories range from roughly 100 to more than 2,000 tokens and span 6 to 25+ interaction steps. - Rewards come from programmatic checks, human judgments, and implicit usage signals, each with different noise and latency. - A single scalar reward can hide regressions in robustness, long-horizon planning, or downstream task success. - Aggregate reward may rise while gradient updates increasingly depend on a small, unrepresentative subset of trajectories. ## Staged Objective Curriculum - The team separates: - **Verifiable objectives**, such as tool syntax and format compliance - **Preference objectives**, such as tool choice and response quality - Training uses only verifiable objectives during the first 30%. - Preference signals are then introduced linearly. - An entropy floor, implemented through a KL penalty activated below a threshold, prevents premature policy collapse. - Entropy bonuses were insufficient because the issue was not simply exploration; optimization was favoring behaviors that were easy to score. - A 30% warmup worked better than 10% or 50% across task families. - Early text-only supervision could also activate multimodal capabilities more reliably than noisy direct multimodal supervision, assuming adequate cross-modal alignment from pretraining. ## Adaptive Curriculum Based on Estimator Health - The team monitors effective sample size (ESS): `ESS = (Σ wᵢ)² / Σ wᵢ²` - ESS measures how many trajectories meaningfully contribute after importance weighting. - ESS falling below 20% of nominal batch size predicted learning stalls by about 35 epochs. - When ESS drops, the system: - Injects near-miss trajectories from a reservoir buffer - Temporarily increases the KL penalty to limit policy drift - Near-misses worked better than hard negatives because they preserve useful distinctions near the decision boundary instead of merely pushing the policy away from failure. - The intervention maintained ESS above 70%, with approximately 15% additional memory usage. ## Variance-Corrected Normalization - Per-task gradient normalization balances task magnitudes but ignores variance within each task. - Broad categories such as “coding” may contain trajectories ranging from 100 to 2,000 tokens, with very different variance. - Importance weighting can cause long trajectories to dominate the effective gradient even after task-level normalization. - The excerpt ends while introducing the team’s variance-correction approach, so its implementation and results are not included here. The central recommendation is to treat estimator health—not just reward and task metrics—as a first-class training signal. Monitoring ESS, controlling objective timing, and accounting for trajectory variance can help prevent healthy-looking dashboards from masking policy collapse and capability regressions.

Read original(opens in new tab)

Reinforcement LearningLarge Language ModelsPost TrainingAgentic Systems+3

spotify·Feb 19, 2026·3 min readCurated summary

Why We Use Separate Tech Stacks for Personalization and Experimentation | Spotify Engineering

Personalization and experimentation overlap, especially with contextual bandits, but they serve different purposes. Personalization chooses the best experience for each user, while experimentation evaluates whether a system or product change improves outcomes overall. Spotify therefore keeps personalization in its ML stack and uses its experimentation platform, Confidence, to measure the impact of those systems. ## What Personalization Does - Personalization adapts products to individual users instead of optimizing for an “average” user. - Spotify uses it for: - Personalized playlists and discovery recommendations - Search results based on listening and search history - Home-screen shortcuts tailored to user behavior - Recommendation systems use models ranging from regression and random forests to neural networks, LLMs, and reinforcement learning. - These systems combine user characteristics, historical behavior, and real-time context to select recommendations and improve them from user responses. ## Where Experimentation and Personalization Overlap - Traditional A/B tests randomly assign users to variants and identify the best option on average. - Multi-armed bandits adapt traffic allocation, sending more users to better-performing treatments. - Contextual bandits use user features to select different treatments for different users. - This creates a conceptual transition: - A/B tests seek the best average treatment. - Multi-armed bandits efficiently find the best average treatment. - Contextual bandits seek the best treatment for each user or context. - Once contextual information determines treatment assignment, the goal is no longer simply measuring one variant’s average effect. - The relevant comparison becomes the value of the personalization system versus a static treatment or an earlier personalization system. ## Why Contextual Bandits Are Not Experimentation Platforms - A contextual bandit can personalize a checkout button based on factors such as cart contents, age, and location. - The bandit is itself a product feature or recommendation system, not the experiment used to evaluate it. - Teams still need experiments to compare different versions of the bandit and determine whether personalization improves user outcomes. - Measuring average or conditional treatment effects may be less important than evaluating the overall benefit of the personalization system. ## Why Spotify Separates the Stacks - Personalization systems require specialized ML infrastructure for: - Training diverse model types - Managing rich feature sets - Serving models with low latency - Collecting real-time data - Computing recommendations at request time - Putting these capabilities into experimentation tools would either greatly expand their scope or limit the sophistication of personalization. - Combining unrelated responsibilities can create technical debt and operational complexity. - Spotify’s ML platform standardizes the development and deployment of personalization systems. - Its experimentation platform, Confidence, evaluates those systems alongside thousands of other product experiments. ## How the Systems Work Together - Personalization algorithms, including contextual bandits, remain in the ML stack. - The experimentation stack treats a personalization system as a treatment to be evaluated. - This separation lets teams improve recommendation logic independently while using controlled experiments to measure its product impact. - It also avoids confusing dependencies that arise when a bandit is both the experiment mechanism and the system being evaluated. Spotify’s approach is to separate building personalized experiences from measuring their effectiveness. Teams should use ML infrastructure for model development and real-time decision-making, then use an experimentation platform to compare personalization systems and quantify their impact.

Read original(opens in new tab)

Reinforcement LearningMachine LearningRecommendation SystemsAb Testing+4

microsoft·Jan 28, 2026·4 min readCurated summary

Diagnosing instability in production-scale agent reinforcement learning

Hugging Face has integrated its Post-Training Toolkit into TRL, bringing production-ready diagnostics to reinforcement learning and agent post-training pipelines. The work identifies a late-phase instability specific to tool-using, on-policy agents: variance can grow in post-tool contexts even while loss, reward, entropy, and global KL remain stable. Targeted tail, distributional, and effective-sample-size diagnostics can expose this failure before it becomes divergence. ## Production Monitoring for Long-Running Agents - Modern agent training runs over long horizons, uses external tools, and adapts continuously. - Failures often develop gradually rather than appearing as a single catastrophic event. - Standard aggregate metrics can hide rare but increasingly severe updates. - The proposed monitoring approach: - Computes diagnostics in-stream. - Separates text-only and post-tool interactions. - Aggregates statistics across workers. - Uses lightweight rolling windows and percentile tracking at fixed intervals. ## Tool-Conditioned Variance Amplification - Tool calls expand the state space through external transitions, exposing the policy to contexts it may rarely encounter in the reference distribution. - Training states can be modeled as: `d(s) = (1−α)·d_text(s) + α·d_tool(s)` - As the proportion of tool-conditioned states, `α`, increases, more updates occur where the reference policy assigns low probability to sampled actions. - This causes importance-weighted updates to develop increasingly large tails. - The mechanism is distinct from global entropy collapse or optimizer instability, though those factors may interact with it. ## Minimal Reproduction and Tail Diagnostics - A small on-policy experiment with an instruction-tuned open-weight model reproduced the pattern. - The 95th percentile of absolute per-token log-ratios, `|r|`, was tracked separately for text-only and post-tool contexts. - Findings included: - Text-only tail magnitudes remained stable or declined. - Post-tool tails grew steadily under fixed-policy baselines. - Drift-aware training substantially reduced tail growth. - Constraining tool outputs also suppressed the effect. - Aggregate loss, reward, and entropy remained stable while the tail was expanding. ## Distributional Shift in the Right Tail - Empirical CDFs across early, middle, and late training showed a change in distribution shape rather than a simple threshold crossing. - In tool-conditioned contexts: - The right tail flattened and stretched. - More probability mass moved toward high-magnitude updates. - Drift-aware baselines muted or reversed the shift. - This supports a distributional explanation rather than an artifact of choosing a particular percentile. ## Importance Ratios and Effective Sample Size - For ratio-based on-policy objectives, gradient variance is related to: `Var[ĝ] ∝ E[(π_θ(a|s) / π_ref(a|s))²]` - When `π_ref(a|s)` is small in tool-conditioned states, a small number of updates can dominate the estimator. - Larger batches and improved baselines may reduce noise but do not fix poor support overlap. - Effective sample size (ESS) provides a supporting signal: - ESS declines as importance weights become concentrated. - It is sensitive to window size and batch structure. - Its trends align with post-tool tail growth, but absolute values should not be over-interpreted. ## Delayed Failure and Misdiagnosis - Instability appears first in tool-conditioned contexts and may remain invisible in global metrics for a long time. - By the time aggregate metrics change, substantial variance amplification may already have accumulated. - The problem is often incorrectly attributed solely to optimizer behavior or inadequate global variance reduction. - Such interventions may delay failure without addressing the underlying support mismatch. - The mechanism is less pronounced when tool outputs are tightly constrained, policies are effectively frozen after tool calls, or interaction diversity plateaus early. The practical recommendation is to add slice-aware, tail-focused diagnostics to production TRL pipelines. Monitoring post-tool log-ratio percentiles, distributional changes, and supporting ESS trends can provide an early warning system for instability that global loss, reward, entropy, and KL metrics miss.

Read original(opens in new tab)

Reinforcement LearningAI AgentsPost TrainingTrl+3

kakao·Jan 15, 2026Original article

Kanana-2 Development Log ( (opens in new tab)

Kakao’s development of the Kanana-2 model family represents a strategic shift toward Agentic AI, prioritizing complex reasoning and execution capabilities over simple conversational fluency. By implementing a sophisticated post-training pipeline—including a specialized Mid-training stage and refined reinforcement learning—the team successfully enhanced the model's instruction-following and tool-calling performance. This methodology ensures that the 30B parameter models excel in logical tasks and real-world agentic environments while maintaining high linguistic stability in both English and Korean. ## Mid-training and Catastrophic Forgetting Prevention * A 250B token Mid-training stage was introduced between Pre-training and Post-training to bridge the gap in reasoning, coding, and tool-calling capabilities. * The dataset comprised 200B tokens of high-quality reasoning data (Chain-of-Thought math and code) and 50B tokens of "replay" data from the original pre-training set. * This replay strategy specifically targeted "Catastrophic Forgetting," preventing the model from losing its Korean linguistic nuances and performance on benchmarks like KoMT-bench while it gained English-heavy reasoning skills. * Experimental results indicated that Mid-training serves as a foundational "force multiplier," leading to faster convergence and higher performance ceilings during subsequent Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) stages. ## Enhanced Instruction Following and Tool Calling * To optimize for Agentic AI, the developers focused on Instruction Following (IFEval) by synthesizing high-quality, long-form responses that strictly adhere to complex constraints. * Tool-calling capabilities were improved using "Rejection Sampling" (Iterative SFT), where model-generated trajectories are validated in a real execution environment; only successful outcomes are retained for training. * The training data was categorized into distinct buckets—such as Chat, Math, Code, and Tool Calling—allowing for a more balanced recipe compared to previous Kanana versions. * This approach specifically addressed multi-turn and multi-tool scenarios, ensuring the model can handle the recursive logic required for autonomous agents. ## Parallel Reinforcement Learning and Calibration Tuning * A "Parallel RL" framework was adopted to optimize different capabilities simultaneously: the "Chat" track focused on helpfulness and safety, while the "Logic" track focused on accuracy in math and programming. * The pipeline moved beyond standard SFT to include Reinforcement Learning from Human Feedback (RLHF), utilizing DPO and PPO-style methods to align the model with human preferences. * A final "Calibration Tuning" step was implemented to ensure the model’s internal confidence levels match its actual accuracy, effectively reducing hallucinations and improving reliability in technical tasks. * Comparative benchmarks show that the Kanana-2 Instruct and Thinking models significantly outperform earlier versions and rival larger open-source models in reasoning and coding benchmarks like HumanEval and GSM8K. The Kanana-2 development cycle demonstrates that achieving "Agentic" performance requires more than just scaling data; it requires a structured transition from general language understanding to execution-verified reasoning. For organizations building AI agents, the Kanana-2 post-training recipe suggests that integrating environment-validated feedback and balancing reasoning data with foundational language "replays" is critical for creating reliable, multi-functional models.

Reinforcement LearningAILarge Language ModelsAI Agents+5

aws·Dec 3, 2025Original article

Amazon Bedrock adds reinforcement fine-tuning simplifying how developers build smarter, more accurate AI models (opens in new tab)

Amazon Bedrock has introduced reinforcement fine-tuning, a new model customization capability that allows developers to build more accurate and cost-effective AI models using feedback-driven training. By moving away from the requirement for massive labeled datasets in favor of reward signals, the platform enables average accuracy gains of 66% while automating the complex infrastructure typically associated with advanced machine learning. This approach allows organizations to optimize smaller, faster models for specific business needs without sacrificing performance or incurring the high costs of larger model variants. **Challenges of Traditional Model Customization** * Traditional fine-tuning often requires massive, high-quality labeled datasets and expensive human annotation, which can be a significant barrier for many organizations. * Developers previously had to choose between settle for generic "out-of-the-box" results or managing the high costs and complexity of large-scale infrastructure. * The high barrier to entry for advanced reinforcement learning techniques often required specialized ML expertise that many development teams lack. **Mechanics of Reinforcement Fine-Tuning** * The system uses an iterative feedback loop where models improve based on reward signals that judge the quality of responses against specific business requirements. * Reinforcement Learning with Verifiable Rewards (RLVR) utilizes rule-based graders to provide objective feedback for tasks such as mathematics or code generation. * Reinforcement Learning from AI Feedback (RLAIF) uses AI-driven evaluations to help models understand preference and quality without manual human intervention. * The workflow can be powered by existing API logs within Amazon Bedrock or by uploading training datasets, eliminating the need for complex infrastructure setup. **Performance and Security Advantages** * The technique achieves an average accuracy improvement of 66% over base models, enabling smaller models to perform at the level of much larger alternatives. * Current support includes the Amazon Nova 2 Lite model, which helps developers optimize for both speed and price-to-performance. * All training data and customization processes remain within the secure AWS environment, ensuring that proprietary data is protected and compliant with organizational security standards. Developers should consider reinforcement fine-tuning as a primary strategy for optimizing smaller models like Amazon Nova 2 Lite to achieve high-tier performance at a lower cost. This capability is particularly recommended for specialized tasks like reasoning and coding where objective reward functions can be used to rapidly iterate and improve model accuracy.

Reinforcement LearningAIMachine LearningGenerative AI+5

aws·Dec 3, 2025Original article

New serverless customization in Amazon SageMaker AI accelerates model fine-tuning (opens in new tab)

Amazon SageMaker AI has introduced a new serverless customization capability designed to accelerate the fine-tuning of popular models like Llama, DeepSeek, and Amazon Nova. By automating resource provisioning and providing an intuitive interface for advanced reinforcement learning techniques, this feature reduces the model customization lifecycle from months to days. This end-to-end workflow allows developers to focus on model performance rather than infrastructure management, from initial training through to final deployment. **Automated Infrastructure and Model Support** * The service provides a serverless environment where SageMaker AI automatically selects and provisions compute resources based on the specific model architecture and dataset size. * Supported models include a broad range of high-performance options such as Amazon Nova, DeepSeek, GPT-OSS, Meta Llama, and Qwen. * The feature is accessible directly through the Amazon SageMaker Studio interface, allowing users to manage their entire model catalog in one location. **Advanced Customization and Reinforcement Learning** * Users can choose from several fine-tuning techniques, including traditional Supervised Fine-Tuning (SFT) and more advanced methods. * The platform supports modern optimization techniques such as Direct Preference Optimization (DPO), Reinforcement Learning from Verifiable Rewards (RLVR), and Reinforcement Learning from AI Feedback (RLAIF). * To simplify the process, SageMaker AI provides recommended defaults for hyperparameters like batch size, learning rate, and epochs based on the selected tuning technique. **Experiment Tracking and Security** * The workflow introduces a serverless MLflow application, enabling seamless experiment tracking and performance monitoring without additional setup. * Advanced configuration options allow for fine-grained control over network encryption and storage volume encryption to ensure data security. * The "Continue customization" feature allows for iterative tuning, where users can adjust hyperparameters or apply different techniques to an existing customized model. **Evaluation and Deployment Flexibility** * Built-in evaluation tools allow developers to compare the performance of their customized models against the original base models to verify improvements. * Once a model is finalized, it can be deployed with a few clicks to either Amazon SageMaker or Amazon Bedrock. * A centralized "My Models" dashboard tracks all custom iterations, providing detailed logs and status updates for every training and evaluation job. This serverless approach is highly recommended for teams that need to adapt large language models to specific domains quickly without the operational overhead of managing GPU clusters. By utilizing the integrated evaluation and multi-platform deployment options, organizations can transition from experimentation to production-ready AI more efficiently.

Reinforcement LearningAILarge Language ModelsMachine Learning+5

netflix·Oct 25, 2025Original article

Post-Training Generative Recommenders with Advantage-Weighted Supervised Finetuning | by Netflix Technology Blog | Netflix TechBlog (opens in new tab)

Netflix is evolving its recommendation systems by moving beyond simple behavior imitation toward generative recommenders that better align with true user preferences. While generative models like HSTU and OneRec effectively capture sequential user patterns, they often struggle to distinguish between habitual clicks and genuine satisfaction. To bridge this gap, Netflix developed Advantage-Weighted Supervised Fine-tuning (A-SFT), a post-training method that leverages noisy reward signals to refine model performance without the need for complex counterfactual data. ### The Shift to Generative Recommenders * Modern generative recommenders (GRs), such as HSTU and OneRec, utilize transformer architectures to treat recommendation as a sequential transduction task. * The models are typically trained using next-item prediction, where the system learns to imitate the chronological sequence of a user’s activities. * A significant drawback of this "behavior cloning" approach is that it captures external trends and noise rather than long-term user satisfaction, potentially recommending content the user finished but did not actually enjoy. ### Barriers to Reinforcement Learning in RecSys * Traditional post-training methods used in Large Language Models, such as Proximal Policy Optimization (PPO) or Direct Preference Optimization (DPO), require counterfactual feedback that is difficult to obtain in recommendation contexts. * Because user sequences span weeks or years, it is impractical to generate and test hypothetical, counterfactual experiences for real-time user validation. * Reward signals in recommendation systems are inherently noisy; for instance, high watch time might indicate interest, but it can also be a result of external circumstances, making it an unreliable metric for optimization. ### Advantage-Weighted Supervised Fine-tuning (A-SFT) * A-SFT is a hybrid approach that sits between offline reinforcement learning and standard supervised fine-tuning. * The algorithm incorporates an advantage function to weight training examples, allowing the model to prioritize actions that lead to higher rewards while filtering out noise from the reward model. * This method is specifically designed to handle high-variance reward signals, using them as directional guides rather than absolute truth, which prevents the model from over-exploiting inaccurate data. * Benchmarks against other representative methods show that A-SFT achieves superior alignment between the generative recommendation policy and the underlying reward model. For organizations managing large-scale recommendation engines, A-SFT offers a practical path to implementing post-training improvements. By focusing on advantage-weighted signals, developers can improve recommendation quality using existing implicit feedback—like watch time and clicks—without the infrastructure hurdles of online reinforcement learning.

Reinforcement LearningAIMachine LearningTransformer+4

google·Oct 1, 2025Original article

A collaborative approach to image generation (opens in new tab)

Google Research has introduced PASTA (Preference Adaptive and Sequential Text-to-image Agent), a reinforcement learning agent designed to transform image generation from a single-prompt task into a collaborative, multi-turn dialogue. By learning individual user preferences through sequential interactions, the system eliminates the frustration of trial-and-error prompting to achieve a specific creative vision. ## Data Strategy and User Simulation * Researchers collected a foundational dataset featuring over 7,000 human interactions, using Gemini Flash for prompt expansion and Stable Diffusion XL (SDXL) for image generation. * To overcome the scarcity of real-world interaction data, the team developed a user simulator that generated over 30,000 additional interaction trajectories. * The simulator is built on two primary components: a utility model that predicts how much a user will like an image, and a choice model that predicts which image a user will select from a given set. ## Latent Preference Discovery * The architecture utilizes pre-trained CLIP encoders paired with user-specific components to capture nuanced aesthetic tastes. * An expectation-maximization (EM) algorithm is employed to identify "user types," allowing the system to cluster users with similar interests, such as a preference for specific artistic styles or subject matter like "Food" or "Animals." * This approach enables the model to generalize preferences quickly, allowing it to adapt to new users based on minimal initial feedback. ## The Collaborative Generation Loop * PASTA operates as a value-based reinforcement learning model that aims to maximize cumulative user satisfaction across an entire interaction session. * The workflow begins with a candidate generator creating diverse prompt expansions; a candidate selector then picks an optimal "slate" of four variations to present to the user. * Each user selection provides a feedback signal that guides the agent’s next set of suggestions, iteratively narrowing the gap between the generated output and the user's intent. ## Training and Performance Validation * The agent was trained using Implicit Q-learning (IQL) to optimize decision-making without requiring online interaction during the training phase. * Performance was measured using several metrics, including Pick-a-Pic accuracy, Spearman’s rank correlation, and cross-turn accuracy. * Results indicated that agents trained on a combination of real-world and simulated data significantly outperformed baseline models and versions trained on only one data type. PASTA demonstrates that integrating iterative feedback loops and reinforcement learning can effectively bridge the "intent gap" in generative AI. For developers building creative tools, this research suggests that move-away from static prompting toward adaptive, simulation-trained agents can provide a more satisfying and intuitive user experience.

Reinforcement LearningAIText To ImageClip+5

google·Jun 2, 2025Original article

Learning to clarify: Multi-turn conversations with Action-Based Contrastive Self-Training (opens in new tab)

Action-Based Contrastive Self-Training (ACT) is a novel approach designed to enhance the multi-turn conversational capabilities of large language models, specifically their ability to ask clarifying questions when faced with ambiguity. While standard models often default to guessing a user's intent or overhedging, ACT optimizes conversational action planning as an implicit subtask of response generation. This method demonstrates that data-efficient tuning can significantly improve dialogue policy learning and reasoning in complex, mixed-initiative interactive scenarios. ## Implicit Action Planning * Traditional conversational agents use separate modules for dialogue planning (deciding when to clarify) and response generation. * ACT introduces "implicit action planning," which integrates these steps by teaching the model to perform planning as an inherent part of the end-to-end generation process. * This approach addresses the limitations of standard Direct Preference Optimization (DPO), which often fails to account for the long-term, multi-turn consequences of specific dialogue actions. ## Action-Based Contrastive Data Generation * The first phase involves building a preference dataset by identifying "winning" and "losing" actions for specific conversation turns. * Using an existing dataset, the system identifies a successful turn (e.g., a clarifying question) as the winning response. * A synthetic "rejected" response is then generated to represent a converse, less-optimal action (e.g., attempting to answer despite ambiguity). * This creates a pairwise dataset that contrastively defines successful versus unsuccessful conversational strategies. ## Quasi-Online Contrastive Self-Training * Instead of relying solely on static, offline pairs, ACT employs on-policy sampling to simulate the multi-turn trajectory of a response. * The model evaluates whether a sampled response (such as a clarifying question) leads to a successful final outcome based on the user's original intent. * If the simulated trajectory is successful, it replaces the winning response in the DPO update; if it fails, it is used to refine the losing response. * This quasi-online feedback loop ensures the model is optimized based on the actual outcomes of its conversational decisions rather than just single-turn labels. ## Evaluation and the AmbigSQL Benchmark * The researchers introduced AmbigSQL, a new benchmark task focusing on disambiguating information-seeking requests for complex SQL code generation. * ACT was also tested on real-world tasks including tabular-grounded question-answering and machine reading comprehension. * Experimental results show that ACT substantially outperforms standard Supervised Fine-Tuning (SFT) and standard DPO in multi-turn conversation modeling. By focusing on the downstream consequences of dialogue actions, ACT provides a practical framework for developers to build more "mixed-initiative" agents that know when to stop and ask for clarification, ultimately leading to higher accuracy in complex data-seeking tasks.

Reinforcement LearningLarge Language ModelsSupervised Fine TuningAction Based Contrastive Self Training+4