github·Jun 25, 2026·3 min readCurated summary

Evaluating performance and efficiency of the GitHub Copilot agentic harness across models and tasks

GitHub argues that an agentic harness—the system coordinating tools, context, and workflow—strongly influences how effectively an AI model solves software tasks. Its shared Copilot harness aims to match model-vendor harnesses in task completion while using fewer tokens. Benchmark results across four models suggest near-parity overall, with performance varying by model and task. ## Benchmarking Approach - GitHub evaluates the harness using public and internal benchmarks, plus real-world metrics and online experiments. - Comparisons hold the following constant: - The same underlying model - The same task - Context-window size - Reasoning effort - Tool selection - MCP servers - Copilot CLI is compared with: - Claude Code for Claude Sonnet 4.6 and Claude Opus 4.7 - Codex CLI for GPT-5.4 and GPT-5.5 - Benchmarks include: - **SWE-bench Verified:** 500 human-validated Python bug fixes - **SWE-bench Pro:** More complex, multi-step engineering tasks - **SkillsBench:** Skill usage and triggering - **TerminalBench:** Command-line workflows - **Win-Hill:** Software tasks in Windows containers ## Token Efficiency - Copilot’s harness generally completes tasks at rates comparable to model-vendor harnesses while consuming fewer tokens. - It performed better across the reported tests for Sonnet 4.6 and Opus 4.7. - For GPT-5.4 and GPT-5.5, Copilot performed better on nearly every benchmark. - The main exception was SWE-bench Verified, where Copilot was 7% worse with GPT-5.4 and 4% worse with GPT-5.5. ## Task Resolution - GitHub emphasizes that lower token use matters only when the agent successfully completes the work. - Overall task-resolution rates were considered on par with vendor harnesses. - Results varied by benchmark: - Copilot generally did better on SWE-bench Pro, especially with GPT models. - It outperformed vendor harnesses on Win-Hill or matched them for every model. - It performed better on TerminalBench with Sonnet and Opus, matched GPT-5.5, and trailed GPT-5.4. - SkillsBench results favored GPT models on Copilot but favored vendor harnesses for Claude models. - Differences are described as statistically comparable because model behavior is stochastic and run-to-run variation can explain many gaps. ## TerminalBench Variance Analysis - GitHub uses TerminalBench 2.0 to study both cost and completion rate over repeated runs. - The preferred outcome is higher resolution with lower cost. - The analysis illustrates that benchmark results should account for variance rather than relying on a single run. - GitHub presents Copilot as equal to or better than the vendor harnesses on this cost-versus-success comparison. The practical conclusion is that harness design is a reusable performance multiplier across Copilot products. GitHub’s results support using its shared harness when developers need broad model choice, efficient token usage, and comparable task-completion performance, while recognizing that the best harness can still vary by model and workload.

Read original(opens in new tab)

Large Language ModelsAI AgentsModel Context ProtocolGithub Copilot+3

meta·Jun 25, 2026·3 min readCurated summary

Privacy-Aware Infrastructure in the AI-Native Era: An Asset Classification Case Study

Privacy-aware infrastructure depends on accurate asset classification before it can enforce retention, access, purpose, sharing, or anonymization policies. Because data is noisy, distributed, and constantly changing—especially in AI-native systems—LLMs are useful for ambiguity but should not make routine production decisions. The recommended approach combines rich contextual evidence, human-reviewed labels, narrowly used LLMs, and versioned deterministic rules that are faster, replayable, and auditable. ## Why Asset Classification Matters - Assets include more than tables and columns: they may be nested payload fields, logs, event parameters, API fields, ML features, embeddings, or derived datasets. - Classification must track the meaning of data as it moves through pipelines and changes representation. - A field such as `age` could represent sensitive personal information or an infrastructure cache TTL, making context essential. - Four recurring challenges shape the problem: - **Noisy signals:** Raw metadata can overwhelm models and hide relevant evidence. - **Distributed context:** Code, lineage, ownership, documentation, annotations, and usage patterns reside in separate systems. - **Changing requirements:** Product and policy changes can outpace static rules and periodic reviews. - **Enforcement consequences:** False positives cause unnecessary restrictions, while false negatives create protection gaps. - Classification must reason about ambiguity while producing decisions that can later be explained and reproduced. ## The Hybrid Classification Pattern - **Context beats prompts:** Improving the evidence supplied to a model generally matters more than endlessly tuning instructions. - Evidence briefs should organize: - Supporting and contradicting signals - Provenance - Relevant code and lineage - Masked or circular fields that could distort reasoning - **Evaluation must remain independent:** Human-reviewed reference labels, frozen test sets, separate models or prompts, and regression gates prevent the classifier from defining its own ground truth. - **Stable behavior should be distilled into rules:** LLMs handle novelty and uncertainty, while validated patterns become deterministic, versioned, and auditable logic. - Over time, the LLM’s production role should shrink as routine cases move to low-latency deterministic enforcement. ## A Stable Classification Contract - The classifier should operate as a platform service with a small, explicit interface. - Inputs include: - An asset identifier - A structured bundle of contextual evidence - Outputs include: - A taxonomy category - A confidence score calibrated against reviewed labels - A decision trace explaining influential evidence - The matching deterministic rule, when applicable - Versions for the context, rules, and prompt - Classifiers should answer one scoped, domain-specific question rather than use a universal taxonomy. - Narrow classifiers are easier to evaluate, debug, govern, and compose across downstream privacy decisions. ## Privacy-Aware Infrastructure Responsibilities Asset classification supports the broader PAI lifecycle: - Understanding what data exists and how it is governed - Discovering data flows relevant to a policy - Enforcing retention, access, purpose, and sharing constraints - Producing verifiable evidence of compliance ## Practical Recommendation Use LLMs selectively for ambiguous or novel assets, but build the surrounding system around structured context, independent human-reviewed evaluation, and deterministic rule promotion. This preserves the flexibility of AI while making routine privacy enforcement predictable, auditable, and operationally efficient.

Read original(opens in new tab)

Large Language ModelsData GovernanceHuman In The LoopData Lineage+3

dropbox·Jun 25, 2026·2 min readCurated summary

How we used DSPy to turn AI evaluations into better responses in Dash chat

Dropbox uses DSPy to turn AI evaluations into improvements for its Dash chat agent. The process first calibrates LLM judges against human-labeled conversations, then uses those judges to optimize the agent’s system prompt. This feedback loop reduced incomplete answers and token usage while maintaining answer quality. ## The Complexity of Evaluating AI Agents - Agent quality depends on more than the final response: - Understanding user intent - Selecting relevant context - Choosing and using tools - Synthesizing information across documents, messages, and meetings - Handling ambiguity and follow-up turns - Producing grounded, complete answers - Evaluations therefore inspect the full interaction trajectory, not just the output. - Separate evaluations for intent understanding, tool use, context selection, grounding, adaptation, and task completion help identify the source of failures. - Reliable judges were necessary before evaluation results could safely guide agent improvements. ## Calibrating LLM Judges with Human Labels - Dropbox sampled internal chats containing final answers and agent trace logs. - Human reviewers scored five dimensions: - User-intent following - Semantic relevance - Tool calling - Instruction following - Context selection - Reviewers followed a structured process: - Determine whether the agent understood the request. - Check whether it selected appropriate context. - Inspect searches, retrievals, and other tool actions. - Verify that final claims were supported by evidence. - Score relevance, grounding, completeness, and instruction adherence. - Many metrics used a 1–5 scale. - Reviewers also added: - Reasoning notes explaining their scores - Failure codes for issues such as stale evidence, missing context, unsupported claims, incomplete coverage, and poor personalization - These richer annotations helped improve judge prompts while also supporting debugging, error analysis, roadmap planning, and prioritization. ## Using DSPy to Improve Evaluation - DSPy was used to make LLM judges align more closely with human evaluations. - Judges were required to follow a retrospective workflow: - Infer the user’s intent - Inspect the conversation and agent trace - Review supporting evidence - Assess context selection and tool use - Produce scores, failure codes, and reasoning notes - GEPA and MIPROv2, optimization algorithms within DSPy, automatically proposed and tested prompt changes against human-labeled examples. - Optimization supported several scenarios: - Rewriting judge instructions entirely - Adapting a judge to another underlying model - Targeting specific failure modes while preserving the existing evaluation behavior The overall approach creates a scalable improvement loop: human labels calibrate the judges, calibrated judges provide consistent evaluation signals, and those signals guide improvements to the chat agent itself.

Read original(opens in new tab)

Large Language ModelsAI AgentsDspyPrompt Optimization+4

line·Jun 24, 2026·4 min readCurated summary

Designing a Semantic Context OS: Beyond Token Stuffing in Agent Systems

The article argues that larger LLM context windows do not automatically produce better software-engineering agents. In long-running workflows, indiscriminately filling the context window can cause attention dilution, context rot, reasoning failures, and potential data exposure. It proposes a “Semantic Context OS,” a local runtime layer that actively governs context as a finite, structured system resource rather than treating it as an unmanaged text stream. ## The Context Window Is Not RAM - The article uses the “Karpathy metaphor”: - The LLM acts like a CPU: a largely stateless inference engine driven by pretrained parameters. - The context window acts like RAM: volatile working memory containing current state, instructions, telemetry, and runtime data. - Unlike physical RAM, LLM context is probabilistic rather than deterministic: - Traditional RAM provides precise address-based retrieval with predictable performance. - LLM retrieval depends on attention weights across Q, K, and V matrices. - Increasing capacity from 32K tokens to 1M or 2M tokens therefore does not guarantee proportionally better retrieval. Larger sequences also increase computational cost and structural noise. ## Attention Dilution and Long-Context Failure - Large codebases and logs contain substantial irrelevant material, including: - Boilerplate definitions - Unused imports - Duplicate syntax - Repeated utilities and naming patterns - As sequence length grows, the attention calculation `QKᵀ` accumulates entropy and background noise. - Softmax then spreads attention energy across more tokens, weakening the sharp attention peaks needed to retrieve important facts. - This contributes to the “lost in the middle” effect: - Information near the beginning and end of a prompt is often retrieved more reliably. - Retrieval accuracy can fall sharply across the middle portion of the context. - The article considers relying on massive, unmanaged contexts an architectural anti-pattern for tasks such as large-scale code review, dependency tracing, and automated refactoring. ## Context Rot in Long-Running Agents The article defines “context rot” as the degradation of an agent’s working context during extended autonomous tasks. - **Context poisoning** - Raw logs, obsolete errors, and previous execution data accumulate over multiple turns. - The model may treat temporary historical failures as current architectural constraints. - **Context distraction** - Monorepos often contain similar names, overloaded methods, and duplicated helper code. - Broad retrieval can overwhelm the model with structurally similar but logically irrelevant code. - **Context clash** - Old instructions may remain after the plan has evolved. - Contradictory directives can cause indecision, infinite reasoning loops, timeouts, or hallucinations. - The article claims that, without active management, failure rates increase nonlinearly with context depth and may reach roughly 40% in deeply nested codebases. ## Semantic Context OS as an AI Kernel The proposed Semantic Context OS sits between agent application logic and external foundation-model APIs, operating as a localhost loopback proxy at `localhost:8080`. Its responsibilities include: - Treating context as a finite hardware-like resource. - Tracking token lifecycles and state access. - Filtering and isolating data before it reaches the model. - Separating physical token limits from semantic governance. - Protecting downstream inference engines from structural noise and helping prevent intellectual-property leakage. The architecture includes: - A POSIX-like virtual file system for managing state topology. - A proprietary “PathAlign” stage for AST-based code-tree pruning. - An asynchronous “sawtooth” memory model for runtime token optimization. ## MVC: Minimum Viable Context The core MVC pipeline—described as “minimum viable context”—aims to provide only the smallest dense set of information required for the agent’s current reasoning step. Its processing stages include: - **Collection and token mapping** - Gather source files, dependency graphs, and runtime logs. - Map them using the target model’s tokenizer, such as `cl100k_base` or `o200k_base`. - **Structural pruning** - Use static analysis and structural rules to remove compiler comments, unused imports, boilerplate, and unrelated utilities. - The broader design replaces passive string concatenation with active context selection, lifecycle management, and bounded transmission policies. The article concludes that reliable enterprise agents require active context orchestration rather than larger prompts alone. A dedicated governance layer should prune, isolate, and refresh context throughout execution so that models receive minimal, relevant, and internally consistent information.

Read original(opens in new tab)

Large Language ModelsAI AgentsTransformerAbstract Syntax Tree+4

google·Jun 24, 2026·3 min readCurated summary

Thinking to recall: How reasoning unlocks parametric knowledge in LLMs

Reasoning can help LLMs recall simple facts even when no genuine multi-step deduction is required. Experiments with Gemini-2.5 and Qwen3 show that reasoning improves access to facts stored in model weights through two mechanisms: extra reasoning tokens provide computational capacity, while related factual statements prime retrieval. However, natural reasoning remains more effective than empty computation, and self-generated intermediate facts can introduce hallucination risks. ## Measuring the Knowledge Boundary - The researchers use **pass@k** to determine whether a correct answer appears among multiple generated attempts, rather than evaluating only the top answer. - They compare reasoning-enabled and reasoning-disabled versions of Gemini-2.5 Flash, Gemini-2.5 Pro, and Qwen3-32B. - Tests use the closed-book **SimpleQA Verified** and **EntityQuestions** datasets, which mainly contain simple, single-hop factual questions. - Reasoning-enabled models recover answers that are nearly unreachable when reasoning is disabled, showing that the benefit is not limited to solving complex problems. ## The Computational Buffer - Generating additional reasoning tokens gives the model more forward passes and therefore more opportunities to update its internal state. - To isolate this effect, the researchers replace the model’s natural reasoning with repeated meaningless text such as “Let me think.” - This dummy reasoning substantially improves factual recall compared with having reasoning completely turned off. - The effect has limits: - Longer dummy traces eventually produce diminishing returns. - Dummy reasoning never performs as well as natural reasoning. - These results indicate that extra computation helps, but the semantic content of the reasoning trace also contributes. ## Factual Priming - Natural reasoning traces often contain related facts rather than logical deductions. - This resembles **spreading activation** in human memory, where recalling one concept makes related concepts easier to access. - The researchers call this mechanism **factual priming**: the model generates nearby facts that create a contextual bridge to the target fact. - When the researchers extract only concrete facts from reasoning traces—removing filler, search plans, and the target answer itself—those facts recover most of reasoning’s benefit. - For example, when asked for Nepal’s 10th king, the model may recall the first nine kings. Listing those related facts primes retrieval of the requested answer. ## The Hallucination Trap - Generative self-retrieval depends on facts produced by the model during reasoning. - Because those intermediate facts may be hallucinated, factual priming can potentially reinforce incorrect information. - The excerpt introduces this risk but does not provide the researchers’ full evaluation or mitigation findings. The practical conclusion is that reasoning traces can function both as a computational workspace and as a semantic memory primer. For factual recall, systems should preserve useful intermediate retrieval while monitoring or verifying generated facts, since the same mechanism that unlocks obscure knowledge can also amplify errors.

Read original(opens in new tab)

Large Language ModelsChain Of ThoughtParametric MemoryLatent Computation+3

stripe·Jun 23, 2026·3 min readCurated summary

Four travel and hospitality trends from HITEC 2026

Hospitality’s AI opportunity is growing, but most operators lack the data, infrastructure, and operational systems needed to turn investment into measurable returns. AI is reshaping how travelers discover and book hotels, while fragmented data and outdated payment systems create lost revenue and guest frustration. The strongest strategy is to connect accurate data, intelligent workflows, and seamless payments so technology improves the experience without becoming visible to guests. ## AI Is Changing the Direct-Booking Battle - Hotels historically relied on SEO to compete with OTAs such as Expedia and Booking.com. - AI-generated search answers are reducing traditional website traffic: - 65% of Google searches with AI Overviews end without a click. - The figure rises to 78% on mobile. - Traditional search traffic is declining by about 25%. - AI systems prioritize accurate, structured, machine-readable information rather than keyword density and backlinks. - More than 90% of accommodation websites are reportedly undetected by AI models. - Hotels should audit whether AI tools can correctly describe: - Room categories - Amenities - Policies and cancellation terms - Local context - Real-time availability - Winning direct bookings will require both AI discoverability and a modern checkout experience supporting local currencies, payment methods, and fraud protection. ## Hospitality AI Is Held Back by Fragmented Data - Only about 25% of hospitality businesses are actively scaling AI, and fewer than 10% are considered “AI future-built.” - Property management, CRM, loyalty, food and beverage, and payment systems often operate in silos. - Incomplete data weakens: - Personalization - Guest profiles - Financial reconciliation - Operational decision-making - The main challenge is not building AI features but operationalizing them reliably in real workflows. - Successful examples connect live data to timely actions: - Delta’s AI concierge uses customer and operational data to provide context-aware support. - Wynn’s revenue managers receive predictive alerts and recommended actions. - For most operators, better data connectivity matters more than using a more advanced AI model. ## Payment Friction Directly Affects Revenue - Payments are increasingly viewed as a competitive capability rather than a back-office commodity. - Survey findings cited in the article include: - 90% of executives consider payments important to growth. - 37% say limited payment options most harm the guest experience. - 58% report that fraud tools block legitimate transactions. - 74% say fragmented systems create excessive reconciliation work. - Guests may abandon a hotel when their preferred payment method is unavailable, shifting the booking to an OTA that supports it. - Modern payment infrastructure allows smaller operators to offer international payment methods and currencies without building large in-house teams. ## Invisible Technology Creates the Best Guest Experience - Guests have little tolerance for technology failures and may simply avoid returning rather than complain. - Effective hospitality technology should anticipate needs without drawing attention to itself. - The desired experience includes details such as: - A room set to the guest’s preferred temperature - Familiar television channels - Preferred pillow firmness - Hospitality is moving from remembering information guests explicitly provided to predicting preferences based on connected guest data. Operators should prioritize clean, connected data, AI systems tied to real operational actions, and flexible payment infrastructure. The goal is not to add AI for its own sake, but to make booking and stays more seamless while quietly improving revenue, efficiency, and guest loyalty.

Read original(opens in new tab)

Large Language ModelsAIAI AgentsSeo+4

line·Jun 23, 2026·4 min readCurated summary

From Manual to AI Prompt Tuning: Genetic Algorithm–Based Automated Optimization and Acceleration

LY Corporation automated LLM prompt tuning with the GEPA genetic algorithm, reducing a process that previously took days or weeks to roughly one hour. GEPA evolves prompt candidates using evaluation scores and natural-language feedback, allowing it to improve prompts without manually inspecting every output. The approach was applied to Yahoo! JAPAN Search’s AI responses for health and medical queries, balancing policy compliance with improved readability. ## Challenges of Manual Prompt Tuning - Each prompt change requires repeated output generation and human review. - Practical tuning knowledge often remains with individual engineers and is difficult to document or explain. - The cycle of editing, generating, and evaluating responses can take days or weeks. - Model changes and version updates can alter output quality, requiring repeated retuning. - Manual effort leaves less time for defining evaluation criteria, judging quality, and verifying policy compliance. ## Automated Prompt Optimization Approaches - **Reinforcement learning:** Learns prompt-generation policies from scalar rewards, such as with GRPO. - **Bayesian optimization:** Efficiently searches candidate instructions and few-shot examples, as in MIPROv2. - **Genetic algorithms:** Iteratively evolve a population of prompt candidates, as in GEPA. - Genetic methods are well suited to discrete, natural-language prompts because they can use natural-language reflection to identify problems and propose improvements rather than relying only on numerical rewards. ## How GEPA Works - Generates and evaluates multiple prompt candidates. - Uses **Reflective Prompt Mutation** to analyze execution results and create improved instructions. - Applies Pareto-frontier selection to preserve candidates that perform well across multiple evaluation dimensions. - Repeats the process over several to dozens of generations until prompts converge toward the evaluation objectives. - The article notes that GEPA has reportedly outperformed previous optimization methods, including results presented at ICLR 2026. ## Implementation with DSPy - DSPy allows prompt optimization to be controlled programmatically. - A task is defined as a DSPy module with a signature containing input and output fields. - The signature’s docstring becomes an instruction for the LLM. - GEPA rewrites this instruction during optimization. - Separate models can be assigned for: - Task inference - Output evaluation - Reflection and prompt improvement ## Designing the Evaluation Function - GEPA requires an overall scalar score, even when quality is judged across multiple criteria. - Individual scores can be assigned to dimensions such as accuracy, completeness, and style, then normalized and averaged. - The evaluator can also return natural-language feedback through `dspy.Prediction(score=..., feedback=...)`. - Feedback explains why a candidate was penalized, giving GEPA a clearer direction for improvement than a score alone. - Evaluation can use: - LLM-as-a-Judge - Gold answers or labels - Rule-based correctness checks - In the example, an evaluator scores three criteria from 0 to 10, averages them into a single score, and passes the explanation to GEPA for reflection. ## Yahoo! JAPAN Search Health and Medical Queries - Health-related answers must follow stricter policies than general search responses. - Requirements include: - Avoiding definitive medical diagnoses or severity judgments - Matching wording to the strength of available evidence - Recommending medical consultation appropriately - Limiting responses to general explanations where necessary - The project pursued two goals simultaneously: - Satisfy medical and health-policy requirements. - Apply readable Markdown formatting, including headings, lists, and emphasis. - Improving one goal manually could easily damage the other, making automated optimization attractive. ## Applying GEPA to the Production Task - The system takes a search query as input and generates an AI answer. - The initial prompt combined an existing general-purpose prompt with additional health and medical policy instructions. - GEPA rewrote and optimized the instruction section rather than requiring engineers to manually redesign the entire prompt. - The optimization aimed to preserve policy compliance while improving structure and readability. Overall, GEPA with DSPy provides a practical way to shorten prompt-tuning cycles and make the improvement process more reproducible. Its effectiveness depends heavily on carefully designed evaluation criteria and meaningful natural-language feedback, especially for high-risk domains such as medical information.

Read original(opens in new tab)

Large Language ModelsLlm As A JudgePrompt OptimizationDspy+3

line·Jun 22, 2026·4 min readCurated summary

Unifying Analysis Through the Power of Analytics Agents: Work Innovation and Role Transformation in the Generative AI Era at a Professional Organization

PJ One Piece is LY Corporation’s initiative to connect business questions, data analysis, insight generation, and next-action planning through generative AI. Its analysis agent reduced typical turnaround times from about two weeks to roughly 10 minutes, enabling hundreds of analyses each month and adoption by more than half of an early-adopter business unit. The project treats AI not as a chat interface, but as an analysis platform that connects data, knowledge, people, and organizational processes. ## Three Disconnects Behind the Project - **Business and data:** Even with a data warehouse and BI tools, business users still needed to understand SQL, tables, column definitions, KPI rules, and result interpretation. - **Within the analysis process:** Task definition, analysis design, execution, review, and action planning were often handled by different people or tools, causing context loss, rework, delays, and inconsistent quality. - **Across domains:** Useful analysis patterns and domain knowledge remained isolated because services used different KPIs, table structures, business assumptions, and review criteria. ## The Analysis Agent as a Connector - Users ask questions in natural language without needing to know SQL or database structures. - The agent: - Clarifies the business objective and missing assumptions. - Finds relevant data and creates an analysis plan. - Executes queries and specialized analyses. - Interprets results and produces visualizations or reports. - Suggests further analysis and possible next actions. - The platform consists of: - A user-facing application. - An LLM-based agent for reasoning and tool use. - Tools for SQL, Python, document search, and visualization. - A knowledge base containing domain information, skills, and table metadata. - Logging, feedback, monitoring, and evaluation systems. - Domain knowledge is added through a plugin-like structure, while logs and feedback continuously improve the system. ## Turning Business Questions into Analysis Requirements - Natural-language questions often leave important assumptions unspecified, such as: - Target population or campaign definition. - Analysis period and comparison group. - KPI definitions. - Aggregation level. - Exclusion conditions. - Rather than requiring users to write detailed prompts, the agent uses domain knowledge to determine what can be inferred and asks only about unresolved points. - Knowledge bases document service context, KPI definitions, aggregation cautions, policy information, and review requirements. - Table metadata explains available tables, columns, appropriate use cases, samples, partition requirements, and usage restrictions. ## Reaching Data Safely and Reliably - Table metadata is revealed progressively: - The agent first narrows down relevant tables. - It then retrieves detailed definitions and usage rules only for those tables. - Analysis-oriented wide tables or logical views combine transaction data with commonly needed attributes, reducing complicated joins and SQL-generation errors. - SQL is checked before and after execution to enforce: - `SELECT`-only access. - Approved tables and usage rules. - Required partition conditions. - Restrictions on sensitive or personal data. - Result-size limits. - These guardrails allow the agent to perform analysis flexibly without exposing data or infrastructure to unnecessary risks. ## Preserving Context Across the Analysis Process - PJ One Piece uses a supervisor-style multi-agent architecture. - A main agent maintains: - The user’s request and business objective. - The current analysis plan. - Findings and constraints discovered so far. - Remaining questions and decision points. - Specialized sub-agents handle tasks such as statistical testing, time-series analysis, clustering, and independent review. - This separates complex or specialized work from the main context while preserving overall continuity. - Progress updates expose discoveries, design decisions, data limitations, and constraints so users can adjust direction during longer analyses. ## Building Reusable Organizational Capability - Logs record agent actions, assumption checks, analysis designs, generated SQL, errors, and outputs. - User and analyst feedback helps identify whether improvements are needed in prompts, tools, data, or reusable skills. - Repeated workflows are formalized as skills, including: - General-purpose methods such as time-series and clustering analysis. - Domain-specific workflows such as monthly reporting or policy monitoring. - Skills document required assumptions, comparison axes, cautions, and interpretation methods. - Over time, isolated domain knowledge becomes reusable organizational analysis capability. ## Business Impact - In early deployment, the platform expanded data use beyond data scientists to product owners and frontline employees. - More than half of the participating business unit’s members use it. - Analysis turnaround fell from an average of approximately two weeks to about 10 minutes. - The platform now supports hundreds of analyses per month and serves as a daily starting point for business questions. PJ One Piece’s main recommendation is to design AI analysis as an end-to-end operating platform—not merely an automated SQL or chatbot tool. Combining structured domain knowledge, safe data access, contextual multi-agent workflows, reusable skills, and continuous evaluation can make analysis faster while steadily improving its quality and organizational reach.

Read original(opens in new tab)

Large Language ModelsAI AgentsGenerative AIPython+4

github·Jun 15, 2026·2 min readCurated summary

GitHub Copilot CLI for Beginners: Overview of common slash commands

GitHub Copilot CLI’s slash commands provide a central way to control sessions directly from the terminal. They help users select models, manage context, inspect changes, switch projects, resume work, and control permissions. The article recommends typing `/` to explore the available commands and build a more deliberate CLI workflow. ## Slash Commands as Copilot CLI Controls - Slash commands are built-in terminal controls for: - Guiding Copilot’s behavior - Inspecting project changes - Managing conversation context - Moving between sessions and repositories - Resetting tool permissions - Typing `/` displays a scrollable list of supported commands. ## Selecting a Model - `/model` opens the list of available models. - Model information includes: - Capabilities for lightweight tasks or deeper reasoning - Availability based on the user’s plan or organization - Cost multipliers indicating relative usage cost - Choosing an appropriate model can balance speed, quality, and cost. ## Managing Context and Tokens - `/context` shows remaining tokens, system usage, and available buffer. - `/compact` summarizes the current conversation to free context space without starting over. - Copilot may compact automatically near the context limit, but users can trigger it manually when changing tasks. - `/clear` completely resets the current session. ## Resuming Previous Sessions - `/resume` lists earlier local and remote sessions. - Users can select a session to review its history and continue where they left off. ## Reviewing Changes - `/diff` displays recent modifications made during the session. - This helps users inspect and validate Copilot’s changes before proceeding. ## Switching Projects - `/cwd` changes Copilot’s current working directory. - It allows users to move between repositories or project directories without exiting the CLI. ## Resetting Permissions - `/reset-allowed-tools` removes previously granted permissions for actions such as editing files. - This is useful when switching to a repository that requires more cautious access controls. Users should start by typing `/` in Copilot CLI to discover available commands, then use commands such as `/model`, `/context`, `/compact`, `/diff`, and `/cwd` to maintain control over their coding sessions.

Read original(opens in new tab)

Large Language ModelsGithub Copilot CliSlash CommandsCommand Line Interface+2

cloudflare·Jun 15, 2026·2 min readCurated summary

Growing the Cloudflare AI team with talent from Ensemble AI

Cloudflare is bringing key members of Ensemble AI onto its team to improve AI infrastructure and inference efficiency. Ensemble’s work on model compression, structured neural architectures, and parameter-efficient fine-tuning complements Cloudflare’s Workers AI platform. The combined effort aims to make powerful AI models faster, cheaper, and easier to deploy globally. ## Incorporating Ensemble AI’s Expertise - Ensemble AI has focused on reducing the memory, compute, and deployment costs of large language and multimodal models. - Its NdLinear technology replaces standard transformer linear layers while preserving multidimensional structure such as attention heads, channels, and spatial dimensions. - NdLinear-LoRA reduces the number of trainable parameters needed to fine-tune large models. - These techniques complement quantization and vector quantization to improve model efficiency without significantly sacrificing quality. ## Improving AI Inference Economics - Cloudflare Workers AI provides serverless GPU-powered inference across Cloudflare’s global network. - Lower model size, memory usage, and compute requirements can improve throughput, GPU utilization, and overall inference costs. - These improvements are increasingly important for agents, multimodal applications, personalization, fine-tuning, retrieval, and reinforcement learning. - The Ensemble team will contribute to Cloudflare’s existing work, including the Infire inference engine, Unweight tensor compression, and systems for running very large language models. ## Supporting Next-Generation Workloads - Developers increasingly need AI infrastructure that is reliable, affordable, globally distributed, and close to end users—not merely access to models. - Cloudflare’s network, serverless platform, and Workers AI provide a foundation for deploying AI with less operational complexity. - Combining Cloudflare’s infrastructure with Ensemble’s efficient model architectures should enable lower-cost, higher-performance AI deployments at scale. Cloudflare’s stated goal is to make advanced AI workloads more accessible by improving the economics and efficiency of inference across its platform.

Read original(opens in new tab)

Large Language ModelsMachine LearningQuantizationCloudflare Workers Ai+4

github·Jun 12, 2026·3 min readCurated summary

How we made GitHub Copilot CLI more selective about delegation

GitHub improved Copilot CLI by making subagent delegation more selective rather than treating delegation as inherently beneficial. The new orchestration policy keeps narrow tasks with the main agent, delegates broad or independent work, and encourages parallel execution instead of waiting. After full production rollout, it reduced tool failures by 23% and improved high-percentile wait times without reducing quality. ## The Cost of Unnecessary Delegation - Subagents help with complex investigations, large repositories, and parallel work, but every handoff adds tool calls, coordination, and latency. - Copilot sometimes delegated simple, well-scoped tasks that the main agent could complete directly. - Common problems included: - Repeated or overlapping repository searches. - Subagents re-discovering context already available to the main agent. - Sequential delegation that left the main agent idle. - Stale paths, incorrect relative paths, and workspace mismatches. - The result was slower execution and more tool failures for tasks that should have required only a few steps. ## How the Problem Was Identified - GitHub used LLMs to analyze complete agent trajectories rather than manually reviewing sessions. - The analysis found that delegation was frequently used for narrow, obvious, or fully described tasks. - This led to a clear target: - Keep focused discovery-and-edit work with the main agent. - Reserve subagents for broad exploration, cross-cutting tasks, or genuinely independent work. ## A More Selective Orchestration Policy - Copilot now starts with the narrowest effective workflow: - Find and read the relevant file. - Make the targeted change. - Verify the result. - Delegation becomes appropriate when additional context, uncertainty, or parallel execution creates real value. - Subagents are treated as a parallelism mechanism, not a reason for the main agent to pause. - Handoffs should clearly specify: - The user’s request. - What the main agent already knows. - Which work the subagent owns. - What result the subagent should return. ## Evaluation and Production Results - GitHub tested the change with generated regression cases and existing benchmarks before rollout. - Staff and public A/B tests measured reliability, responsiveness, subagent workload, and quality. - Production results showed: - 23% fewer tool failures per session. - 27% fewer search-tool failures. - 18% fewer edit-tool failures. - 5% lower P95 wait time. - 3% lower P75 wait time. - No quality regression. - The improvements came mainly from avoiding unnecessary subagent paths and reducing orchestration overhead, not from making individual model calls faster. Copilot CLI users can access the improvement by running `/update` and upgrading to version 1.0.42 or later. The broader recommendation is to delegate selectively: use the main agent for focused tasks and subagents only when independent context or parallel work provides meaningful leverage.

Read original(opens in new tab)

Large Language ModelsGithub Copilot CliOrchestrationAb Testing+2

dropbox·Jun 12, 2026·3 min readCurated summary

How Dropbox uses MCP and Dash to close the design-to-code security gap

Dropbox found a significant gap between security design reviews and implementation. Only 12% of implementing pull requests linked back to their original threat models, and the median delay between review and code submission was about five weeks. To close this gap, Dropbox built a system using Dash, Model Context Protocol (MCP), and foundational models to automatically retrieve relevant security requirements and compare them with code during review. ## The Design-to-Code Gap - Threat models document risks, attack scenarios, and agreed-upon mitigations during security review. - These documents often remain in wikis or documentation systems, while implementation happens later through pull requests. - At Dropbox: - Only 12% of implementing PRs linked to their original design review. - Among 79 verified pairs, 54% of PRs were opened more than a month after the review. - The median delay was approximately five weeks, with some delays exceeding 11 months. - Only 29% of PRs were opened within two weeks of the security review. - About 15% of design reviews were filed retroactively, suggesting that some security-sensitive work was not identified early enough. ## Why Existing Tools Fall Short - Static analysis can detect whether certain security patterns or controls exist. - It generally cannot determine whether those controls satisfy the specific requirements agreed upon during design review. - Manual linking between PRs and threat models depends on engineers remembering additional workflow steps. - Reminder bots may improve compliance temporarily, but adherence tends to decline. - The core problem is not a lack of security documentation; it is the difficulty of making that existing context available during implementation. ## Dash and MCP as a Context Bridge - Dash already indexes Dropbox content and connected applications, including threat models and engineering documentation. - Dropbox used Dash’s MCP server to let an AI security agent search and read this content. - MCP provides a common interface for bringing multiple context sources into one agent session. - When a PR is opened, the agent retrieves relevant threat models and supporting documents. - A foundational model then compares the documented requirements with the proposed code. - For example, it can identify whether code implementing an endpoint enforces authentication required by the corresponding threat model. - Unlike traditional static analysis, this approach evaluates code against documented security intent, not just known code patterns. ## Integrating Security into Code Review - Dropbox surfaced the system directly within the existing code-review workflow rather than creating a separate security process. - This allows developers and reviewers to receive relevant security context where implementation decisions are already evaluated. - The same mechanism can potentially identify missing reviews when code appears security-sensitive but has no corresponding threat model. Dropbox’s approach shows how retrieval and AI reasoning can reconnect design decisions with implementation. Organizations can apply the same pattern beyond security—for example, to compliance requirements, privacy reviews, accessibility standards, or other design-to-code checks.

Read original(opens in new tab)

Large Language ModelsModel Context ProtocolCode ReviewStatic Analysis+2

github·Jun 11, 2026·1 min readCurated summary

Making secret scanning more trustworthy: Reducing false positives at scale

Mariko is a Principal Applied Scientist at Microsoft who leads the development of agentic AI workflows for cybersecurity operations. Her work centers on using large language models and agentic systems to bring frontier AI research into practical products and operational environments. ## Professional Role - Principal Applied Scientist at Microsoft. - Leads agentic AI workflow development for cybersecurity operations. ## Research and Technical Focus - LLM-powered systems. - Agentic workflows. - Applying frontier AI research to real-world products and operations. Overall, Mariko’s work connects advanced AI research with practical cybersecurity and enterprise applications.

Read original(opens in new tab)

Large Language ModelsAI AgentsCybersecurityAi Workflows+1

slack·Jun 11, 2026·3 min readCurated summary

Agentic Testing: Where Agents Fit in the E2E Testing Stack

Agentic E2E testing validates whether users can achieve goals rather than enforcing one fixed sequence of UI actions. Slack’s experiment with more than 200 runs found that agents can reliably explore workflows, especially through Playwright MCP, but they are slower and more expensive than deterministic tests. The conclusion is that agents should complement—not replace—traditional E2E tests. ## Goal-Based Testing vs. Fixed Journeys - Traditional tests follow predefined steps: click, type, navigate, and assert. - Agent-driven tests receive a goal and adapt their actions to reach it. - Agents may use different paths to achieve the same result, such as: - Selecting a search suggestion or pressing Enter - Reusing existing navigation state or reopening a view - Adding or skipping intermediate actions - This flexibility improves exploration but introduces tradeoffs in reliability, runtime, and cost. ## Experiment Design Slack evaluated three execution models across more than 200 runs: - **Agent + Playwright MCP** - Uses predefined browser actions and persistent DOM snapshots and logs. - **Agent + Playwright CLI** - Runs Playwright commands through the shell and reassesses the UI after each step. - **Generated Playwright tests** - Produces deterministic test code from natural language, then iteratively refines it. The experiments used Claude Sonnet 4.5 for MCP and CLI workflows and Claude Opus 4.6 for generated tests. All tests ran in non-production Slack workspaces using test data. Two workflows were tested 20 times per configuration: - **Thread Reply:** A simple 15–20-step flow involving channel creation, messaging, thread replies, and verification. - **Search Discovery:** A 25–30-step flow involving search, result navigation, channels, threads, and state verification. Inputs were provided either as detailed natural-language instructions or structured YAML describing actions and expected outcomes. ## Results: Reliability, Cost, and Runtime | Approach | Thread Reply failures | Search Discovery failures | Average runtime | |---|---:|---:|---:| | Agent with Playwright MCP | 0% | Approximately 12% | 5–8 minutes | | Agent with Playwright CLI | Approximately 12% | Approximately 20% | 9–11 minutes | | Generated Playwright tests | Approximately 8% | Approximately 48% | About 3 minutes | - Playwright MCP was the most reliable agentic approach, particularly for simple workflows. - Playwright CLI failed more often due to authentication, navigation timing, and session instability. - Generated tests were fast and reasonably successful on simple flows but degraded sharply as workflows became more complex. - Generated tests often completed 70–80% of a complex workflow before failing on a final interaction or assertion. ## Why Complexity Exposes Differences - MCP maintains a live, stable view of the application through persistent context. - CLI-based agents reconstruct state from updated snapshots, allowing small timing or interpretation inconsistencies to accumulate. - Generated tests can suffer from: - Variable UI state - Imprecise element targeting - Mismatches between generated code and existing page-object abstractions - The results suggest agent-native execution models handle increasingly complex exploratory flows better than generated deterministic tests, despite taking longer. Agentic testing is best used as an exploratory layer for validating user goals and discovering unexpected paths. Deterministic Playwright tests remain preferable for fast, repeatable regression checks, while Playwright MCP appears to be the strongest option when flexible, goal-oriented E2E coverage is needed.

Read original(opens in new tab)

Large Language ModelsAI AgentsModel Context ProtocolClaude+3

toss·Jun 9, 2026·3 min readCurated summary

How Do TAMs Solve Problems in Fast-Moving Organizations?

TAM CONNECT 2025 brought together Technical Account Managers from Toss and Kakao Pay to compare how they connect technology, business, customers, and internal teams. Although their organizations differ, they face similar challenges: operational complexity, cross-team coordination, recurring incidents, and the need to improve customer experience. The event framed TAMs not as basic support staff, but as technology-driven problem solvers whose role is expanding through automation and AI. ## The Scope of a TAM’s Role - TAMs resolve partner integration issues and advise on API adoption. - They coordinate multiple teams during incidents and improve operational processes. - Their work includes automating recurring problems and influencing product and platform design. - Depending on the situation, a TAM may act like a developer, product manager, or incident leader. - At Toss, their responsibilities span authentication, Face Connect, financial platforms, online and offline payments, and partner APIs. ## Reducing Alert Noise and Preventing Recurring Problems - Toss’s Dayoung Park presented a problem-solving framework that began by redefining which alerts truly require attention. - Excessive notifications can obscure serious incidents, so the team focused on identifying meaningful operational signals. - They structured incident patterns, detected recurring issues automatically, and analyzed the root causes of settlement discrepancies. - The goal was not merely to resolve incidents faster, but to build systems that prevent them from recurring. ## Making Operations Independent of Individual Owners - PayToss’s Gimun Lee discussed reducing dependency on specific people’s knowledge. - Response histories and operational information were shared transparently so anyone could handle an issue. - Their Discord developer community used n8n workflows, LLM-based log analysis, and automatically generated incident-cause and resolution suggestions. - These tools helped the team maintain an average response time of under ten minutes. ## Using Customer Experience to Improve TAM Work - Toss’s Seongmin Chun drew on previous experience working for a customer organization. - Understanding customer frustrations and the information needed during incidents influenced his communication and support practices. - The team used the PDCA cycle to continually improve integration guides, standardize repeated communications, and structure operational processes. - Effective TAM work means not only solving current problems but also ensuring the same problems do not happen again. ## Shared Challenges Across Companies - TAMs must balance customer expectations with internal development teams. - They operate in rapidly changing services with increasingly complex systems. - The role requires simultaneous understanding of technology, business priorities, and customer needs. - TAMs often appear to sit ambiguously between support, operations, development, and business, but their actual work involves structuring complex problems and mobilizing multiple teams. - The event characterized TAMs as technology-based problem solvers rather than simple support personnel. ## AI and the Future of TAM Work - AI is already being applied to: - Log analysis - Incident-cause recommendations - Operations guide generation - Automated responses to recurring inquiries - Anomaly detection - Document search and summarization - As AI handles repetitive responses, TAMs are likely to focus more on complex problem-solving, structural improvements, cross-team coordination, customer-experience design, and operational strategy. TAM CONNECT demonstrated that TAM organizations can learn significantly from one another. As technology and services become more complex, TAMs will likely become increasingly strategic, combining automation and AI with the judgment needed to improve systems, processes, and customer experiences.

Read original(opens in new tab)

Large Language ModelsAPIIncident ManagementFintech+4