Model Evaluation | Techlist.io

githubJun 25, 20263 min readCurated summary

Evaluating performance and efficiency of the GitHub Copilot agentic harness across models and tasks

GitHub argues that an agentic harness—the system coordinating tools, context, and workflow—strongly influences how effectively an AI model solves software tasks. Its shared Copilot harness aims to match model-vendor harnesses in task completion while using fewer tokens. Benchmark results across four models suggest near-parity overall, with performance varying by model and task. ## Benchmarking Approach - GitHub evaluates the harness using public and internal benchmarks, plus real-world metrics and online experiments. - Comparisons hold the following constant: - The same underlying model - The same task - Context-window size - Reasoning effort - Tool selection - MCP servers - Copilot CLI is compared with: - Claude Code for Claude Sonnet 4.6 and Claude Opus 4.7 - Codex CLI for GPT-5.4 and GPT-5.5 - Benchmarks include: - **SWE-bench Verified:** 500 human-validated Python bug fixes - **SWE-bench Pro:** More complex, multi-step engineering tasks - **SkillsBench:** Skill usage and triggering - **TerminalBench:** Command-line workflows - **Win-Hill:** Software tasks in Windows containers ## Token Efficiency - Copilot’s harness generally completes tasks at rates comparable to model-vendor harnesses while consuming fewer tokens. - It performed better across the reported tests for Sonnet 4.6 and Opus 4.7. - For GPT-5.4 and GPT-5.5, Copilot performed better on nearly every benchmark. - The main exception was SWE-bench Verified, where Copilot was 7% worse with GPT-5.4 and 4% worse with GPT-5.5. ## Task Resolution - GitHub emphasizes that lower token use matters only when the agent successfully completes the work. - Overall task-resolution rates were considered on par with vendor harnesses. - Results varied by benchmark: - Copilot generally did better on SWE-bench Pro, especially with GPT models. - It outperformed vendor harnesses on Win-Hill or matched them for every model. - It performed better on TerminalBench with Sonnet and Opus, matched GPT-5.5, and trailed GPT-5.4. - SkillsBench results favored GPT models on Copilot but favored vendor harnesses for Claude models. - Differences are described as statistically comparable because model behavior is stochastic and run-to-run variation can explain many gaps. ## TerminalBench Variance Analysis - GitHub uses TerminalBench 2.0 to study both cost and completion rate over repeated runs. - The preferred outcome is higher resolution with lower cost. - The analysis illustrates that benchmark results should account for variance rather than relying on a single run. - GitHub presents Copilot as equal to or better than the vendor harnesses on this cost-versus-success comparison. The practical conclusion is that harness design is a reusable performance multiplier across Copilot products. GitHub’s results support using its shared harness when developers need broad model choice, efficient token usage, and comparable task-completion performance, while recognizing that the best harness can still vary by model and workload.

Read original

Model Evaluation Large Language Models AI Agents Model Context Protocol+3

naverDec 2, 2025Original article

Naver TV (opens in new tab)

The development of NSona, an LLM-based multi-agent persona platform, addresses the persistent gap between user research and service implementation by transforming static data into real-time collaborative resources. By recreating user voices through a multi-party dialogue system, the project demonstrates how AI can serve as an active participant in the daily design and development process. Ultimately, the initiative highlights a fundamental shift in cross-functional collaboration, where traditional role boundaries dissolve in favor of a shared starting point centered on AI-driven user empathy. ## Bridging UX Research and Daily Collaboration * The project was born from the realization that traditional UX research often remains isolated from the actual development cycle, leading to a loss of insight during implementation. * NSona transforms static user research data into dynamic "persona bots" that can interact with project members in real-time. * The platform aims to turn the user voice into a "live" resource, allowing designers and developers to consult the persona during the decision-making process. ## Agent-Centric Engineering and Multi-Party UX * The system architecture is built on an agent-centric structure designed to handle the complexities of specific user behaviors and motivations. * It utilizes a Multi-Party dialogue framework, enabling a collaborative environment where multiple AI agents and human stakeholders can converse simultaneously. * Technical implementation focused on bridging the gap between qualitative UX requirements and LLM orchestration, ensuring the persona's responses remained grounded in actual research data. ## Service-Specific Evaluation and Quality Metrics * The team moved beyond generic LLM benchmarks to establish a "Service-specific" evaluation process tailored to the project's unique UX goals. * Model quality was measured by how vividly and accurately it recreated the intended persona, focusing on the degree of "immersion" it triggered in human users. * Insights from these evaluations helped refine the prompt design and agent logic to ensure the AI's output provided genuine value to the product development lifecycle. ## Redefining Cross-Functional Collaboration * The AI development process reshaped traditional Roles and Responsibilities (RNR); designers became prompt engineers, while researchers translated qualitative logic into agentic structures. * Front-end developers evolved their roles to act as critical reviewers of the AI, treating the model as a subject of critique rather than a static asset. * The workflow shifted from a linear "relay" model to a concentric one, where all team members influence the product's core from the same starting point. To successfully integrate AI into the product lifecycle, organizations should move beyond using LLMs as simple tools and instead view them as a medium for interdisciplinary collaboration. By building multi-agent systems that reflect real user data, teams can ensure that the "user's voice" is not just a research summary, but a tangible participant in the development process.

Model Evaluation AI Large Language Models User Experience+5

lineMay 21, 2025Original article

How to evaluate AI-generated images? (opens in new tab)

To optimize the Background Person Removal (BPR) feature in image editing services, the LY Corporation AMD team evaluated various generative AI inpainting models to determine which automated metrics best align with human judgment. While traditional research benchmarks often fail to reflect performance in high-resolution, real-world scenarios, this study identifies a framework for selecting models that produce the most natural results. The research highlights that as the complexity and size of the masked area increase, the gap between model performance becomes more pronounced, requiring more sophisticated evaluation strategies. ### Background Person Removal Workflow * **Instance Segmentation:** The process begins by identifying individual pixels to classify objects such as people, buildings, or trees within the input image. * **Salient Object Detection:** This step distinguishes the main subjects of the photo from background elements to ensure only unwanted figures are targeted for removal. * **Inpainting Execution:** Once the background figures are removed, inpainting technology is used to reconstruct the empty space so it blends seamlessly with the surrounding environment. ### Comparison of Inpainting Technologies * **Diffusion-based Models:** These models, such as FLUX.1-Fill-dev, restore damaged areas by gradually removing noise. While they excel at restoring complex details, they are generally slower than GANs and can occasionally generate artifacts. * **GAN-based Models:** Using a generator-discriminator architecture, models like LaMa and HINT offer faster generation speeds and competitive performance for lower-resolution or smaller inpainting tasks. * **Performance Discrepancy:** Experiments showed that while most models perform well on small areas, high-resolution images with large missing sections reveal significant quality differences that are not always captured in standard academic benchmarks. ### Evaluation Methodology and Metrics * **BPR Evaluation Dataset:** The team curated a specific dataset of 10 images with high quality-variance to test 11 different inpainting models released between 2022 and 2024. * **Single Image Quality Metrics:** Evaluated models using LAION Aesthetics score-v2, CLIP-IQA, and Q-Align to measure the aesthetic quality of individual generated frames. * **Preference and Reward Models:** Utilized PickScore, ImageReward, and HPS v2 to determine which generated images would be most preferred by human users. * **Objective:** The goal of these tests was to find an automated evaluation method that minimizes the need for expensive and time-consuming human reviews while maintaining high reliability. Selecting an inpainting model based solely on paper-presented metrics is insufficient for production-level services. For features like BPR, it is critical to implement an evaluation pipeline that combines both aesthetic scoring and human preference models to ensure consistent quality across diverse, high-resolution user photos.

Model Evaluation AI Generative AI Computer Vision+5

googleApr 29, 2025Original article

Benchmarking LLMs for global health (opens in new tab)

Google Research has introduced a benchmarking pipeline and a dataset of over 11,000 synthetic personas to evaluate how Large Language Models (LLMs) handle tropical and infectious diseases (TRINDs). While LLMs excel at standard medical exams like the USMLE, this study reveals significant performance gaps when models encounter the regional context shifts and localized health data common in low-resource settings. The research concludes that integrating specific environmental context and advanced reasoning techniques is essential for making LLMs reliable decision-support tools for global health. ## Development of the TRINDs Synthetic Dataset * Researchers created a dataset of 11,000+ personas covering 50 tropical and infectious diseases to address the lack of rigorous evaluation data for out-of-distribution medical tasks. * The process began with "seed" templates based on factual data from the WHO, CDC, and PAHO, which were then reviewed by clinicians for clinical relevance. * The dataset was expanded using LLM prompting to include diverse demographic, clinical, and consumer-focused augmentations. * To test linguistic distribution shifts, the seed set was manually translated into French to evaluate how language changes impact diagnostic accuracy. ## Identifying Critical Performance Drivers * Evaluations of Gemini 1.5 models showed that accuracy on TRINDs is lower than reported performance on standard U.S. medical benchmarks, indicating a struggle with "out-of-distribution" disease types. * Contextual information is the primary driver of accuracy; the highest performance was achieved only when specific symptoms were combined with location and risk factors. * The study found that symptoms alone are often insufficient for an accurate diagnosis, emphasizing that LLMs require localized environmental data to differentiate between similar tropical conditions. * Linguistic shifts pose a significant challenge, as model performance dropped by approximately 10% when processing the French version of the dataset compared to the English version. ## Optimization and Reasoning Strategies * Implementing Chain-of-Thought (CoT) prompting—where the model is directed to explain its reasoning step-by-step—led to a significant 10% increase in diagnostic accuracy. * Researchers utilized an LLM-based "autorater" to scale the evaluation process, scoring answers as correct if the predicted diagnosis was meaningfully similar to the ground truth. * In tests regarding social biases, the study found no statistically significant difference in performance across race or gender identifiers within this specific TRINDs context. * Performance remained stable even when clinical language was swapped for consumer-style descriptions, suggesting the models are robust to variations in how patients describe their symptoms. To improve the utility of LLMs for global health, developers should prioritize the inclusion of regional risk factors and location-specific data in prompts. Utilizing reasoning-heavy strategies like Chain-of-Thought and expanding multilingual training sets are critical steps for bridging the performance gap in underserved regions.

Model Evaluation AI Large Language Models Gemini+5

datadogSep 23, 20243 min readCurated summary

How we optimized LLM use for cost, quality, and safety to facilitate writing postmortems

Bits AI uses large language models to generate first drafts of incident postmortems from Datadog Incident Management metadata and Slack discussions. The goal is to reduce documentation effort without replacing the human-led reflection that makes postmortems valuable. Datadog found that production-quality LLM features require extensive prompt experimentation, safeguards against hallucinations, careful model selection, and strong privacy and feedback mechanisms. ## Why LLMs Are Difficult to Use for Postmortems - Postmortems must remain factual, consistent, and useful, unlike more creative LLM applications. - LLMs can ignore instructions, produce inconsistent formatting, repeat information, or hallucinate plausible but false details. - More than 100 hours were spent refining section structures and instructions so they worked across varied incidents. - Building these systems requires a combination of software engineering, product management, data science, and technical writing. ## Balancing Structured and Unstructured Incident Data - Structured Datadog data includes incident metadata, alerts, graphs, and manually entered fields such as customer impact. - Slack discussions provide current, detailed, and unstructured context from the incident response process. - Manually maintained fields are often more reliable but can become outdated, while LLMs can continually reassess live discussions. - Combining both sources helped the model understand the full incident and prioritize information appropriately. ## Cost, Speed, and Quality Trade-offs - Datadog compared models including GPT-3.5 and GPT-4. - Results varied by as much as 50 times in cost and speed. - GPT-4 generally produced more accurate output but was substantially slower and more expensive. - Model choice therefore depended on the required balance between quality, latency, and operating cost. ## Preserving Human Control and Privacy - AI-generated drafts were intended to support authors, not become final documents automatically. - Keeping engineers involved preserves the discovery and learning process that postmortems are meant to encourage. - Because incidents may contain sensitive information or secrets, Datadog scrubbed data before sending it to an LLM. - The system was designed to ensure authors remained responsible for reviewing and improving the draft. ## Experimentation and Reliability Safeguards - A custom API extracted and structured incident data from Datadog, Slack, and other sources. - The framework enabled rapid testing of different datasets, models, input configurations, and output token limits. - Datadog refined prompts and instructions to guide the model toward useful recommendations while retaining author agency. - Lowering model temperature and tuning other parameters reduced irrelevant or fabricated content. - A related feature generated concise incident summaries for responders joining Slack channels, providing a faster testing environment for LLM-based incident analysis. ## Evaluating Generated Drafts - Datadog used both qualitative and quantitative evaluation methods. - Authors of earlier postmortems compared their human-written documents with AI-generated drafts for the same incidents. - Evaluation included criteria such as factual accuracy and conciseness, with the provided article ending before the full evaluation approach was described. - Continuous feedback was used to refine prompts, models, and the overall drafting workflow. Overall, reliable AI-assisted postmortems require more than selecting a capable model. The strongest approach combines carefully prepared incident data, privacy controls, model and prompt experimentation, systematic evaluation, and mandatory human review.

Read original

Model Evaluation Large Language Models Datadog Slack+4