techlist.io
KREN

Large Language Models

178 posts

cloudflare3 min readCurated summary

Building the foundation for running extra-large language models

Cloudflare is building infrastructure for serving extra-large open-source language models efficiently, especially for agentic applications with long prompts and frequent tool calls. Its approach combines specialized hardware configurations, prefill/decode disaggregation, prompt caching, distributed KV-cache management, and speculative decoding. These optimizations substantially improve latency, throughput, and cost efficiency without requiring more GPUs. ## Hardware Configurations for Agent Workloads - Different applications stress models differently: - Content generation sends fewer input tokens but produces many output tokens. - Summarization sends very large inputs and generates relatively short outputs. - Agent workloads typically involve: - Large system prompts - Tool and MCP definitions - Accumulated conversation history - Generated code and previous interactions - Workers AI therefore prioritizes fast input processing and tool-calling performance. ## Prefill-Decode Disaggregation - LLM inference has two stages: - **Prefill:** Processes input tokens and populates the KV cache; generally compute-bound. - **Decode:** Generates output tokens; generally memory-bound. - Running both stages on one server can underutilize GPUs because they stress different resources. - Cloudflare separates them across dedicated inference servers: - A prefill server processes the request and stores its KV cache. - A decode server retrieves the cache and generates the response. - This enables independent tuning, scaling for input- or output-heavy traffic, and use of heterogeneous hardware. - The architecture requires a sophisticated load balancer that: - Transfers KV-cache metadata between stages. - Rewrites streaming SSE responses. - Handles different inference-server protocols. - Balances traffic based on estimated in-flight prefill and decode tokens. - After adopting this design, Cloudflare saw: - Lower p90 time to first token and reduced tail-latency variance. - Intertoken latency fall from roughly 100 ms to 20–30 ms. - About a threefold improvement while using the same number of GPUs. ## Prompt Caching and Session Affinity - Long agent conversations repeatedly reuse the same context, making prompt caching essential. - The `x-session-affinity` header routes requests toward regions containing previously computed input tensors. - Cloudflare added support for this header to agent harnesses such as OpenCode. - Cached prompts improve: - Overall throughput - Interactive response times - Pricing, with discounted cached tokens - GPU efficiency - Adoption by heavy internal users increased peak input-token cache hit rates from 60% to 80%. ## Distributed KV-Cache Optimization - Larger models span multiple GPUs, requiring KV caches to be shared across devices and nodes. - For Kimi, Cloudflare uses Moonshot AI’s: - **Mooncake Transfer Engine** for high-speed memory transfers using RDMA technologies such as NVLink and NVMe over Fabric. - **Mooncake Store** to extend cache storage beyond GPU VRAM onto NVMe. - Combined with LMCache or SGLang HiCache, the system can: - Reuse cached prompts from any node in a cluster. - Reduce reliance on session-aware routing. - Balance traffic more evenly. - Keep sessions cached longer. - Increase cache hit rates and supported throughput. ## Speculative Decoding - The post begins introducing speculative decoding as another optimization. - It describes the basic LLM process of predicting successive tokens, but the provided text ends before explaining the technique or its results. Cloudflare’s overall strategy is to match infrastructure to real usage patterns rather than rely on a single hardware configuration. Separating inference stages, maximizing cache reuse, and distributing KV caches are practical ways to make large-model hosting faster and more economical.

Read original(opens in new tab)
Large Language ModelsCloudflareGpu OptimizationPrompt Caching+4
slack4 min readCurated summary

Managing context in long-run agentic applications

Long-running multi-agent applications cannot rely on unlimited conversation history: model APIs are stateless, and growing context windows eventually reduce quality or hit hard limits. Slack’s security-investigation system addresses this by giving agents complementary, purpose-specific context rather than exposing every agent to the full investigation history. Its three main channels—the Director’s Journal, Critic’s Review, and Critic’s Timeline—preserve coherence while leaving room for independent reasoning. ## The Challenge of Long-Run Coherence - Agent frameworks usually maintain continuity by resending the complete message history with every inference request. - Long investigations can involve hundreds of requests and megabytes of generated output. - Context windows impose both: - A hard limit on how much history can be supplied. - A quality limit, because performance may degrade before the window is completely full. - Multi-agent systems need carefully scoped views: - Too little shared context makes agents disconnected from the investigation. - Too much shared context can suppress creativity and encourage confirmation bias. ## Three Complementary Context Channels Slack uses separate information sources for different purposes: - **Director’s Journal** - Structured working memory for the orchestrating Director. - Records decisions, observations, findings, questions, actions, and hypotheses. - **Critic’s Review** - An annotated report evaluating Expert findings. - Includes credibility scores to distinguish reliable evidence from weaker claims. - **Critic’s Timeline** - A consolidated chronological view of findings. - Also attaches credibility scores, helping agents understand the sequence and evidential strength of events. Together, these channels provide continuity without forcing every agent to process the entire raw conversation. ## The Director’s Journal The Director coordinates the investigation by choosing questions, assigning specialist Experts, assessing progress, and deciding when to stop. The Journal gives it persistent working memory across phases and rounds. - The Director is encouraged to update the Journal frequently with short notes. - Entries can represent: - **Decisions** about investigative strategy - **Observations** about emerging patterns - **Findings** representing confirmed facts - **Questions** that remain unresolved - **Actions** taken or planned - **Hypotheses** about what may be happening - Entries can also include: - Priority levels - Follow-up actions - References to supporting evidence - Investigation phase, round number, and timestamp - The journaling tool itself simply accumulates entries; the agents’ prompts explain how to interpret them. ## Maintaining Alignment Across Agents - The Journal creates a shared narrative around the Director’s evolving plan. - It helps the Director: - Track progress - Identify dead ends - Revise investigative direction - Preserve decisions between rounds - Guide other agents toward a conclusion - Every agent receives the current Journal chronologically, along with instructions describing: - The Director’s role - Each agent’s relationship to the Director - The Journal’s purpose - How its entries should influence their work - This approach keeps specialists anchored to the overall investigation without requiring them to read every prior interaction. ## Example Investigation Context The sample Journal comes from an investigation into an apparent kernel-module-loading alert that turned out to be a false positive. - The Director recorded that: - The event originated from a package-installation hook rather than a direct `modprobe` command. - The host appeared to be a personal development workstation. - Root access was expected in that environment. - The detection rule matched “kmod” in a script path rather than confirming module loading. - The Director identified relevant Expert domains, including: - Endpoint telemetry - Identity and access - Configuration management - User behavior - The Journal captured both the preliminary conclusion and remaining verification tasks, such as checking the parent process chain. The design therefore preserves the reasoning trail while keeping it structured and compact. A practical design for long-running agentic systems is to replace indiscriminate transcript accumulation with multiple, curated context channels. Persistent journals can maintain leadership and continuity, while independent reviews and timelines provide evidence-focused context without overwhelming agents or biasing their reasoning.

Read original(opens in new tab)
Large Language ModelsAI AgentsMulti Agent SystemsContext Management+2
cloudflare3 min readCurated summary

Welcome to Agents Week

Cloudflare argues that AI agents require a fundamental shift in Internet and cloud infrastructure. Unlike traditional one-to-many applications, agents create unique, ephemeral execution environments for individual users and tasks, making current container-based economics and scaling inadequate. The company positions lightweight V8 isolates, alongside containers and browser support, as the foundation for making agents practical at global scale. ## The Internet Was Built for Applications, Not Agents - Cloud infrastructure evolved during the smartphone era to serve many users through a finite number of application instances. - Microservices, containers, Kubernetes, load balancing, and replication all support this one-to-many model. - Agents differ because an LLM dynamically determines code paths, tool usage, and task duration. ## One User, One Agent, One Task - Each agent may need its own execution environment, filesystem, tools, and state. - Coding agents currently use containers with access to Git, Bash, filesystems, and arbitrary binaries. - As agents spread to assistants, analysts, customer service, and planning tasks, the number of simultaneous environments could grow dramatically. ## The Scale Challenge - If 100 million US knowledge workers used agents at 15% concurrency, infrastructure would need about 24 million simultaneous sessions. - At 25–50 users per CPU, that implies roughly 500,000 to 1 million server CPUs in the US alone. - Multiple agents per person and global adoption would increase demand by orders of magnitude. ## Isolates as Agent Infrastructure - Cloudflare’s Workers platform uses V8 isolates instead of containers. - Isolates start in milliseconds, use only a few megabytes of memory, and provide secure sandboxing. - They can be up to 100 times faster to start and up to 100 times more memory-efficient than containers. - Dynamic Workers can create execution environments on demand, run code, and discard them at a scale of millions per second. - This efficiency could make one-agent-per-user economics viable beyond expensive coding assistants. ## The “Horseless Carriage” Phase - Early agent infrastructure often adapts existing systems instead of using designs built specifically for agents. - Agents use headless browsers to navigate human-oriented websites, though structured protocols such as MCP could provide direct service access. - Many MCP servers simply wrap REST APIs, despite LLMs often being better at writing and executing code than making long sequences of tool calls. - CAPTCHAs and behavioral fingerprinting ask whether a requester is human, while agent systems need identity, authorization, and permission controls. - Full containers are frequently used for tasks that require only a few API calls and a response. ## Supporting Both Old and New Models - Infrastructure transitions rarely happen all at once; technologies such as IPv4/IPv6, HTTP/2/HTTP/3, and TLS 1.2/1.3 coexist. - Cloudflare plans to support existing agent workloads while developing more efficient primitives. - Containers remain important for coding agents that need filesystems, Git, Bash, and arbitrary binaries. - Cloudflare is also expanding container-based sandbox environments and browser-rendering capabilities for services that do not yet support agent-native protocols. Cloudflare’s broader recommendation is to build infrastructure that can serve today’s container-based agents while moving toward lightweight, ephemeral isolates designed for billions of specialized agent sessions.

Read original(opens in new tab)
Large Language ModelsDatabase DesignKubernetesCloudflare Workers+4
google3 min readCurated summary

ConvApparel: Measuring and bridging the realism gap in user simulators

ConvApparel addresses the “realism gap” between LLM-based user simulators and genuine human behavior. It combines over 4,000 human-AI shopping conversations with a controlled Good-versus-Bad agent setup and evaluates simulators through statistical alignment, human-likeness, and counterfactual adaptation. The framework aims to determine whether simulators genuinely model human reactions or merely reproduce patterns from their training data. ## Why User Simulator Realism Matters - Conversational agents often fail during long, multi-turn interactions by forgetting constraints or producing irrelevant responses. - Human testing provides valuable feedback but is expensive, slow, and difficult to scale. - LLM-based user simulators offer a scalable alternative, but often behave unlike real users: - They may be excessively verbose. - They can lack consistent personas or coherent preferences. - They may possess unrealistic, encyclopedic knowledge. - They are often unusually patient and assistant-like. - Training systems only against unrealistic simulators may cause them to fail with real users. ## The Need for Counterfactual Validation - A simulator should respond plausibly not only to situations represented in its training data, but also to novel assistant behaviors. - The authors introduce **counterfactual validation**: training a simulator on helpful-agent conversations, then testing it against an unexpectedly frustrating agent. - A realistic simulator should recognize poor assistance and show increased frustration, reduced satisfaction, and behavior changes similar to those of real users. - This tests whether the simulator has learned general human behavior rather than memorized training patterns. ## The ConvApparel Dataset - ConvApparel contains more than 4,000 human-AI multi-turn conversations and nearly 15,000 total turns in the apparel-shopping domain. - Participants were unknowingly assigned to one of two recommendation agents: - **Good agent:** Helpful, efficient, and supported by robust search. - **Bad agent:** Intentionally confusing, tangential, and based on degraded search retrieval. - The dataset captures reactions ranging from satisfaction to significant annoyance. - Participants provided turn-by-turn retrospective annotations, including: - Satisfaction - Frustration - Likelihood of making a purchase ## Three-Part Evaluation Framework ### Population-Level Statistical Alignment - Simulated conversations are compared with human conversations using aggregate measures such as: - Conversation length - Words per turn - Dialogue acts, including rejecting recommendations - This reveals whether simulators reproduce broad behavioral distributions. ### Human-Likeness Score - An automated discriminator is trained on human and simulated conversations. - It produces a probability indicating how human-like a conversation appears. - The score is intended to detect subtle stylistic differences that simple statistics may miss. ### Counterfactual Validation - A simulator is trained only on conversations with the Good agent. - It then interacts with the unseen Bad agent. - High-fidelity simulation should produce a human-like increase in frustration and decline in satisfaction when the assistant behaves poorly. ## Simulator Configurations The experiments compare three Gemini-based user simulators: - **Prompted simulator:** Uses high-level behavioral instructions without additional task-specific training. - **In-context learning (ICL) simulator:** Retrieves semantically similar human conversations from ConvApparel and supplies them as examples at each turn. - **Supervised fine-tuning (SFT) simulator:** Trains a Gemini 2.5 Flash model directly on the dataset. The post presents ConvApparel as a structured way to measure simulator realism and test whether simulated users can adapt to assistant behavior outside their training distribution. Its central recommendation is to evaluate user simulators not only by surface-level similarity, but also by how naturally they react to unexpected failures.

Read original(opens in new tab)
Large Language ModelsDatabase DesignGeminiConversational Ai+4
google3 min readCurated summary

Improving the academic workflow: Introducing two AI agents for better figures and peer review

AI is being positioned as an active participant in academic research, not merely a tool for drafting text. The post introduces PaperVizAgent, which creates publication-ready figures, and ScholarPeer, which produces literature-grounded peer reviews. Both use multi-agent workflows and iterative verification to reduce researchers’ administrative burden while improving visual quality and review rigor. ## PaperVizAgent: Generating Publication-Ready Figures - PaperVizAgent converts manuscript text and a detailed figure caption into academic illustrations. - It uses five specialized agents: - **Retriever:** Finds relevant literature and reference figures. - **Planner:** Organizes the technical content. - **Stylist:** Develops appropriate visual and aesthetic guidelines. - **Visualizer:** Produces images or executable Python code for statistical plots. - **Critic:** Checks the result against the source text and requests revisions. - The critic-driven refinement loop is designed to ensure that figures are both technically faithful and visually clear. - Inputs typically include: - The manuscript’s method or technical sections. - A communicative-intent description explaining what the figure should convey. ### Evaluation Results - PaperVizAgent was compared with direct prompting, few-shot prompting, GPT-Image-1.5, Nano-Banana-Pro, and Paper2Any. - Figures were scored from 0 to 100 on: - Faithfulness - Conciseness - Readability - Aesthetics - It achieved an overall score of **60.2**, exceeding the human baseline of **50.0** and outperforming the evaluated automated systems. - Its strongest results were in conciseness and aesthetics, while its statistical plots reached human-competitive quality. ## ScholarPeer: Automating Rigorous Peer Review - ScholarPeer is a search-enabled, context-aware multi-agent system designed to emulate the workflow of a senior academic reviewer. - Rather than treating review as simple text generation, it combines literature retrieval, adversarial checking, and technical verification. - Its main components include: - A **sub-domain historian** that builds a current domain narrative from literature. - A **baseline scout** that searches for overlooked datasets, methods, and comparisons. - A **multi-aspect Q&A engine** that tests novelty and technical claims. - A **review generator** that follows conference-specific review guidelines. - The resulting review includes a summary, strengths, weaknesses, and questions for the authors. ### Evaluation Results - ScholarPeer was evaluated on public datasets against fine-tuned models and other agentic reviewing systems. - Its active web-search and verification process produced highly critical reviews grounded in existing research. - Side-by-side evaluations showed strong win rates against competing automated reviewers. - The system also narrowed the gap between AI-generated reviews and human reviews in terms of realism, diversity, and alignment with expert judgments. ## Implications for Academic Research - The two agents address separate bottlenecks in the publication process: - PaperVizAgent improves technical communication through better figures. - ScholarPeer helps scale peer review amid growing submission volumes and reviewer fatigue. - Their multi-agent designs suggest that specialized agents, coordinated through retrieval and iterative critique, may be more effective than a single general-purpose language model. - The systems are intended to support researchers rather than replace scientific judgment. Researchers could use PaperVizAgent for early figure prototyping and ScholarPeer for preliminary, literature-informed critique, while retaining human oversight for final scientific and editorial decisions.

Read original(opens in new tab)
Large Language ModelsAIMulti Agent SystemsData Visualization+4
google3 min readCurated summary

Evaluating alignment of behavioral dispositions in LLMs

The post introduces a framework for evaluating whether LLM behavior aligns with human behavioral tendencies in realistic social and workplace situations. Instead of relying on self-report questionnaires, it converts validated psychological traits into situational judgment tests and compares model responses with judgments from human annotators. Across 25 models, larger systems align better when humans strongly agree, but models remain overconfident and often fail to represent legitimate human disagreement. ## From Psychological Self-Reports to Situational Tests - The researchers adapt statements from established instruments measuring traits such as empathy, emotion regulation, and assertiveness. - Because LLM self-reports can vary with prompt wording and may not predict real behavior, the statements are transformed into realistic user-assistant scenarios. - Each scenario presents two possible actions: - One expressing or supporting a behavioral trait. - One opposing or suppressing it. - Three annotators review each generated test to ensure the scenario and actions accurately represent the intended trait. - Models respond naturally, and an LLM judge maps each response to one of the two actions. - Human preferences are collected from 10 annotators per scenario, drawn from a pool of 550 participants. ## Measuring Directional Alignment - Directional alignment measures whether a model gives greater probability to the action favored by the human majority. - The analysis focuses on scenarios with strong human consensus: - Unanimous agreement: 10 of 10 annotators. - Very high agreement: 9 or 10. - High agreement: 8 or 9. - Smaller models, particularly those under 25 billion parameters, often perform near chance and struggle to distinguish when a trait should be expressed or restrained. - Larger models over 120 billion parameters and frontier closed-weight models perform substantially better. - These models approach near-perfect alignment when human agreement is unanimous, but performance generally plateaus in the low-to-mid 80% range when consensus is weaker. - Qualitative deviations included: - Encouraging emotional openness in professional situations where humans preferred composure. - Favoring harmony in disputes instead of standing up for one’s position. - Recommending immediate action in time-sensitive situations without sufficient logistical verification. ## Representing Human Disagreement - The study also evaluates distributional alignment: whether model confidence reflects the diversity of human opinions. - When human annotators disagree, a well-aligned model should distribute its probability more evenly between the available actions. - The results show systematic model overconfidence across all 25 evaluated systems. - Models tend to favor one action too strongly even when human preferences are divided, indicating that they often fail to preserve pluralism in human judgment. ## Broader Implications - The framework distinguishes two types of alignment gaps: - Directional gaps, where models choose differently from a clear human majority. - Distributional gaps, where models fail to reflect uncertainty or disagreement among people. - The findings suggest that scale improves behavioral alignment but does not fully solve nuanced social judgment. - Evaluating behavior in realistic scenarios may reveal limitations that conventional personality questionnaires or direct model self-reports miss. Future alignment work should assess not only whether models choose the human-majority response, but also whether their confidence and range of responses appropriately reflect genuine variation in human perspectives.

Read original(opens in new tab)
Large Language ModelsBehavioral AnalysisSituational Judgment TestsPsychological Questionnaires+4
cloudflare3 min readCurated summary

Cloudflare Client-Side Security: smarter detection, now open to everyone

Cloudflare is making its Client-Side Security Advanced product self-serve and offering domain-based threat intelligence free to users of its basic bundle. The service detects malicious browser-side JavaScript through browser reporting, AST-based behavioral analysis, and a new LLM review layer. Its goal is to catch sophisticated skimming attacks while reducing false positives and avoiding performance impacts on customer applications. ## Growing Threat of Client-Side Attacks - Browser skimmers can steal credentials, payment data, and personal information without disrupting page loads or checkout flows. - Recent examples include: - A browser keylogger placed on a major U.S. bank’s employee merchandise store. - Malicious npm package releases capable of enabling browser-based crypto theft when bundled into front-end applications. - These attacks often exploit trusted first-party or third-party scripts rather than obvious server vulnerabilities. ## Broader Access to Client-Side Security - Client-Side Security Advanced, formerly the Page Shield add-on, is now available to self-serve customers. - Domain-based threat intelligence is complimentary for customers using the free Client-Side Security bundle. - Advanced capabilities include: - Machine-learning and LLM-assisted malicious script detection. - Continuous code-change monitoring for compliance requirements such as PCI DSS v4.0 requirement 11.6.1. - Proactive positive security rules maintained through ongoing monitoring. ## Browser-Based Monitoring Without Application Changes - Cloudflare evaluates approximately 3.5 billion scripts per day, with enterprise zones averaging about 2,200 scripts. - The system gathers signals through browser reporting mechanisms such as Content Security Policy. - Customers do not need scanners or application instrumentation. - Traffic must be proxied through Cloudflare. - The approach adds no latency to web applications. ## Detecting Script Intent - Enterprise sites may contain thousands of scripts, and roughly one-third change within a 30-day period. - Manually approving every DOM interaction or outbound connection would create excessive operational overhead. - Cloudflare instead analyzes what scripts are attempting to do. - JavaScript is represented as an Abstract Syntax Tree (AST), allowing the system to identify behavioral patterns even when code is minified, renamed, or obfuscated. ## Reducing False Positives - Client-side compromises are relatively rare but potentially severe, unlike the high-volume attacks typically handled by a WAF. - Because genuine incidents are uncommon, even accurate detection systems can produce more false alarms than real alerts. - False positives contribute to security-team fatigue and can obscure actual compromises. - Legitimate but heavily obfuscated code—such as bot challenges, tracking pixels, advertising bundles, and minified frameworks—can resemble malicious code structurally. ## GNN and LLM Detection Pipeline - Cloudflare’s primary detector is a Graph Neural Network (GNN) operating on JavaScript ASTs. - The GNN learns structural representations of code and can recognize similar behavior despite syntactic changes. - It is optimized for high recall to detect novel and zero-day threats. - Although fewer than 0.3% of analyzed traffic is incorrectly flagged, Cloudflare’s scale makes that percentage a significant number of alerts. - An LLM provides semantic context, recognizing common JavaScript frameworks, domain-specific coding patterns, and benign forms of suspicious-looking obfuscation. - The LLM complements rather than replaces the GNN: - Scripts classified as benign stop after the fast GNN evaluation. - Scripts exceeding the GNN’s risk threshold are sent to an open-source LLM hosted on Cloudflare Workers AI for a second opinion. Cloudflare’s approach combines low-overhead browser telemetry, structural machine learning, and semantic LLM review. For organizations handling payments or sensitive user data, enabling these controls can improve visibility into third-party scripts, detect unexpected code changes, and reduce the chance that false alarms overwhelm security teams.

Read original(opens in new tab)
Large Language ModelsMachine LearningJavaScriptAbstract Syntax Tree+4
grammarly3 min readCurated summary

What Is AI Chat? Definition, How It Works, and Key Benefits

AI chat enables open-ended, context-aware conversations with systems that generate responses dynamically rather than following fixed scripts. Powered by large language models (LLMs), it supports tasks such as writing, brainstorming, learning, summarizing, planning, and coding. Its flexibility comes with limitations: responses reflect learned patterns rather than true understanding, so users should provide clear context and verify results. ## What AI Chat Is - AI chat allows users to ask questions naturally and refine requests through follow-up messages. - It can answer questions, explain complex subjects, draft and revise text, summarize documents, generate code, and provide feedback. - Unlike fixed chatbot flows, it handles unstructured requests and evolving conversations without requiring users to restart. ## How AI Chat Works - **LLM training:** Models learn language patterns from massive text datasets rather than memorizing a fixed set of answers. - **Natural language processing:** The system analyzes prompts to infer meaning, intent, tone, and context beyond exact keyword matches. - **Response generation:** The model predicts and selects text one word at a time based on the prompt and patterns learned during training. - **Conversation context:** Recent messages help the system interpret follow-up requests, such as understanding that “make it shorter” refers to a previously generated summary. - **Ongoing refinement:** Fine-tuning and human feedback improve safety, accuracy, and alignment. Models generally do not learn from individual conversations in real time. ## AI Chat Compared with Traditional Chatbots - Traditional chatbots commonly use rules, decision trees, and scripted responses. - They work well for narrow, repeatable tasks such as FAQs, appointment booking, and order tracking. - AI chat is better suited to open-ended activities including brainstorming, drafting, explanations, and problem-solving. - “Conversational AI chatbot” usually describes a chatbot interface powered by generative AI, making it more flexible than a fully rules-based system. ## Common Uses - **Writing and editing:** Draft emails, rewrite passages, adjust tone, improve clarity, and revise reports or presentations. - **Brainstorming:** Generate ideas, outlines, alternatives, and new perspectives through iterative discussion. - **Learning and planning:** Explore unfamiliar topics, simplify complex information, and develop plans. - **Coding support:** Generate code, explain technical concepts, and help troubleshoot problems. ## Effective Use - Write clear prompts and provide relevant context. - State the goal, desired format, audience, and preferences. - Use follow-up questions to refine the response. - Review outputs for factual errors, bias, and inappropriate assumptions. AI chat is most useful as a flexible assistant rather than an unquestionable authority. Use it for exploration and productivity, but verify important information and apply human judgment before relying on its output.

Read original(opens in new tab)
Large Language ModelsMachine LearningAutomationNatural Language Processing+4
cloudflare3 min readCurated summary

Sandboxing AI agents, 100x faster

Cloudflare argues that AI-generated code needs secure execution, but traditional containers are too slow, memory-intensive, and difficult to scale for consumer-scale agents. Its Dynamic Worker Loader uses lightweight V8 isolates to create disposable, isolated sandboxes in milliseconds, with controlled access to APIs and no internet connectivity. The result is a sandbox roughly 100 times faster and substantially more memory-efficient than containers, provided agents can write JavaScript. ## Why Containers Fall Short - AI-generated code cannot safely run directly through `eval()`, since prompts could cause the model to introduce vulnerabilities. - Containers provide isolation but typically: - Take hundreds of milliseconds to start - Consume hundreds of megabytes of memory - Require warm instances to reduce latency - May encourage unsafe container reuse - These limitations make containers poorly suited to running a fresh sandbox for every request or user agent. ## Dynamic Worker Loader - Cloudflare’s Dynamic Worker Loader lets a Worker instantiate another Worker dynamically from runtime-provided code. - The host can: - Supply generated JavaScript modules - Expose selected APIs through RPC stubs - Disable or intercept outbound internet access - Invoke methods exported by the dynamically loaded Worker - The feature is in open beta for paid Workers users. ## Faster, Smaller Isolates - Dynamic Workers use V8 isolates, the same sandboxing technology underlying Cloudflare Workers. - Isolates: - Start in a few milliseconds - Use only a few megabytes of memory - Are approximately 100 times faster and 10–100 times more memory-efficient than typical containers - A new isolate can be created for one request and discarded afterward without maintaining a pool of warm sandboxes. ## Scalability and Latency - Dynamic Worker Loader has no container-style global concurrency or sandbox-creation limits. - It relies on the infrastructure that already scales Cloudflare Workers to millions of requests per second. - Each request could theoretically load its own isolated sandbox, even at very high concurrency. - Dynamic Workers commonly run on the same machine or thread as their parent Worker, avoiding network round trips and warm-sandbox lookup delays. - They are available across Cloudflare’s global network. ## JavaScript as the Agent Runtime - The main limitation is that agent-generated code should generally be JavaScript. - Workers also support Python and WebAssembly, but JavaScript is faster to load for short-lived snippets. - Cloudflare argues this is acceptable because: - LLMs can generate major programming languages - JavaScript has extensive training data - JavaScript was designed for web-based sandboxed execution ## TypeScript APIs for Agent Tools - Agents still need access to external capabilities such as chat systems and APIs. - TypeScript interfaces provide a concise way to describe these programming APIs. - Compared with MCP’s flat tool schemas or verbose OpenAPI specifications, TypeScript can express: - Methods and parameters - Return types and promises - Objects such as messages - Subscription and disposal behavior - This gives agents precise API knowledge with fewer tokens and lets them write direct code rather than issuing numerous tool calls. Dynamic Worker Loader is presented as a practical foundation for secure, disposable AI-agent execution: use V8 isolates for low-latency sandboxing, expose only narrowly defined TypeScript/RPC capabilities, and block network access unless explicitly required.

Read original(opens in new tab)
Large Language ModelsTypeScriptJavaScriptCloudflare Workers+4
toss3 min readCurated summary

Automating Service Vulnerability Analysis using LLM #2

The post explains how Toss Security Research improved AI-driven vulnerability analysis in a research network. Its main challenges were efficiently providing large codebases to an AI and making analysis results consistent and complete. The solution combined a custom code-browsing MCP server with SAST tools used not to identify vulnerabilities directly, but to enumerate all input-to-function paths that the AI must review. ## Efficiently Providing Large Codebases - Tools such as Cursor and Claude Code can search large projects, but primarily rely on pattern matching with tools like ripgrep. - Without prebuilt indexes, they may miss relevant code or waste tokens exploring unnecessary files. - The team built an MCP server that: - Uses **ctags** to index symbol definitions. - Uses **tree-sitter** to parse function boundaries. - Allows AI to access code remotely, similar to IDE features such as “Go to Definition” and “Find References.” ### SourceCode Browse MCP The MCP server provides four main tools: - **`find_references()`** - Searches for symbols or patterns using ripgrep. - Returns file paths, line numbers, snippets, total matches, and whether results were truncated. - **`read_definition()`** - Looks up definitions through the ctags index. - Returns metadata such as file, line, symbol type, language, signature, and scope. - Uses tree-sitter to include the complete function body when requested. - **`read_source()`** - Reads a configurable number of lines before and after a target line. - Lets the AI retrieve only the relevant local context instead of entire files. - **`get_project_structure()`** - Returns the indexed project’s directory structure. - Provides the AI with a project “blueprint,” which is especially important in remote environments where it cannot inspect the repository locally. The MCP workflow is to locate relevant symbols with `find_references()` and `read_definition()`, inspect nearby code with `read_source()`, and use `get_project_structure()` to understand the overall project. ## Improving Consistency and Accuracy - AI analysis produced inconsistent results: for example, it might find all 10 XSS vulnerabilities in one run but only 8 in another. - This variability made the results difficult to trust. - The team combined AI analysis with SAST tooling to ensure complete coverage. ## Using SAST to Enumerate Review Candidates - Rather than passing SAST-detected vulnerabilities directly to the AI, the team used SAST as a candidate-generation tool. - This avoids limiting the AI to vulnerabilities that the SAST engine itself knows how to detect. - SAST extracts every location where untrusted input enters the application and tracks its possible flow to function calls. - Custom Semgrep taint rules identify sources such as: - Spring `@RequestParam` - `@PathVariable` - `@RequestHeader` - Fields read from `@RequestBody` DTOs - `@RequestPart` - `@ModelAttribute` - `@RequestAttribute` - Potential sinks include generic function calls and object method calls. - The AI then reviews every extracted source-to-sink path, combining the completeness of static analysis with the broader reasoning ability of an LLM. The overall approach is to use deterministic indexing and SAST for coverage, while relying on AI for deeper vulnerability interpretation.

Read original(opens in new tab)
Large Language ModelsDatabase DesignKotlinAi Security+4
cloudflare3 min readCurated summary

Powering the agents: Workers AI now runs large models, starting with Kimi K2.5

Cloudflare is expanding Workers AI beyond smaller models by adding Moonshot AI’s Kimi K2.5, a frontier open-source model designed for agentic workloads. With a 256k context window, tool calling, vision, and structured outputs, Kimi can power an agent’s full lifecycle directly on Cloudflare’s platform. Cloudflare argues that its price-performance makes open-source models essential as personal and enterprise agents dramatically increase inference demand. ## Kimi K2.5’s Price-Performance Advantage - Cloudflare uses Kimi internally for: - Agentic coding through OpenCode - Automated code review via the Bonk public code review agent - Security analysis of Cloudflare codebases - A security-review agent processes more than 7 billion tokens daily and has found over 15 confirmed issues in one codebase. - Compared with a mid-tier proprietary model, switching to Kimi reduced the estimated cost of this workload by 77%, from roughly $2.4 million annually. - As employees increasingly run multiple agents continuously, inference costs become a major barrier to scaling. - Cloudflare positions open-source, frontier-quality models as a more economical alternative to proprietary systems. ## Serving Large Models on Workers AI - Supporting Kimi required upgrades to Workers AI’s inference stack, which historically focused on smaller models. - Cloudflare uses its proprietary Infire inference engine and custom kernels to improve: - Model performance - GPU utilization - Throughput - The platform applies advanced serving strategies such as: - Data, tensor, and expert parallelization - Disaggregated prefill, separating input processing from generation across machines - Workers AI handles these infrastructure optimizations so developers do not need specialized machine learning, DevOps, or reliability engineering expertise. ## Prefix Caching for Agent Workloads - Agents frequently resend large prompts containing: - System instructions - Tool definitions - MCP server tools - Conversation history - Entire codebases - Prefix caching avoids reprocessing unchanged input tokens during multi-turn interactions. - This reduces prefill work, improving: - Time to First Token (TTFT) - Tokens Per Second (TPS) - Overall inference cost - Workers AI now exposes cached tokens as a usage metric and charges less for them than regular input tokens. - Cloudflare has also introduced techniques to improve cache hit rates. ## Session Affinity - Workers AI provides an `x-session-affinity` header to route requests from the same session or agent to the same model instance. - Keeping requests on the same instance increases prefix-cache reuse. - Higher cache hit rates lead to faster responses, greater throughput, and lower costs. - Clients should provide a unique session or agent identifier with the header. Cloudflare’s recommendation is to use Workers AI when building agents that need frontier-level reasoning without the cost and operational burden of proprietary models or self-hosted infrastructure.

Read original(opens in new tab)
Large Language ModelsDatabase DesignKubernetesCloudflare+4
toss3 min readCurated summary

Embracing the Software 3.0 Era

Software 3.0 replaces hand-written rules with natural-language instructions to LLMs, but models alone cannot reliably perform real-world work. The missing piece is the harness: tools, context, and environments that connect an LLM to codebases, commands, databases, and users. Claude Code illustrates how familiar Software 1.0 architecture can guide agent design while adding a new capability—asking humans for judgment when uncertainty arises. ## From Software 1.0 to Software 3.0 - **Software 1.0:** Developers explicitly write logic using languages such as Python, Java, or C++. - **Software 2.0:** Data and training produce neural-network weights that function as the program. - **Software 3.0:** Prompts and natural-language instructions direct LLM behavior. - Karpathy’s central claim is that Software 3.0 is increasingly absorbing both traditional code and trained models. ## Harnesses Make LLMs Useful - A raw LLM cannot independently read a codebase, execute commands, modify files, or access databases. - A **harness** supplies the tools and environment needed to turn model capability into practical work. - Claude Code is presented as a harness for Claude: it transforms a language model into an agent capable of completing and shipping tasks. ## Mapping Agent Concepts to Layered Architecture The terminology of agent systems can be understood through familiar Software 1.0 design patterns: - **Slash commands → Controllers** - They serve as entry points for user requests, such as `/review` or `/refactor`. - **Sub-agents → Service layer** - They coordinate multiple skills to complete a workflow. - Each sub-agent has an independent context and acts as a self-contained unit of work. - **Skills → Domain components** - Each skill should have one focused responsibility, such as reviewing code, generating tests, or writing documentation. - **MCP → Infrastructure or adapters** - MCP provides abstraction boundaries for external systems such as APIs and databases. - **CLAUDE.md → Project constitution** - It records stable project information: technology choices, conventions, and build commands. - Frequently changing task details should be provided through the conversation or injected into an agent’s context instead. ## Agent Design Has Familiar Anti-Patterns Traditional code smells also apply to agent systems: - **Feature Envy:** A skill relies excessively on another skill’s data. - **Duplication:** Prompts are copied across multiple skills. - **Long Method:** A single sub-agent performs an overly long sequence of many skills. - Clear boundaries, single responsibility, and limited coupling remain valuable. ## The Difference: Agents Can Ask Humans Layered architecture generally requires every failure and edge case to be handled through predefined exceptions, policies, or branches. - Traditional code must decide what to do when an unusual case occurs. - An agent using human-in-the-loop interaction can pause and ask the user for clarification. - In this model, exceptions become questions, allowing the agent to continue after receiving a decision. Agents should ask when: - An action is difficult to reverse, such as deletion or deployment. - Several valid options exist without a clear best choice. - The decision has significant consequences. They should proceed automatically when: - The operation is safely repeatable. - Existing conventions provide a clear answer. - The action is easy to undo. ## What Carries Forward into Software 3.0 The new paradigm does not make established engineering practices irrelevant. - Move away from explicitly coding every possible rule and edge case. - Do not reduce LLMs to simple autocomplete tools. - Preserve layered design, single responsibility, abstraction, dependency management, and interface design. - Continue emphasizing testability, debugging, code review, and iterative improvement. The practical approach is to combine Software 3.0’s flexible reasoning with Software 1.0’s architecture and engineering discipline, while giving agents a clear way to involve humans when decisions require judgment.

Read original(opens in new tab)
Large Language ModelsKubernetesReactTypeScript+4
google3 min readCurated summary

Testing LLMs on superconductivity research questions

LLMs may help physicists navigate complex research, but their reliability depends heavily on the quality and curation of their sources. In a high-temperature superconductivity study, systems grounded in expert-selected literature—especially NotebookLM and a custom retrieval-augmented generation system—outperformed general web-access models. The results suggest that trustworthy scientific AI requires balanced reasoning, strong evidence, and carefully controlled reference collections. ## Evaluating LLMs on Superconductivity - Researchers from Google Research and Cornell University tested whether LLMs could answer expert-level questions in condensed matter physics. - The study focused on cuprate high-temperature superconductors, whose underlying mechanism remains unresolved despite decades of research. - Understanding superconductivity in these materials could help scientists discover compounds that work at higher temperatures. - The field contains thousands of experimental and theoretical papers and competing explanations, making it difficult for researchers—especially newcomers—to establish a balanced view. ## Study Design and Sources - Six systems were evaluated: - GPT-4o - Perplexity - Claude 3.5 - Gemini Advanced Pro 1.5 - Google NotebookLM - A custom retrieval-augmented generation (RAG) system - Four models had broad web access, including 765 open-access experimental papers and 1,553 theoretical papers. - NotebookLM and the custom RAG system used a curated database: - Twelve superconductivity experts selected 15 review articles. - Those reviews contained approximately 3,300 references. - A final collection of 1,726 experimental papers and reviews was assembled. - Experts created 67 difficult questions, including questions about doping levels and evidence for quantum criticality in cuprates. ## Evaluation Criteria Experts used masked reviews and scored responses from 0 to 2 on: - Balance between competing scientific perspectives - Comprehensiveness and factual depth - Conciseness and clarity - Evidence and links to sources - Relevance of supplied images - Qualitative comments ## Results - NotebookLM achieved the strongest overall performance. - The custom RAG system ranked second overall, showing the value of retrieval from the same expert-curated sources. - NotebookLM, Gemini, and the custom RAG system performed best at presenting balanced and comprehensive answers. - NotebookLM provided the strongest evidence and citations but was less concise than the other systems. - Image quality was generally weaker; the custom RAG system performed best among the models that regularly supplied images. - All systems showed areas needing improvement, particularly when addressing nuanced, unresolved research questions. ## Practical Implication For scientific research, LLMs should be paired with expert-curated, quality-controlled literature rather than relying solely on unrestricted web searches. Such systems can serve as research tutors or thought partners, but their answers still require expert verification, especially in fields with competing theories and rapidly evolving evidence.

Read original(opens in new tab)
Large Language ModelsDatabase DesignGeminiNotebooklm+2
kakao4 min readCurated summary

Finding Real Threats Among Hundreds of Millions of Security Signals — Transforming the Security Monitoring Paradigm with AI

Kakao argues that monitoring hundreds of millions of daily security events cannot scale through human analysts and increasingly complex rules alone. Its solution is a hybrid AI pipeline that filters noise early, analyzes only high-value events with multiple models, and continuously improves through verified feedback. The goal is not to generate more alerts, but to understand context and identify threats worth investigating. ## The Scale Problem: Finding Threats in a Haystack - Endpoint activity such as process execution, network connections, file changes, and privilege escalation produces hundreds of millions of events. - The volume grows rapidly as services expand, while the proportion of genuine attacks remains very small. - Increasing the number of analysts alongside event volume is economically and operationally unsustainable. - AI is needed to correlate events, interpret behavior statistically and contextually, and dynamically distinguish normal activity from anomalies. ## Limitations of Rule-Based Monitoring - Rules can identify what happened, but not why, who initiated it, or whether it fits the environment. - Legitimate deployment commands can resemble backdoor installation, causing high false-positive rates. - Analysis quality varies by analyst experience, shift, and time of day. - Analysts must manually assemble host information, network sessions, process histories, and related logs into an incident narrative. - Expanding detection categories—behavior sequences, statistical anomalies, multi-source correlations, and rare events—makes manual rule maintenance impractical. - SIEM correlation improves on single-event rules but remains limited to predefined scenarios and struggles with unknown attack patterns. - As rule sets and event volumes grow, both maintenance costs and matching performance become problematic. ## A Funnel-Based Hybrid Architecture - Kakao filters events through multiple stages before using AI: - Rule-based filters remove obvious noise. - Learned normal patterns are automatically excluded. - AI performs detailed analysis only on the small remainder requiring judgment. - Rules handle clear, deterministic patterns quickly, while AI evaluates complex contextual situations. - The framework is designed to accommodate new threat types and detection categories without creating a separate system for each scenario. ## Multi-Model Verification and Operational Resilience - Multiple AI models independently analyze the same event and cross-check one another. - Disagreement is treated as an uncertainty signal that can trigger deeper analyst review. - Model diversity helps reduce bias, false positives, and missed detections. - It also provides resilience against model failures, API outages, and quality changes after model updates. - The design balances cost, processing speed, and accuracy rather than optimizing only for detection precision. ## Teaching AI the Environment’s Context - Generic LLMs initially misclassified legitimate activity because they lacked knowledge of Kakao’s infrastructure. - The system supplies structured context, including: - Host roles - Services running on each host - Accounts used for automation - Normal communication and operational patterns - This context allows the model to act more like an analyst familiar with the organization than a generic security classifier. ## Analyzing Complete Behavior Flows - Individual commands such as `curl`, `chmod`, and script execution can occur in both normal deployments and attacks. - Kakao therefore reconstructs activity at the host level, linking: - Process execution history - Network sessions - File changes - Temporal ordering - The same command can have different meanings depending on when, where, and in what sequence it occurred. - AI evaluates the complete sequence to distinguish routine operations from intrusion behavior. ## Translating Events into AI-Usable Data - Sending raw events directly to an LLM wastes tokens on irrelevant information and reduces accuracy. - Different detection tasks require different signals; statistical anomaly detection and sequence analysis cannot rely on one fixed format. - Kakao introduced: - A standardized event schema - Dynamic feature construction tailored to each detection type - This reduces token usage while improving the relevance and precision of AI analysis. ## WALT: A Self-Learning Detection Loop - Initially, analysts had to manually convert AI conclusions into new detection policies. - Kakao developed WALT, or **Whitelist-Assisted Learning and Tuning**, to automate this feedback process. - Repeatedly verified normal patterns are converted into exception policies. - Those policies filter future matching events before they reach the AI engine. - Thousands of detection policies are reportedly being generated and operated this way, allowing accuracy to improve over time. ## Cost and Performance Constraints - Sending every event to an AI model caused unsustainable costs and processing delays. - The funnel architecture addresses this by reserving expensive AI analysis for events that survive earlier filtering. - The overall system must continuously balance economic cost, response speed, detection accuracy, and reliability. Kakao’s practical recommendation is to treat AI as part of a carefully designed security pipeline—not as a replacement for rules or analysts. Effective large-scale monitoring combines deterministic filtering, contextual multi-model analysis, structured data, and a controlled feedback loop that learns from verified outcomes.

Read original(opens in new tab)
Large Language ModelsAIAnomaly DetectionSecurity Monitoring+4
kakao4 min readCurated summary

From Student to Developer: Learning Rational Choices Over Right Answers—From DB and Security to AI

The onboarding of 40 new Kakao developers shifted their perspective from making features work to designing systems that survive real-world operations. Across databases, security, and AI, they learned that there is rarely one perfect answer; the best choice depends on scale, risk, maintainability, and business needs. The central lesson was to replace theoretical correctness with responsible, adaptable engineering judgment. ## Database: From Finding the Right Answer to Preparing for Change - Database design must be evaluated by whether it can withstand traffic, schema changes, and operational demands—not only by theoretical correctness. - Foreign keys are not automatically the best choice: - They can introduce locking, performance, and flexibility concerns. - Referential integrity can instead be managed at the application layer, provided testing and correction processes are strong. - Soft deletion, using fields such as `deleted_at`, supports auditability and recovery and is often an essential operational strategy. - Indexes should be selected according to the questions the database must answer: - B-tree, GIN, GiST, SP-GiST, and vector indexes serve different data and query patterns. - Execution plans reveal whether SQL uses indexes or performs full table scans, directly affecting I/O and response times. - Duplication is not always harmful: - Intentional denormalization can avoid expensive joins. - Snapshot data can simplify reads and preserve the information needed by a business workflow. - In MongoDB, embedding selected related data can make screen queries much simpler than relying exclusively on references. - Different database systems embody different trade-offs among performance, consistency, scalability, and operational cost. - The training covered MySQL high availability, PostgreSQL primary-key structures, cloud-native systems such as Neon, and the broader storage-to-analysis pipeline of Hadoop and Spark. - The resulting mindset favors designs that are safe to change and affordable to operate over designs that are theoretically perfect. ## Security and IT: From Someone Else’s Responsibility to a Personal Default - Security became a direct consequence of developers’ code rather than merely a compliance or infrastructure concern. - Everyday safeguards such as development/production separation, VPNs, and antivirus software demonstrate that safety often requires accepting some inconvenience. - DDoS defense is not only about blocking traffic: - It can be difficult to distinguish an attack from legitimate traffic spikes caused by a popular event. - Developers should apply basic controls such as rate limiting and escalate suspicious activity through established response channels. - Hands-on API exploitation made vulnerabilities concrete and encouraged developers to view security through an attacker’s perspective. - Security must be continuous: - AI is increasingly being used both to discover vulnerabilities and to strengthen attacks. - Social-engineering methods involving QR codes, app permissions, and human behavior require more than purely technical defenses. - Security checks should be integrated from the beginning of development, not performed only at the end. - Software quality also depends on people: - Code should remain understandable enough for another developer to take over quickly. - Strong engineering means choosing and communicating the most appropriate solution for the business context, not merely finding a technically possible one. ## AI: From Chatting with Models to Designing Systems - An AI agent is not simply a model; it is an architecture composed of tools, routing logic, error handling, and model calls. - Agent development applies familiar software-engineering practices to probabilistic models. - Because LLM outputs can vary, reliable systems need deliberate controls: - Prompt chaining breaks large tasks into smaller steps and limits context contamination. - Few-shot examples clarify required output formats. - Routing selects different prompts or workflows based on conditions. - Multi-agent systems divide responsibilities among specialized agents, echoing the modularity and scalability principles of microservices. - RAG reduces hallucinations structurally by: - Chunking documents. - Searching for semantically similar vectors. - Supplying retrieved information to the model as additional context. - MCP exposes internal systems and data as callable tools, effectively enabling remote function calling and connecting AI to enterprise capabilities. - Effective AI use shifted from criticizing poor answers to specifying clear objectives, formats, examples, context, and supporting data. - The goal is not merely to receive an intelligent response, but to design a system that consistently produces intelligent behavior. The training ultimately marked a transition from student-style problem solving to professional engineering. Developers should consider operational resilience, security, maintainability, and business value, then make and clearly explain the most reasonable choice for the circumstances.

Read original(opens in new tab)
Large Language ModelsDatabase DesignAI AgentsPostgreSQL+4