Techlist.io - Korean Tech Blog Curator

cloudflareAug 3, 20262 min readCurated summary

Cloudflare Workers and Containers now support inbound TCP connections and gRPC

Cloudflare is expanding Workers to support low-latency, TCP-based applications such as real-time voice AI and gRPC services. New inbound socket handling lets Workers route connections through Durable Objects and Containers, while Cloudflare also enables full-duplex gRPC servers running in containers. Together, these capabilities allow developers to deploy language-agnostic TCP and gRPC services closer to users across Cloudflare’s global network. ## Inbound TCP with `connect(socket)` - Workers can now accept inbound TCP sockets through a new `connect()` handler. - The socket exposes readable and writable streams, allowing Workers to send, receive, and proxy raw bytes. - Connections can be routed: - Between Workers - From Workers to Durable Objects - From Durable Objects to Cloudflare Containers - Developers can pipe data in both directions to preserve full-duplex communication. - Containers can run arbitrary TCP servers written in any language, such as Python services listening on port `8080`. - Cloudflare Spectrum provides the TCP ingress layer and routes incoming connections to the selected Worker. ## Full-Duplex gRPC in Containers - Developers can deploy gRPC servers written in languages such as Go inside Cloudflare Containers. - Bidirectional streaming allows clients and servers to exchange messages over one persistent connection. - This is particularly useful for: - Real-time voice AI - Low-latency inference - Mobile and distributed applications - Streaming RPC workflows - A sample Go server sends an initial connection message, echoes incoming messages, and sends a closing message when the client disconnects. - Cloudflare’s network of more than 330 locations can bring gRPC workloads closer to users, reducing latency. ## gRPC APIs from Workers - Workers can serve unary and server-streaming gRPC APIs. - Workers can also call external gRPC servers. - Developers write the application using gRPC-Web, while Cloudflare automatically converts incoming and outgoing requests to standard gRPC. - This provides a simpler integration path for applications that need gRPC without managing raw protocol translation themselves. ## Availability - The features are being introduced through a private beta. - Interested developers must sign up to gain access. Cloudflare recommends these capabilities for applications requiring persistent, low-latency, bidirectional communication. The combination of Spectrum, Workers, Durable Objects, and Containers provides a flexible path for running raw TCP protocols and gRPC services close to end users.

Cloudflare Workers Grpc Durable Objects Tcp+3

cloudflareAug 3, 20263 min readCurated summary

Smaller, faster, safer: running Kimi and GLM at scale

Cloudflare improves the efficiency of serving large, long-context models by optimizing both GPU memory usage and request concurrency. Its approach combines FP8 KV-cache quantization, INT4 weight compression, and integrity checks for shared caches, while using separate prefill and decode pools to apply each optimization where it helps most. These techniques increase throughput and reduce costs without materially affecting model accuracy. ## Quantizing the KV Cache - Long-context models store attention keys and values in a KV cache, which often exhausts GPU memory before model weights do. - Cloudflare stores the cache in FP8 instead of BF16, cutting its size in half. - On Kimi K2.6, this increases available context from roughly 686,000 to 1.37 million tokens. - FP8 is slightly slower at the same concurrency because values must be converted during attention. - However, FP8 supports more concurrent requests: - BF16 runs out of memory at 32 requests. - FP8 reaches 2,192 tokens per second at 64 requests. - Peak throughput improves by about 41%, with roughly 30% lower cost per token. - Cloudflare keeps BF16 for prefill, where workloads are compute-bound. - Evaluation results show FP8 and BF16 produce effectively indistinguishable quality across reasoning, knowledge, tool-calling, and internal benchmarks. ## Compressing Model Weights - GLM 5.2 weights are compressed from FP8 to INT4 for the decode phase. - The checkpoint shrinks from 705 GB to 421 GB, while per-GPU memory in an eight-way deployment falls from about 88 GB to 52 GB. - The freed memory supports approximately 1.18 million tokens of KV cache. - INT4 improves decode performance because generation is memory-bandwidth-bound: - Single-request throughput rises from 60 to 92 tokens per second, a 55% gain. - Gains range from 16% to 27% at higher concurrency. - Prefill becomes slower with INT4 because compressed weights must be expanded before computation: - FP8 prefill: about 10,160 tokens per second. - INT4 prefill: about 8,660 tokens per second. - Cloudflare therefore uses FP8 for prefill and INT4 for decode. - Accuracy remains within 0.8 percentage points of the FP8 model across tested benchmarks. ## Protecting a Shared KV Cache - Greater memory efficiency allows hundreds of requests to share physical KV-cache pages, increasing the risk of page-allocation or bookkeeping errors. - Cloudflare assigns each cache page a changing tag whenever it is reallocated. - Requests record the pages and tags they expect, and the server validates these mappings before supported decode operations. - If a mismatch occurs, the request is aborted instead of reading incorrect data. - In production-style tests, integrity checking caused: - Less than 1% throughput reduction. - Less than 1% increase in p95 latency. - Validation runs as a separate batch check rather than inside the attention kernel, avoiding GPU synchronization races. - The feature is enabled per deployment, while deployments that do not use it incur no measurable overhead. ## Future Work - Cloudflare is expanding FP8 KV caches across its fleet. - It is testing NVFP4 weight compression on NVIDIA Blackwell GPUs. - The company is also working toward making cache integrity checks universally enabled at negligible cost. Together, these optimizations let Cloudflare serve larger models with more concurrent users, lower inference costs, and essentially unchanged model quality. Separating prefill and decode workloads is central to applying each precision choice where it delivers the best trade-off.

Gpu Inference Mixture Of Experts Kv Cache Continuous Batching+4

cloudflareAug 3, 20263 min readCurated summary

Introducing the Billable Usage API: programmatic cost visibility for Cloudflare

Cloudflare is launching a Billable Usage API to make account spending accessible to automation, not just human-facing dashboards. The single endpoint reports usage and costs across products such as Workers, R2, D1, Workers AI, Vectorize, Images, and Stream. Its FOCUS-inspired schema is designed to integrate with FinOps tools, although full FOCUS conformance is still planned. ## Billable Usage API - Endpoint: - `GET /accounts/{account_id}/billable-usage` - Optional date filters: - `from=YYYY-MM-DD` - `to=YYYY-MM-DD` - Requires a Cloudflare API token with appropriate billing access. - Returns HTTP 200 with a standard Cloudflare JSON envelope: - `result` - `success` - `errors` - `messages` - Usage and cost data currently refresh daily, with more real-time reporting planned. ## Usage and Cost Data Each result row represents one product and charge period. - `ServiceName` and `ServiceFamilyName` identify the product and product family. - `ChargePeriodStart` and `ChargePeriodEnd` define the covered period. - `PricingQuantity` and `ConsumedUnit` show billable usage, such as requests, GB-months, or GB-seconds. - `ContractedCost` reports the charge in `BillingCurrency`. - `CumulatedPricingQuantity` and `CumulatedContractedCost` provide running billing-period totals. - `ZoneId` and `ZoneName` identify the relevant zone when usage is zone-specific. - Results can be filtered by date and include billing-period metadata such as `BillingPeriodStart`. ## FOCUS-Compatible Schema Cloudflare adopted naming that closely follows the FinOps Open Cost and Usage Specification (FOCUS). - Exact or near matches include: - `BillingCurrency` - `ChargePeriodStart` and `ChargePeriodEnd` - `ServiceName` - `ConsumedQuantity` and `ConsumedUnit` - `PricingQuantity` - `ContractedCost` - `ServiceFamilyName` is similar to FOCUS’s `ServiceCategory`, but uses Cloudflare’s native grouping. - `ZoneId` and `ZoneName` serve a role similar to `ResourceId` and `ResourceName`. - `CumulatedContractedCost` is a convenience field; FOCUS generally treats cumulative calculations as query-time operations. - The API is not yet fully FOCUS-compliant because some required columns are missing. ## Vantage Integration Cloudflare partnered with Vantage to bring Cloudflare costs into multi-provider FinOps workflows. - Vantage connects with a read-only Cloudflare API token granting Billing Read access. - It retrieves Billable Usage data daily and breaks spending down by: - Product - Zone - Account - Cloudflare costs appear alongside AWS, Azure, AI, and SaaS provider costs in: - Cost Reports - Budgets - Cost Alerts - Supported workflows include cross-provider allocation, anomaly detection, and FinOps-agent or MCP queries. ## Motivation for Programmatic Cost Visibility As agents increasingly deploy Workers, provision R2 buckets, and manage D1 databases, they can also create unexpected spending. Cloudflare argues that programmatic infrastructure management requires programmatic cost visibility, enabling automated monitoring, allocation, alerts, and optimization. The API is intended as a foundation for agent-driven and FinOps tooling, while Cloudflare works toward more real-time data and complete FOCUS conformance.

Model Context Protocol Cloudflare Cloudflare Workers Finops+3

tossAug 3, 20264 min readCurated summary

How DS and MLE Work Together

The post explains how Toss Bank improved collaboration between Data Scientists (DS) and ML Engineers (MLE) by progressively formalizing their responsibilities. What began as manually transferring notebooks evolved into standardized Python files and finally into installable model packages built around explicit interfaces. The result was faster deployments, consistent observability, and clearer ownership, while AI-generated code introduced a new need to standardize coding style as well. ## Problems with Notebook-Based Handoffs Initially, DS built models and inference code in Jupyter notebooks, then handed them to MLE. - MLE had to recreate the serving code from scratch. - Dependencies, configuration files, and source code were often missing or difficult to reproduce. - Preprocessing logic could be interpreted differently by DS and MLE. - “It works in the notebook” did not guarantee that it would work in production. - As the number of models increased, communication and rework grew rapidly. This approach separated people, not code, so the division of responsibility remained unclear. ## Phase 1: Separating Logic into `.py` Files The team next moved the collaboration boundary from people to files. - DS kept notebooks for experimentation and training. - Core inference logic was extracted into `.py` files. - MLE reviewed these files and validated them through CI. - DS’s intended model behavior was preserved more reliably. - Communication costs decreased. However, the files lacked a standardized structure. - Models used inconsistent function names such as `predict()`, `run()`, and `inference()`. - Code still required modifications when moved into the serving environment. - Global configuration changes in one model could affect other models sharing the same process. - Logging, metrics, and error handling could not be applied consistently across models. ## Phase 2: Defining an Interface Contract The team ultimately standardized the boundary through the `commons-ml-model` package. - A base abstraction defines a common model structure. - DS implements three methods: - `pre_process` - `inference` - `post_process` - The base class handles shared concerns such as: - Logging - Metrics - Tracing - Timing and request tracking - DS packages the implementation as a reusable library. - MLE installs the package with `pip install` and deploys it without rewriting the model. This turns the division of work into a code-level contract. DS focuses on model behavior, while MLE owns serving infrastructure and operational concerns. Updating the base abstraction can also add observability features to every model at once. ## Monorepo Collaboration The team manages the abstraction package and individual model packages in a single monorepo using `uv` workspaces. - Changes to the abstraction and affected models can be reviewed in one pull request. - DS and MLE review the same code in the same repository. - CI, release, and versioning policies are centralized. - Switching from Poetry to `uv` improved build speed by three to five times. The tradeoff is that changes to shared packages can affect many models, and the repository becomes heavier as more packages accumulate. ## Standardizing AI-Generated Code AI-assisted development created a separate collaboration problem: consistent structure did not guarantee consistent coding style. The team introduced `pfmls-stylepack` to encode team conventions for AI tools. - Naming conventions are standardized. - Exception-handling patterns are prescribed. - Rules determine when to use enums instead of hard-coded strings. - Hooks apply conventions while code is being generated. - AI-generated code can explain when a particular rule influenced its implementation. The team therefore distinguishes between: - **Structural consistency:** interfaces define what each role implements. - **Style consistency:** shared rules define how code should be written. Both are necessary for smooth reviews. ## Lessons from the Evolution - The hardest decision is choosing the right collaboration boundary: excessive structure limits flexibility, while insufficient structure recreates inconsistency. - Documentation and early DS–MLE pairing reduce the learning curve for the package-based workflow. - Shared libraries are a double-edged sword: one change can cause broad impact, but one fix can also benefit every model. - In the age of AI-generated code, teams must standardize not only responsibilities and interfaces but also implementation style. The practical recommendation is to make collaboration contracts executable: define stable interfaces, package model code for reuse, centralize shared serving behavior, and enforce coding conventions automatically.

Machine Learning CI/CD Python API Design+4

gitlabAug 3, 20263 min readCurated summary

Secure every commit to production with Claude and GitLab

Claude’s security tools can identify and fix vulnerabilities during coding, but they do not govern everything that happens afterward. The post argues that GitLab should secure the remaining path to production through enforced scans, approval policies, data controls, lifecycle-wide coverage, and audit evidence. Together, Claude handles in-session authoring while GitLab provides organization-wide governance. ## From In-Session Findings to Enforced Controls - GitLab applies security configuration profiles across projects and pipelines, preventing teams from bypassing required scans. - Merge request approval policies enforce separation of duties, ensuring an agent or developer cannot approve and merge its own changes. - Critical vulnerabilities can block merges until a designated approver reviews them. - Vulnerability reports and security dashboards permanently track whether findings were detected, dismissed, or resolved. ## Turning Security Activity into Audit Evidence - Compliance controls require scans to run on every merge request and expose findings for human review. - Pipeline logs, approval records, and audit events provide reproducible evidence of what was scanned and who approved each change. - Compliance frameworks such as SOC 2, PCI DSS, and FedRAMP can map evidence to named requirements. - Compliance status reports show which controls have passed, failed, or remain pending, including activity involving agents. ## Controlling Sensitive Data Sent to Models - Context exclusions prevent secrets, proprietary files, and regulated data from being sent to AI models. - Self-managed GitLab environments and self-hosted models can keep code and inference within an organization’s boundary. - Teams can select approved models, restrict model access, and prevent code from being used for training. - GitLab Duo prompt guardrails detect secrets and limit the content available to prompts, reducing prompt-injection risk. ## Scanning Beyond the Coding Session - Session-based reviews cannot detect vulnerabilities introduced after code is written, such as newly disclosed dependency flaws like Log4Shell. - GitLab provides dependency, container, infrastructure-as-code, secret, and DAST scanning across the delivery lifecycle. - Security Review Flow uses reasoning to identify business-logic flaws, authorization errors, and race conditions that deterministic scanners may miss. - Advanced SAST produces reproducible, CWE-mapped results, providing more consistent evidence than potentially variable LLM-based findings. ## Consistent Coverage for Humans and Agents - Claude’s plugin primarily reviews code written and committed within its session. - Developer shell commands, including Claude’s `!` shell escape, may fall outside that review scope. - Claude Security can review broader codebases when explicitly invoked by a developer or administrator. - GitLab pipeline scans and merge request policies apply to every change, regardless of whether it was written by a human or an agent. GitLab’s policies and lifecycle-wide scanning provide the durable controls needed after an AI coding session ends. Teams can use Claude for immediate security feedback while relying on GitLab to enforce consistent review, protect sensitive data, block unsafe releases, and maintain audit-ready evidence through production.

Gitlab Model Context Protocol Infrastructure As Code Sast+4

figmaAug 3, 20262 min readCurated summary

Code, Craft, and the Making of Nested Folders | Figma Blog

Figma’s nested folders feature required much more than a file-browser update: it involved rethinking content structure, permissions, sharing, administration, and core infrastructure. The team responded to changing priorities and faster AI-assisted development by replacing a sequential product process with code-led experimentation, flexible roles, and continuous collaboration. This approach helped them turn early ideas into working solutions and ship a complex, long-requested feature. ## Nested folders required foundational changes - Nested folders help growing teams organize files within shared projects. - Implementing them affected: - The file browser - Admin controls - Sharing and permissions - Figma’s underlying content model - Core infrastructure - The feature was therefore treated as a systems-level redesign rather than a simple quality-of-life improvement. ## Moving beyond the traditional product process - Initially, the team followed a conventional sequence: - Product defined requirements. - Design mapped the user experience. - Engineering started implementation after the concepts were considered complete. - Midway through the project, priorities shifted as resources were also needed for AI-native initiatives. - Tools such as Figma Make, the Figma MCP server, agent skills, codebase prototyping, and context-aware agents made it possible to begin work from prototypes, code, or even informal ideas shared in Slack. ## Leading with code - As code became faster and cheaper to produce, the team used pull requests as a way to explore product ideas. - Working prototypes and PRs helped the team test assumptions directly instead of spending too long debating theoretical solutions. - Code became a proposal and a communication tool, not merely the final implementation step. ## Flexible roles and shared ownership - Responsibilities shifted according to the problem being solved: - Engineers made design decisions. - The designer contributed code. - The product manager focused less on daily execution and more on strategic questions. - This flexibility allowed the team to respond quickly as technical and product requirements evolved. ## Replacing handoffs with conversations - Blurred roles turned team members into both students and teachers. - Instead of relying on formal handoffs between product, design, and engineering, the team collaborated continuously. - Sharing specialized knowledge and previously guarded parts of each craft helped build the trust needed to make decisions quickly. The broader lesson is that complex product work benefits from treating code, design, and product thinking as shared responsibilities. For teams building foundational features under changing conditions, early prototypes, cross-functional ownership, and ongoing conversation can be more effective than rigid sequential handoffs.

Figma AI Agents Mcp Server Code Prototyping+2

cloudflareAug 2, 20262 min readCurated summary

Welcome to Agents Week

The post introduces “Agents Week” by reframing the question from what an Agent Cloud should be to what agents themselves need. Unlike today’s human-oriented cloud and web, an Agent Cloud must provide agent-native primitives while also translating between existing human-designed systems and an emerging agent-shaped web. The week will explore how infrastructure, development, security, organizational access, and web interactions should evolve for agents. ## Why the Existing Cloud Falls Short - Today’s cloud and web assume a human is monitoring and interacting with them. - Interfaces are built around attention, visual dashboards, clicking, reading, and human decision-making. - Agents have different priorities: speed, structure, reliable access, and the ability to operate continuously without fatigue or distraction. ## Two Responsibilities of an Agent Cloud - Build an agent-native foundation with primitives designed for agents from the beginning rather than adapted from human tools. - Serve as a translation layer between: - The existing human-shaped web and infrastructure. - The agent-shaped systems being developed for the future. ## Topics for Agents Week The series will examine: - Storage, compute, and execution primitives required by agents. - An agentic development lifecycle, or “ADLC,” that reduces or removes humans from routine development loops. - Secure ways for employees and agents to access organizational systems of record and perform meaningful work. - The evolution of the agentic web, including discovery, access, and payments. - How humans and agents will work together during the transition to agent-oriented systems. ## Invitation to Ask Agents Directly The post encourages readers to ask their own agents what they need from an Agent Cloud instead of relying only on human assumptions. Suggested areas include cloud infrastructure, development processes, secure enterprise access, and web interactions, with readers invited to share their agents’ responses. In practical terms, designing an Agent Cloud requires starting with agents’ operational needs while supporting compatibility with the human-oriented systems that still dominate today.

AI Agents Cloud Computing Sdlc Agent Cloud+2

netflixJul 31, 20262 min readCurated summary

Modeling Device Capabilities for Analytics

Netflix models device capabilities to determine which features can be safely supported across its diverse hardware ecosystem. By tracking hardware, software, and platform limitations in scalable analytical datasets, Netflix can measure feature reach and identify adoption bottlenecks. This enables more precise feature management for capabilities such as 4K, spatial audio, cloud gaming, and new UI experiences. ## Building a Device Capability Model - Devices vary significantly in RAM, CPU cores, display resolution, audio support, and platform capabilities. - Netflix maintains detailed capability data for each device model, including: - Screen dimensions and resolution - Supported video profiles and codecs - Surround sound support - RAM capacity - Software version and platform information - Internal feature flags are integrated into the model to connect device capabilities with feature availability. ## Cumulative Tables for Current Device State - Netflix uses a cumulative table to track the latest known capabilities for each device. - Capabilities are stored in a structured format, such as supported screen sizes and video profiles. - This design supports large-scale analytics and reporting by providing an up-to-date view of device functionality. ## Histogram Tables for Feature Distribution - A histogram table measures active devices over the previous 28 days. - Results are broken down by device model and software version. - The table also counts how many devices support particular capabilities. - For example, Netflix can analyze external display support on streaming sticks: - 100% of devices may support the HD PlayReady profile. - Only 20% may support the UHD HEVC profile. ## Using Analytics for Feature Management - Netflix uses these datasets to evaluate feature penetration for products such as: - 4K Ultra HD - Netflix Spatial Audio - Cloud Gaming - Updated user interfaces - Capability data helps teams identify hardware or software bottlenecks. - Feature decisions can therefore be made at a more granular level, improving performance, reliability, and user experience. Netflix’s approach demonstrates that a detailed, analytics-focused capability model is essential for managing features across a global and highly varied device ecosystem.

Data Modeling Video Streaming Cloud Gaming Feature Flags+2

githubJul 31, 20263 min readCurated summary

Don’t stop early: Case-folding source code at memory speed

Case folding converts text into a canonical, case-insensitive form for comparisons, making it essential to GitHub’s large-scale code search. GitHub optimized this operation by removing an apparent optimization: instead of stopping at the first non-ASCII byte, it scans the entire buffer branchlessly, enabling SIMD vectorization. The resulting Rust `casefold` crate processes ASCII at over 45 GiB/s—close to memory-bandwidth limits. ## Case Folding Is Not Lowercasing - Lowercasing is intended for display and can depend on locale or context. - Greek sigma may become `ς` or `σ`. - Turkish `I` has locale-specific behavior. - Case folding is intended for comparison and must be locale-independent and symmetric. - Unicode provides explicit rules in `CaseFolding.txt`. - The crate supports simple one-to-one folds (statuses C and S), but not: - Full folds such as `ß → ss` - Turkic-specific folds such as dotted `İ` - This restriction matches tools such as ripgrep and helps maintain consistent matching behavior. ## Why Case-Folding Performance Matters - GitHub’s Blackbird search engine indexes more than: - 180 million repositories - 480 TB of source code - Source bytes are case-folded before n-gram extraction and indexing. - Folding is also needed when evaluating potential query matches. - Since most source code is ASCII, optimizing the ASCII path provides the largest benefit. ## Removing the Early Exit - A conventional implementation scans until it finds a non-ASCII byte, then switches to Unicode processing. - On an Apple M4, this branch-heavy approach reached only about 3.1 GiB/s. - The optimized loop: - ORs every byte into an accumulator to detect non-ASCII data once. - Uses `b.wrapping_sub(b'A') < 26` as a branchless uppercase test. - Sets bit 5 with `| (is_upper << 5)` to lowercase uppercase ASCII letters. - The loop always processes and writes the entire buffer, then checks whether Unicode processing is necessary. ## Vectorization Beats Early Termination - Removing the data-dependent `break` allows LLVM to vectorize the loop with 16-byte NEON instructions. - Performance progression on a 5.7 KB ASCII buffer: - Naive branchy loop: 3.1 GiB/s - Branchless body with early exit: 2.6 GiB/s - Early exit removed: 7.6 GiB/s - Fully branchless loop: over 45 GiB/s - The early exit prevents vectorization even when the loop body is otherwise branch-free. - Branchless arithmetic then eliminates compare-and-blend overhead and enables full memory-speed performance. ## Why Branchless Code Can Be Slower - In scalar code, the branchless version writes every byte, even when no change is needed. - The branchy version skips stores for the majority of lowercase letters, digits, spaces, and other unchanged bytes. - Its conditional branch is highly predictable, so it is inexpensive. - Branchless writes become beneficial only after vectorization, where the processor handles a whole vector at once. The practical lesson is to avoid data-dependent loop exits when they block vectorization. For predominantly ASCII workloads, a complete branchless scan can outperform “stop as soon as possible” logic by a wide margin, while an accumulated high-bit check efficiently identifies inputs requiring Unicode handling.

Rust Performance Optimization Unicode Llvm+3

cloudflareJul 31, 20263 min readCurated summary

An API for MoQ: provision your own isolated relays

Cloudflare has moved MoQ from an open testing network toward production use by adding isolated relays and authentication. Its provisioning API and dashboard let applications create globally available relay scopes and issue separate publisher and subscriber credentials, without deploying infrastructure. The beta supports MoQ Transport drafts 14 and 16 and is currently free. ## MoQ and Its Architecture - MoQ is an open IETF publish/subscribe protocol built on QUIC, the transport used by HTTP/3. - Publishers send named data streams, while subscribers request those streams through relays. - Relays copy data to subscribers without inspecting its contents, enabling efficient fan-out. - The same system can support live video, video calls, low-latency messaging, and other real-time workloads. - Using CDN-based relays avoids the cost and complexity of operating specialized media servers. ## From Open Preview to Production - Cloudflare’s initial MoQ preview exposed an unauthenticated relay on servers in more than 330 cities. - Over 1,000 clients continue to use the open endpoints daily for testing and development. - The lack of authentication made the preview unsuitable for applications requiring confidentiality or role-based permissions. - For example, live auction applications need broadcasters to publish while viewers can only subscribe. ## Isolated Cloudflare Relays - Provisioning a relay creates an isolated scope across Cloudflare’s existing global network rather than starting a VM, container, or dedicated process. - Each scope separates an application’s namespaces, tracks, and objects from those of other applications. - Clients connect through an Anycast endpoint, with Cloudflare handling global routing. - Relays become available within seconds, without regional capacity planning, load balancers, or server management. - Cloudflare compares the model to creating a virtual host rather than deploying a new web server. ## Provisioning API and Access Tokens - The control-plane API manages relay configuration and credentials but does not handle the media flowing through relays. - A relay defines the isolated application scope. - Tokens grant `publish`, `subscribe`, or both operations for a specific relay. - Tokens can have expiration times and can be revoked independently. - Creating a relay automatically returns: - A token capable of publishing and subscribing. - A subscribe-only token intended for viewers. - Additional narrowly scoped tokens can be created through the API or dashboard. - Current tokens apply to an entire relay; Cloudflare is working with the MoQ community on more granular authorization. ## Using the API and Dashboard - A relay can be created with one authenticated API request containing its name. - Tokens can be added through an endpoint such as `/moq/relays/$RELAY_ID/tokens`. - The dashboard workflow is available under **Media > Realtime > MoQ Relay**. - Applications should provide broadcasters with publish-capable credentials and viewers with subscribe-only credentials. ## Connecting Clients - Clients send their token when opening a MoQ session. - The relay enforces the token’s permitted operations. - Cloudflare’s open-source `moq-rs` tools can be used with media generated by tools such as `ffmpeg`. Cloudflare’s authenticated, isolated MoQ relays make the protocol more practical for production real-time applications. Developers can use the API or dashboard to provision a globally distributed relay and manage separate, expiring credentials without operating their own media infrastructure.

Cloudflare Jwt Cdn Webrtc+3

lineJul 31, 20265 min readCurated summary

Analyzing Incident Causes with Natural Language in Grafana: Developing an LLM Agent-Based SRELens

SRELens is a Grafana-based natural-language observability assistant created by LY Corporation’s Home SRE team. It connects metrics, logs, traces, and profiles so engineers can investigate incidents without switching between tools or manually transferring context. The project’s central conclusion is that production reliability depends less on natural-language querying itself and more on controlling the LLM’s tools, prompts, permissions, cost, and failure behavior through backend code and policy. ## The Observability Analysis Problem - Incident investigation traditionally requires moving among: - Grafana or IMON for metrics - LaaS or IU for logs - IMON Trace or Tempo for traces - A separate profiling system - Engineers must manually connect: - Error-rate increases - Error messages - Trace IDs and slow requests - Relevant time ranges, services, and labels - This context switching is especially costly during outages. - The team first consolidated data with a self-hosted LGTM-P stack: - Mimir for metrics - Loki for logs - Tempo for traces - Pyroscope for profiles - OpenTelemetry Collector as the ingestion layer - Centralizing the data helped, but engineers still needed to know the correct datasource, labels, query syntax, and relationships between signals. ## Why an Existing Open-Source PoC Was Not Enough The team initially evaluated an open-source Grafana LLM plugin, but identified several production limitations: - It could not reliably propagate Grafana-authenticated user context for chat history, permissions, and usage limits. - System prompts could not be controlled strongly enough to enforce organizational policies. - Short tool-call limits interrupted multi-step investigations. - Datasource-specific naming differences often produced empty results: - Metrics might use `service_name` - Tempo might require `resource.service.name` - Loki might require JSON parsing or structured metadata filters - Modifying and deploying the solution internally raised operational and licensing concerns. The PoC showed that the key requirement was not merely asking questions in natural language, but retaining control over how the agent operates. ## SRELens Architecture - SRELens runs as a Grafana application plugin. - The frontend provides the chat interface. - The backend handles: - LLM requests - Tool orchestration - Prompt composition - Usage and quota enforcement - Observability queries are executed through an MCP gateway. - A `CompositeClient` combines: - Upstream FlavaMCP observability tools - Local Grafana tools such as `find_grafana_panel` and `render_grafana_panel` - The backend is an orchestration and policy layer, not just a proxy. ## Three-Layer System Prompt Design ### Base System Prompt Defines organization-wide behavior and safety rules, including: - Tool-call ordering - Safe handling of dashboard creation, modification, and deletion - Fallback behavior for empty results - Re-querying with aggregation when results are truncated - Response structure and evidence requirements Only administrators can change this layer. ### Datasource Fragment Encodes environment-specific operational knowledge in YAML: - Preferred Mimir, Loki, and Tempo datasource UIDs - Candidate service-name labels - Loki parsing and filtering rules This prevents the agent from wasting tool-call rounds discovering basic datasource conventions. ### User Prompt Stores personal or team-specific context in Redis, such as: - Owned services - Preferred response formats - Frequently used dashboards User preferences are added as context but cannot override organizational safety policies. ## Backend Tool Orchestration and Guardrails The backend exclusively assembles system prompts and runs the agent loop: 1. Send the user’s question to the LLM. 2. Execute requested MCP or local tools. 3. Return tool results to the LLM. 4. Repeat until a final answer is produced. Safety and reliability controls include: - A default maximum of 10 tool-call rounds - Duplicate-call prevention using call hashes - A default retry limit of two attempts per tool - Per-tool result-size limits - Trimming older tool results when the request history becomes too large - Preserving `tool_call_id` relationships when trimming history - Hints that encourage changing labels, time ranges, or datasources after empty results These safeguards reduce dependence on the LLM making perfect decisions. ## Usage Limits and Degraded Operation - Per-user daily token quotas - Per-user requests-per-minute limits - HTTP 429 responses after limits are exceeded - Post-response accounting based on actual prompt and completion tokens returned by OpenAI - Daily quota reset at midnight in the Asia/Seoul timezone - Redis stores conversation history, user prompts, and quotas. - If Redis is unavailable, personalization and history are reduced, but a single chat request can still proceed. ## Incident Analysis Scenario In one beta service, SRELens was asked to investigate an error spike between 09:50 and 10:05. - Instead of separately searching alerts, logs, and traces, the agent examined the relevant dashboard and observability data together. - It narrowed the incident to a surge in `CopyMedia` requests. - The analysis was intended to connect the request pattern with the underlying errors and supporting telemetry, demonstrating how SRELens can move from an aggregate error spike toward a specific API-level cause. SRELens demonstrates that an LLM can accelerate incident analysis when it is grounded in an integrated observability stack and constrained by explicit backend policies. For production use, organizations should treat prompt control, tool orchestration, permissions, quotas, retries, and failure handling as core system components rather than leaving them entirely to the model.

Observability Model Context Protocol Redis Grafana+3

figmaJul 31, 20262 min readCurated summary

AI Fluency Isn’t the Finish Line | Figma Blog

AI skills are increasingly viewed as essential, but Figma argues that tool fluency is only the starting point. As AI makes it easier to generate work, the more valuable capabilities are building shared systems, guiding teams toward decisions, and creating an environment where people can experiment together. The goal is not for one person to work dramatically faster alone, but for entire teams to move faster collectively. ## Become an Internal Product Builder - Individual AI expertise has greater impact when turned into shared tools that benefit the whole team. - Useful examples include: - Prototyping agents - Brand plugins - Shared prompt libraries - Internal prototyping playgrounds - Figma researcher Shane Johnston used AI to build an interactive website for exploring the company’s AI report data, making the information accessible to cross-functional stakeholders. - Figma’s Brand Studio created an image-effect generator in Figma Make so teammates could apply custom, on-brand textures to designs with one click. - AI enables more employees—not just engineers—to identify workflow friction and build tools that solve it. - The broader opportunity is shifting from one person working “10x faster” to the entire team becoming more productive. ## Guide People to a Decision - When AI can produce dozens of possible directions quickly, evaluating and selecting among them becomes a core product skill. - Effective facilitation requires involving the right stakeholders, including: - People with dissenting or contrarian perspectives - Colleagues with historical context - Experts who can identify operational, security, or governance risks - One team discovered that an internally vibe-coded app exposed sensitive company project information, illustrating why data governance experts should be involved early. - Teams should provide context before review meetings through: - Prototype demonstrations - Loom videos - Annotated FigJam files - At Figma, these materials help shift meetings away from explaining options and toward discussing trade-offs and making decisions. - Facilitators should ensure discussions reach a clear outcome by inviting quieter participants, clarifying vague recommendations, asking forward-moving questions, and confirming next steps. ## Share Bad Ideas - AI adoption is occurring at different speeds across teams and organizations. - The report found that: - 20% of respondents said individual contributors were advancing faster than their organizations could support. - 27% said leadership was pushing AI adoption while teams struggled to keep up. - Without deliberate knowledge-sharing and collaboration, the gap between early adopters and less experienced users can continue to widen.

AI Large Language Models Prototyping Figma Make+2

gitlabJul 31, 20263 min readCurated summary

How to govern agentic AI, MCPs, and AI code assistants

Agentic AI changes software governance because agents can independently modify code, call tools, alter CI/CD configurations, and deploy without line-by-line human review. The central control model must therefore focus on agent identity, permissions, data handling, human approval points, and auditable activity. Organizations should treat agents like privileged users, with consistent policies governing what they can access, which models they may use, and when humans must intervene. ## Why Agentic AI Requires New Governance - Traditional code assistants keep humans involved in accepting or rejecting each suggestion. - Agentic systems can perform multi-step actions autonomously across the software delivery lifecycle. - Governance must answer: - What can the agent access? - What is it authorized to do? - What actions did it take, and can those actions be proven later? - Major organizational concerns include: - Distinguishing AI-generated code from human-written code. - Linking generated code to the business requirement or intent behind it. - Maintaining documentation as AI-generated code scales. - The article cites research showing widespread concern about maintainability, technical debt, and governance challenges. ## Controlling Agents, MCPs, and Tools Permissioning becomes especially important when agents use the Model Context Protocol (MCP) to connect to external tools and data. - A governance model should define: - Which agents and workflows are approved. - Where those agents may operate. - Which models they may use. - Recommended controls include: - **Central agent catalog:** Administrators publish and manage approved agents and flows. - **Composite identity:** Actions should be linked both to the agent and the human who initiated it. - **Tool approval guardrails:** Tools can run autonomously, require human approval, or be blocked. - **Prompt guardrails:** Systems should detect attempts to manipulate agents through untrusted webpages, files, or issue comments. - The objective is a role-based, auditable control plane for agents that is as consistent as existing human access management. ## Data Privacy and Self-Hosted AI Because source code is highly sensitive, organizations should establish clear data-handling policies before broad AI adoption. - Key questions include: - Whether the vendor trains models on company code. - Who owns prompts, inputs, and outputs. - Where subprocessors are located and how changes are communicated. - Self-hosted AI can keep agents and data within infrastructure controlled by the organization, which is particularly important for regulated industries. - Bring-your-own-model capabilities allow administrators to: - Use internally validated models. - Assign different models to specific agent workflows. - Reserve trusted models for sensitive operations while using managed models for lower-risk tasks. ## Defining Human Review Boundaries Governance should not eliminate autonomy; it should deliberately establish where autonomy ends. - **Interactive workflows:** Developers review and approve suggestions directly. - **Headless workflows:** Agents operate in CI/CD or other automated environments, requiring approval before sensitive actions or inspection afterward through audit trails. - Organizations should define approval requirements for: - Code review. - Testing and validation. - Deployment. - Enforcement mechanisms can include: - Merge request approval policies that apply equally to human- and agent-created changes. - Tool-level approval settings. - Scanner and pipeline controls that block changes failing security or quality checks. - An organization-wide AI governance policy is preferable to informal team-by-team practices because it promotes consistency and simplifies auditing. ## Practical Recommendation Treat agentic AI as a privileged software participant: give every agent a controlled identity, least-privilege permissions, approved model and tool access, strong privacy protections, and complete auditability. Keep human approval at security-sensitive decision points, while allowing lower-risk actions to proceed autonomously under enforced policies.

AI Agents CI/CD Model Context Protocol DevSecOps+4

netflixJul 30, 20264 min readCurated summary

GenRec: Towards LLM-Native Recommendation at Netflix

GenRec is Netflix’s LLM-backed recommendation ranker, designed to reduce dependence on thousands of hand-engineered features. It verbalizes user history, item metadata, and context, then post-trains a Netflix-adapted foundation model for catalog-aware ranking and long-term member value. In large-scale A/B testing, it reportedly improved both short- and long-term metrics while using far fewer labeled examples and input signals than an established production ranker. ## Motivation for an LLM-Native Recommender - Netflix’s existing recommendation stack supports many content types and product surfaces but is costly to extend. - New use cases can require substantial feature engineering, architectural changes, infrastructure work, and experimentation. - LLMs offer: - Shared semantic representations for users, items, and interactions - Natural-language steering through prompts - Rich understanding of content and user histories - General-purpose LLMs are not production-ready on their own because they may: - Over-recommend globally popular titles - Hallucinate items outside the catalog - Ignore business constraints - Provide limited personalization ## Ranking Problem and Long-Term Utility - GenRec ranks the full Netflix catalog, or a provided candidate set. - It uses the user, interaction history, current context, and time to produce a personalized ranking. - The optimization target is expected long-term member utility—a proxy for satisfaction and retention—rather than only immediate clicks or plays. ## Two-Phase Training ### Netflix-Adapted Foundation Model - Netflix first adapts an open-source LLM using proprietary Netflix data. - The model learns: - Netflix content and metadata - Member behavior and preference patterns - General language understanding and generation - This relatively stable foundation can support multiple Netflix applications. ### GenRec Post-Training - A second training phase specializes the foundation model for recommendation. - It focuses on ranking quality, steering, reward alignment, and serving-cost constraints. - This phase is refreshed more frequently to reflect new content and changing member preferences. ## Interaction Data as Conversations - Hundreds of billions of events—including views, play duration, feedback, add-to-list actions, and abandons—are converted into single- or multi-turn recommender conversations. - Each user message includes verbalized: - Context and profile - Interaction history - Item metadata - The recommendation task - Assistant messages represent actual member behavior, such as titles played, viewing duration, or feedback. - During inference, GenRec uses the verbalized request and a catalog-aware scoring head; it does not generate conversational responses. - The conversational format mainly supports language-model training and comprehension of rich textual inputs. ## Context Engineering Instead of Feature Engineering - GenRec represents histories and context as natural language rather than relying primarily on dense, manually designed features. - The token budget becomes the new feature budget, so histories are selectively compressed: - Retain detailed, high-signal events such as long plays and thumbs-up - Remove low-signal events such as brief plays and hovers - Summarize repetitive behavior such as binge-watching - Add detail for important or cold-start items, including new releases - Recent and high-value interactions receive priority, while older information is compressed or dropped. - Prompt structure is also optimized for shared prefixes and prefix caching, reducing serving costs. ## Ranking, Language, and Reward Objectives - GenRec combines several training objectives: - **Catalog-aware ranking:** Uses high-value engagements as positive labels, applies denoising and thresholds, and trains the model with cross-entropy over the catalog or candidate set. - **Language modeling:** Preserves understanding of natural-language histories and metadata and supports potential future features such as recommendation explanations. - **Reward-weighted alignment:** Incorporates business requirements and longer-term satisfaction into training. - Reward signals can help balance content categories such as movies, series, games, live content, and podcasts instead of optimizing only immediate engagement. ## Serving and Results - GenRec runs in prefill-only mode on Netflix’s LLM serving infrastructure, avoiding the cost of decoding generated text. - A catalog-aware scoring head converts the model’s representations into item rankings. - Compared with a mature production ranker, GenRec achieved statistically significant improvements in short- and long-term online metrics. - It did so with a small fraction of the labeled data and input signals used by the existing system. GenRec suggests that recommendation systems can shift from extensive manual feature construction toward careful context engineering, LLM post-training, and reward alignment. The approach is most promising when paired with catalog constraints, efficient serving, and objectives that reflect long-term member value rather than raw engagement alone.

Large Language Models Natural Language Processing Recommendation Systems Reinforcement Learning+2

githubJul 30, 20262 min readCurated summary

Stacked sessions and pull requests in the GitHub Copilot app

GitHub Copilot’s stacked sessions let developers split large, dependent changes into smaller pull requests while preserving their order. Cassidy Williams demonstrates this by modernizing a decade-old React application, recovering from an incorrect branch choice, and then starting a separate `react-bootstrap` replacement on top of the styling work. The approach made a difficult modernization more manageable and reduced the temptation to create an unwieldy “everything” pull request. ## Modernizing a Legacy Application - Williams’ personal dashboard had accumulated outdated dependencies and patterns: - React 15 - Less - An old version of `react-bootstrap` - Updating the application manually had previously seemed too time-consuming. - She used the GitHub Copilot app to plan a frontend modernization focused on: - Replacing Less with Tailwind or vanilla CSS - Improving accessibility and responsiveness - Modernizing dependencies - Cleaning up links, inputs, labels, wrapping, and container widths - Claude Opus 4.8 helped formulate the plan, while GPT-5.5 provided a review. - The initial attempt failed because the work began from the wrong branch. ## Recovering from the Wrong Branch - Williams discovered that an old `dev` branch already contained partial modernization work and was the version she actively used. - The new session had incorrectly branched from `main`, creating compatibility problems. - Rather than discard the work, she asked Copilot to: - Close the incorrect pull request - Start a fresh session from `dev` - Port the styling and accessibility changes onto that branch - Copilot handled the branch and pull request transition, preserving useful decisions from the failed attempt. ## Investigating Legacy Warnings - Testing exposed warnings involving: - `findDOMNode` - `componentWillReceiveProps` - The outdated code was largely coming from `react-bootstrap`, not Williams’ own application code. - She used Plan mode to compare upgrading or migrating existing components with removing the library. - Copilot recommended replacing `react-bootstrap` entirely. ## Stacking Dependent Sessions - Replacing `react-bootstrap` represented substantial scope beyond the current styling work. - Williams chose to submit the existing work first, then create a second session branched from it. - The new session would: - Build on the completed styling changes - Replace `react-bootstrap` - Produce a separate pull request - Eventually merge into `dev` after the first pull request - This structure keeps each change easier to review and test while maintaining the dependency between them. The practical recommendation is to use stacked sessions for large, related modernization efforts: isolate coherent tasks into separate pull requests, branch later work from earlier changes, and avoid allowing AI-assisted development to turn every improvement into one oversized change.

React Accessibility TypeScript Github Copilot+4