AI has made software execution dramatically faster, but it has not made judgment easier. The article argues that teams must deliberately decide what is worth building, provide agents with strong context and constraints, and preserve a distinct human point of view. Otherwise, polished prototypes can create hidden tech debt and increasingly average products.
## AI Accelerates Execution, Not Clarity
- AI can quickly produce polished, production-like outputs, but apparent polish may hide weak decisions and fragile implementation.
- Large language models often fill in missing requirements themselves, causing prototypes to fail under real-world constraints.
- **Cognitive surrender** describes accepting AI-generated decisions without scrutiny or independent deliberation.
- Teams should follow a “consideration imperative”: pause to understand the problem and define the right outcome before accepting the first plausible solution.
## Context Has to Come First
- Agentic engineering shifts developers from manually writing every line of code to expressing intent clearly and directing AI.
- Effective intent requires:
- **Deterministic layers**, such as tests, type checks, and validation, to catch model errors consistently.
- **High-signal context**, including specifications and documented components.
- **Clear interfaces**, so agents understand how systems and components connect.
- Tools such as Figma MCP’s Code Connect can provide agents with real production components, including props and variants, rather than forcing them to infer implementation from pixels.
- Investing in design systems and documented decisions gives agents a precise vocabulary and guardrails, producing more consistent output, leaner code, and less technical debt.
## Good Can Still Be Average
- AI tends to generate work that resembles common patterns in its training data, or work that is “in distribution.”
- Typical AI-generated results—geometric gradient logos, familiar presentation fonts, or rounded-corner cards—are competent but interchangeable.
- When teams repeatedly accept adequate results, their judgment can narrow from asking “What should this be?” to choosing “Which option is least wrong?”
- As AI raises the baseline of acceptable work, products can become unremarkable unless people deliberately define what makes them distinctive.
## The Point of View Needs to Be Yours
- AI can improve execution, but it cannot replace a team’s responsibility to establish intent, standards, and a meaningful perspective.
- Without a clear point of view, the model’s default assumptions determine both what gets built and how it looks.
- Moving quickly is valuable only when speed is paired with careful consideration, strong context, and deliberate choices.
Teams should treat AI as an execution partner—not as the source of product judgment. Define the problem, encode decisions in systems and safeguards, and challenge generic outputs before shipping.
LLMs make it easy to generate financial content, but producing trustworthy investment information requires much more than fluent summaries. Toss Securities argues that AI must pass three gates before reaching users: selecting reliable evidence, controlling how responses are generated, and making outputs measurable and improvable. The central principle is to constrain AI autonomy where reproducibility and traceability matter, while preserving it for open-ended exploration.
## Why Investment Information Is Different
- **Timeliness:** Market interpretations can change within hours due to earnings, geopolitical events, or policy news.
- **Accuracy:** A company mentioned in an article may not be the company whose stock moved; it could be a subsidiary, a similarly named firm, or merely a promotional mention.
- **Traceability:** Every generated claim needs supporting evidence, evaluation records, and reproducible processing.
- **Non-stationarity:** Market behavior changes across earnings seasons, interest-rate events, elections, and geopolitical crises. Prompts and models tuned to one period may degrade later.
LLMs and autonomous agents amplify these challenges:
- LLMs can produce fluent but incorrect answers when evidence is incomplete or ambiguous.
- Agents add more failure points through search, tool calls, planning, and state transitions.
- Errors can propagate through different execution paths, increasing operational cost and making debugging difficult.
## Gate One: Selecting What the AI Should Say
The first gate is a context-engineering process that filters and organizes evidence before it reaches the LLM.
### Classify Data at Ingestion
- News, disclosures, and financial data are classified as they arrive using internally developed BERT-based models.
- Metadata includes:
- Taxonomy tags
- Related companies and entities
- Embeddings for vector search
- Pre-classifying data avoids waiting until retrieval to determine whether it is relevant.
### Retrieve Broadly, Then Narrow the Candidates
A hybrid retriever first prioritizes recall, after which candidates are reduced through:
- **Deduplication:** Semantically similar articles are clustered so one event is not treated as many independent events.
- **Reranking and filtering:** Evidence is evaluated for direct relevance to the company’s price movement.
- **Taxonomy labels:** Items are categorized by explanation type, such as earnings, guidance, or corporate actions.
- **Failure labels:** Promotional content, insufficient evidence, and other unsuitable sources are explicitly marked and filtered out.
- **Rubrics:** Evidence is ranked according to predefined relevance criteria.
### Build Reasoning-Friendly Context
The final context is arranged so the model checks:
- What happened
- How the event connects to the target company
- Whether the evidence’s polarity matches the stock’s price direction
- Whether the evidence is sufficient and current
This ordering combines the filtered evidence with metadata such as the company, price direction, and time window.
## Gate Two: Controlling How Responses Are Generated
The second gate limits the action space of LLMs and agents to satisfy product requirements such as cost, latency, reproducibility, and observability.
### Use Task Graphs for Clearly Defined Work
Instead of leaving the entire process to an autonomous agent, Toss Securities separates it into explicit stages:
- Candidate retrieval
- Relevance assessment
- Deduplication
- Evidence construction
- Final response generation
Each stage has defined input and output schemas, making it a debugging and evaluation point while simplifying fallbacks and operational monitoring.
### Choose Autonomy Based on Requirement Clarity
- **Autonomous agents** are useful for open-ended tasks such as discovering investment ideas or exploring possible market scenarios.
- **Procedural orchestration** is better for fixed tasks, such as explaining why a specific stock moved.
- Long ReAct loops increase tool calls, token usage, latency, and trace-management costs.
- For structured products, deterministic pipelines let LLMs focus on summarization, rewriting, and evidence-based explanation rather than tool selection.
Procedural graphs are not merely a replacement for agents. Once defined, they can become reusable tools or sub-agents that other agents call through structured interfaces, such as:
```text
input: ticker, direction, time_window
output: explanation, evidences, reasoning_type
```
## Gate Three: Making the System Evaluatable
Subjective judgments such as “the answer feels weak” do not provide a reliable improvement loop. The system therefore generates structured classifications alongside natural-language responses.
### Generate Rubric Categories with Each Answer
- Outputs include event or reasoning types and failure categories.
- Structured fields make it possible to measure:
- Relevance false positives
- Directional mismatches
- Irrelevant evidence passing the filter
- Precision, recall, and F1 score
- The taxonomy must evolve as new market regimes and failure patterns appear.
- Operational failures, evaluation sets, prompt versions, and model versions should be linked so improvements can be reproduced and quantified.
### Retrieve Context-Specific Few-Shot Examples
Fixed few-shot examples are insufficient because event and failure types vary widely across market conditions. Instead:
- Store operational samples with their decisions, failure labels, and embeddings.
- Embed each new classification or verification task.
- Retrieve similar positive and negative examples.
- Include both successful and failed examples to show the model the decision boundary.
This approach reuses production failures as future evaluation guidance and significantly improves precision and accuracy while preserving recall. Since false positives are especially damaging in investment services, filtering out unsupported explanations is more important than producing fluent text alone.
## Work Beyond Prompts and Model Training
Building an investment-information AI service also requires substantial infrastructure outside the model itself:
- Retrieval strategies and embedding models for finding relevant evidence
- Separately trained classifiers for categorization
- Evidence filtering, validation, and metadata management
- Structured orchestration, monitoring, evaluation, and feedback loops
The practical recommendation is to treat the LLM as one component in a controlled evidence pipeline—not as the sole decision-maker. Use autonomous agents for exploratory tasks, but rely on traceable procedural graphs, evolving taxonomies, and retrieval-based examples when the product must deliver repeatable, defensible financial information.
Frontier LLMs often fail at factual questions not because they lack the information, but because they cannot reliably retrieve information already encoded in their parameters. The authors’ knowledge-profiling framework finds that models such as Gemini 3 and GPT-5 encode 95–98% of tested facts, yet still fail to directly recall 26–34% of them. This suggests that improving factuality increasingly depends on better knowledge utilization, not simply larger models or more training data.
## Knowledge Profiling
- The framework evaluates facts rather than isolated question-answer pairs.
- Each fact is assigned one of five profiles:
- Encoding failure
- Recall failure
- Direct recall
- Recall with thinking
- Inference without encoding
- It distinguishes three behaviors:
- **Encoding:** reproducing a fact in a pre-training-like context.
- **Knowledge:** answering semantically equivalent direct and reverse questions.
- **Recall:** retrieving an encoded fact without external cues.
- “Thinking” can help models recover facts through intermediate reasoning, multi-hop inference, or educated guesses.
## The WikiProfile Benchmark
- WikiProfile contains 2,150 facts extracted from Wikipedia.
- Each fact is tested with 10 tasks:
- Two encoding tasks
- Four knowledge questions
- Four multiple-choice recognition questions
- Questions include different phrasings and relational directions, such as asking either what entity *B* is associated with or what entity *A* is associated with.
- An automated pipeline using Gemini-2.5-Pro generated, refined, searched, and filtered questions, followed by manual validation.
- The study evaluated 13 LLMs, with and without thinking.
- Eight responses were sampled for every model, fact, and task, producing roughly 4.5 million responses graded by LLM-based autoraters.
## Recall, Not Encoding, Is the Main Bottleneck
- Frontier models show near-saturated factual encoding:
- Gemini 3 and GPT-5 encode approximately 95–98% of the benchmark’s facts.
- They nevertheless fail to directly recall 26–34% of those facts.
- Thinking reduces, but does not eliminate, the failures; 11–12% remain inaccessible.
- Scaling improves encoding more strongly than recall.
- In the Gemma 3 family, larger models have substantially fewer encoding failures, while recall failures remain significant and account for a larger share of residual errors.
- The results indicate a shift from knowledge acquisition toward knowledge accessibility and utilization.
## Why Recall Fails
- Recall appears sensitive to the conditions under which a fact was learned.
- Retrieval becomes more difficult when a question differs from the fact’s training-time context, wording, or ordering.
- Rare, long-tail facts illustrate this distinction:
- Their encoding rates are relatively close to those of popular facts.
- Their recall rates are substantially worse.
- This suggests that many rare facts are present in model parameters but are difficult to access, rather than completely absent.
The practical implication is that future factuality improvements should target retrieval, post-training, prompting, and inference-time reasoning—not only model scale or broader data coverage.
Cloudflare Radar Researcher is an AI-powered assistant that lets users explore Radar’s Internet data through natural-language questions. It replaces manual API queries, filters, and chart hunting with interactive visualizations, explanations, follow-up questions, and auditable analysis. The beta tool is designed for both casual users and technical researchers, while grounding responses in real data from Radar’s API.
## Why Cloudflare Built Radar Researcher
- Cloudflare Radar publishes global Internet data covering:
- DNS queries from the 1.1.1.1 resolver
- HTTP traffic across Cloudflare’s network
- Internet quality measurements from Cloudflare Speed Test
- Network outages and other datasets
- Radar serves a wide audience, from journalists and curious users to network operators and academics.
- AI lowers the barrier to using complex datasets by allowing users to ask questions without knowing a dataset’s structure, terminology, or API.
- The tool is especially useful for people working under time pressure, such as journalists investigating current Internet disruptions.
## Natural-Language Data Exploration
- Radar Researcher is available from every Radar page in a docked panel.
- Users can:
- Ask questions in plain language.
- Receive short answers or more detailed reports.
- View real, interactive Radar charts alongside explanations.
- Continue with suggested follow-up questions.
- Use text, voice input, or Radar’s search bar.
- Conversations are saved in searchable history, can be pinned, and can be shared through links that expire after 30 days.
- Users can inspect how the assistant interpreted their question, which datasets it queried, and how it derived its answer.
## Explaining Existing Charts
- An “Explain with AI” action lets users start a conversation from any Radar visualization.
- The assistant receives:
- A screenshot of the chart for visual context.
- The exact underlying data from Radar’s API.
- The current location, date range, and applied filters.
- This combination lets the model explain the exact chart being viewed while preserving numerical accuracy and context.
## Example: Internet Quality in Portugal
- A user can ask about home Internet quality in Portugal without manually selecting API parameters or searching Radar pages.
- Researcher queries the Internet quality API and presents the results through familiar interactive charts.
- Users can ask follow-up questions, such as comparing Portugal with Spain or another neighboring country.
- The assistant can also suggest related investigations, including common Internet outages.
## Example: Investigating an Internet Shutdown
- Researcher can support more open-ended investigations for engineers, researchers, journalists, and network operators.
- For Iran’s 2026 government-directed shutdowns, it:
- Retrieved recorded outage events.
- Collected corresponding HTTP traffic data.
- Explained the timeline in natural language.
- The analysis described traffic falling from approximately 0.58 on January 7 to nearly zero by January 9, partially recovering around January 17, and approaching normal levels by January 27.
- It displayed the findings as an annotated traffic chart and a table of outages, including another shutdown beginning February 28.
- Suggested follow-ups included comparing Iran’s traffic with neighboring countries.
## Cloudflare’s Technical Implementation
- The application runs entirely on Cloudflare’s developer platform.
- A Cloudflare Worker uses the Cloudflare Agents SDK.
- Each conversation is stored in a stateful Durable Object with its own SQLite database.
- Chat history, titles, and streaming responses persist even if users leave the page during generation.
- Workers AI runs open models, including Kimi K2.7.
- An ordered fallback chain across three model families helps maintain availability if one model provider reaches capacity.
- Requests are routed through AI Gateway.
Radar Researcher makes Cloudflare’s extensive public datasets easier to use without sacrificing precision or transparency. It is particularly valuable when users need fast, context-specific analysis backed by interactive charts and verifiable source data.
Cloudflare built the Codex to turn scattered engineering knowledge into governed, machine-readable standards that both engineers and AI agents can apply consistently. It now supports code reviews, technical design reviews, and incident reviews, with AI systems flagging nearly 230,000 violations and blocking about 16,000 merges. The central approach is to combine human-owned RFCs with structured extraction, staged enforcement, and context-aware agents.
## Why Cloudflare Built the Codex
- Engineering guidance previously existed across formal documentation, repositories, chat, and individual experience.
- Engineers struggled to determine whether guidance was current, authoritative, or relevant.
- Growth made it difficult for anyone to know every standard or for reviewers to check every requirement.
- The Codex provides a shared source of truth that can be retrieved and applied at the point of work.
## Governance and RFC Workflow
- The Codex is divided into domains such as:
- Architecture and control plane systems
- Security and reliability
- Programming languages including TypeScript and Rust
- Each domain has an owner responsible for content quality and consistency.
- Standards follow an RFC format using RFC 2119 terminology:
- **SHOULD** for recommendations
- **MUST** for mandatory requirements
- Employees can propose RFCs through structured merge requests.
- Proposals undergo increasingly broad review before domain-owner approval.
- Approved RFCs are published to an internal Astro-powered site.
- Enforcement is deliberately separated from approval:
- Approved standards can generate findings.
- Only enforced standards can block merges.
- This gives teams time to adopt requirements and implement enforcement mechanisms.
## Structured Standards for Agents
- Feeding all 60-plus RFCs directly into an LLM would consume too much context and reduce accuracy.
- A dedicated agent extracts SHOULD and MUST statements into structured JSON.
- Each statement includes:
- A stable slug
- RFC and domain metadata
- Requirement level
- Section and source link
- Stable identifiers allow Cloudflare to track requirements across RFC revisions, systems, monitoring, and exception handling.
- Cloudflare moved from concise Markdown extraction to JSON to enable filtering and progressive disclosure.
- Future metadata may identify which SDLC stage applies, such as design, implementation, or runtime.
## AI Code Review
- The AI code reviewer retrieves relevant statements first and loads complete RFCs only when more context is needed.
- Approved-RFC findings are non-blocking recommendations.
- Violations of MUST requirements in enforced RFCs can withhold approval or block a merge.
- Since launch, the reviewer has:
- Flagged nearly 230,000 violations
- Withheld approval for almost 16,000 violations
## Faster Code Review Alternatives
- Full AI reviews generally take several minutes because they use coordinators and multiple agents.
- To reduce remediation delays, Cloudflare is also developing mechanically verifiable checks.
- Language-specific Codex requirements can be distributed through custom linter configuration packages.
- TypeScript was the first language to receive Codex linter support, alongside standardization on oxlint.
The Codex’s practical value comes from connecting governed human standards to automated enforcement. Cloudflare’s staged RFC lifecycle, stable statement identifiers, and combination of AI review with fast linters provide a scalable way to preserve engineering knowledge while reducing review inconsistency.
Meta’s Generative Ads Recommendation Model (GEM), which powers ad recommendations across Instagram and Facebook, now trains at LLM scale across several thousand GPUs. By co-designing kernels, numerical precision, parallelism, networking, and memory management, Meta doubled end-to-end training efficiency to 20–25% Model FLOPs Utilization (MFU) while increasing training compute fourfold in 12 months. The work shows that recommendation models require infrastructure specifically adapted to their hybrid architecture and data patterns rather than a direct reuse of LLM techniques.
## GEM’s Architecture and Training Challenges
- GEM combines:
- Trillions of sparse embedding parameters.
- Billions of dense parameters.
- Sequence features, such as user activity history.
- Non-sequence features, such as user location and ad representations.
- Different feature groups use customized attention mechanisms while still supporting cross-feature learning.
- Recommendation workloads differ substantially from typical LLMs:
- User histories have highly variable lengths, making padding inefficient and potentially wasting up to 50% of computation.
- Attention patterns are asymmetric, including long sequences with short windows and long queries with short key/value sets.
- Small embedding dimensions and normalization layers create memory-bound operations.
- CTR and CVR optimization are numerically sensitive, so aggressive low-precision training can harm model quality.
## Scaling Across Thousands of GPUs
- GEM’s distributed training latency is determined by the slowest rank and the larger of its local computation or communication time.
- Efficient scaling requires:
- Computation to dominate communication.
- Communication to overlap with computation without resource contention.
- Minimal activation recomputation.
- Balanced workloads across GPU ranks.
- GEM makes these requirements difficult because:
- Trillion-scale sparse parameters generate substantial communication.
- Different layer types provide uneven opportunities for communication overlap.
- Long sequences and large activations pressure GPU memory.
- Jagged inputs create changing load imbalance and stragglers.
## Separating Compute and Scaling Efficiency
- Meta measures end-to-end efficiency with:
- **E2E MFU = Local MFU × Scaling Ratio**
- **Local MFU** measures how effectively one GPU uses its compute hardware, including Tensor Cores and memory hierarchies.
- **Scaling Ratio** measures how much single-GPU performance is retained across thousands of GPUs.
- This framework separates:
- Kernel design and numerical precision issues affecting individual GPUs.
- Parallelism, networking, memory, and load-balancing issues affecting distributed training.
## Compute-Efficiency Optimizations
- Meta developed recommendation-specific GPU kernels, including:
- Jagged Flash Attention (JFA) for variable-length sequences.
- Generalized Dot-Product Attention (GDPA).
- BlockAttention.
- These kernels are designed around GEM’s irregular shapes and asymmetric attention patterns rather than conventional LLM assumptions.
- Mixed ultra-low-precision training, including MXFP8 for attention and MLP layers, improves throughput while accounting for recommendation models’ numerical sensitivity.
- The kernels and precision recipes are customized to exploit the architecture of the latest-generation GPUs.
## Scaling-Efficiency Optimizations
- Meta uses topology-aware five-dimensional parallelism to distribute GEM efficiently.
- Dense parameters use:
- Two-dimensional Fully Sharded Data Parallelism (FSDP).
- Expert Parallelism.
- Sparse parameters use fully sharded two-dimensional model parallelism.
- These strategies are co-designed with Meta’s multi-tier network hierarchy to reduce communication overhead.
- Streaming Multiprocessor (SM)-free collectives help communication run with less interference from GPU computation.
- The overall design targets communication overlap, memory constraints, load balance, and the differing behavior of dense and sparse parameters.
## Results
- GEM’s end-to-end training efficiency increased to 20–25% MFU.
- Efficiency doubled over a 12-month period.
- Total training FLOPs increased fourfold.
- The results demonstrate that recommendation foundation models can reach LLM-scale training, but only through coordinated hardware and software optimization across kernels, precision, parallelism, networking, and memory.
For large recommendation models, LLM infrastructure provides a starting point but is not sufficient. The practical recommendation is to optimize compute and distributed scaling as separate but connected problems, using workload-specific kernels, carefully validated low precision, topology-aware parallelism, and communication strategies tailored to sparse and dense model components.
AI skills are increasingly viewed as essential, but Figma argues that tool fluency is only the starting point. As AI makes it easier to generate work, the more valuable capabilities are building shared systems, guiding teams toward decisions, and creating an environment where people can experiment together. The goal is not for one person to work dramatically faster alone, but for entire teams to move faster collectively.
## Become an Internal Product Builder
- Individual AI expertise has greater impact when turned into shared tools that benefit the whole team.
- Useful examples include:
- Prototyping agents
- Brand plugins
- Shared prompt libraries
- Internal prototyping playgrounds
- Figma researcher Shane Johnston used AI to build an interactive website for exploring the company’s AI report data, making the information accessible to cross-functional stakeholders.
- Figma’s Brand Studio created an image-effect generator in Figma Make so teammates could apply custom, on-brand textures to designs with one click.
- AI enables more employees—not just engineers—to identify workflow friction and build tools that solve it.
- The broader opportunity is shifting from one person working “10x faster” to the entire team becoming more productive.
## Guide People to a Decision
- When AI can produce dozens of possible directions quickly, evaluating and selecting among them becomes a core product skill.
- Effective facilitation requires involving the right stakeholders, including:
- People with dissenting or contrarian perspectives
- Colleagues with historical context
- Experts who can identify operational, security, or governance risks
- One team discovered that an internally vibe-coded app exposed sensitive company project information, illustrating why data governance experts should be involved early.
- Teams should provide context before review meetings through:
- Prototype demonstrations
- Loom videos
- Annotated FigJam files
- At Figma, these materials help shift meetings away from explaining options and toward discussing trade-offs and making decisions.
- Facilitators should ensure discussions reach a clear outcome by inviting quieter participants, clarifying vague recommendations, asking forward-moving questions, and confirming next steps.
## Share Bad Ideas
- AI adoption is occurring at different speeds across teams and organizations.
- The report found that:
- 20% of respondents said individual contributors were advancing faster than their organizations could support.
- 27% said leadership was pushing AI adoption while teams struggled to keep up.
- Without deliberate knowledge-sharing and collaboration, the gap between early adopters and less experienced users can continue to widen.
GenRec is Netflix’s LLM-backed recommendation ranker, designed to reduce dependence on thousands of hand-engineered features. It verbalizes user history, item metadata, and context, then post-trains a Netflix-adapted foundation model for catalog-aware ranking and long-term member value. In large-scale A/B testing, it reportedly improved both short- and long-term metrics while using far fewer labeled examples and input signals than an established production ranker.
## Motivation for an LLM-Native Recommender
- Netflix’s existing recommendation stack supports many content types and product surfaces but is costly to extend.
- New use cases can require substantial feature engineering, architectural changes, infrastructure work, and experimentation.
- LLMs offer:
- Shared semantic representations for users, items, and interactions
- Natural-language steering through prompts
- Rich understanding of content and user histories
- General-purpose LLMs are not production-ready on their own because they may:
- Over-recommend globally popular titles
- Hallucinate items outside the catalog
- Ignore business constraints
- Provide limited personalization
## Ranking Problem and Long-Term Utility
- GenRec ranks the full Netflix catalog, or a provided candidate set.
- It uses the user, interaction history, current context, and time to produce a personalized ranking.
- The optimization target is expected long-term member utility—a proxy for satisfaction and retention—rather than only immediate clicks or plays.
## Two-Phase Training
### Netflix-Adapted Foundation Model
- Netflix first adapts an open-source LLM using proprietary Netflix data.
- The model learns:
- Netflix content and metadata
- Member behavior and preference patterns
- General language understanding and generation
- This relatively stable foundation can support multiple Netflix applications.
### GenRec Post-Training
- A second training phase specializes the foundation model for recommendation.
- It focuses on ranking quality, steering, reward alignment, and serving-cost constraints.
- This phase is refreshed more frequently to reflect new content and changing member preferences.
## Interaction Data as Conversations
- Hundreds of billions of events—including views, play duration, feedback, add-to-list actions, and abandons—are converted into single- or multi-turn recommender conversations.
- Each user message includes verbalized:
- Context and profile
- Interaction history
- Item metadata
- The recommendation task
- Assistant messages represent actual member behavior, such as titles played, viewing duration, or feedback.
- During inference, GenRec uses the verbalized request and a catalog-aware scoring head; it does not generate conversational responses.
- The conversational format mainly supports language-model training and comprehension of rich textual inputs.
## Context Engineering Instead of Feature Engineering
- GenRec represents histories and context as natural language rather than relying primarily on dense, manually designed features.
- The token budget becomes the new feature budget, so histories are selectively compressed:
- Retain detailed, high-signal events such as long plays and thumbs-up
- Remove low-signal events such as brief plays and hovers
- Summarize repetitive behavior such as binge-watching
- Add detail for important or cold-start items, including new releases
- Recent and high-value interactions receive priority, while older information is compressed or dropped.
- Prompt structure is also optimized for shared prefixes and prefix caching, reducing serving costs.
## Ranking, Language, and Reward Objectives
- GenRec combines several training objectives:
- **Catalog-aware ranking:** Uses high-value engagements as positive labels, applies denoising and thresholds, and trains the model with cross-entropy over the catalog or candidate set.
- **Language modeling:** Preserves understanding of natural-language histories and metadata and supports potential future features such as recommendation explanations.
- **Reward-weighted alignment:** Incorporates business requirements and longer-term satisfaction into training.
- Reward signals can help balance content categories such as movies, series, games, live content, and podcasts instead of optimizing only immediate engagement.
## Serving and Results
- GenRec runs in prefill-only mode on Netflix’s LLM serving infrastructure, avoiding the cost of decoding generated text.
- A catalog-aware scoring head converts the model’s representations into item rankings.
- Compared with a mature production ranker, GenRec achieved statistically significant improvements in short- and long-term online metrics.
- It did so with a small fraction of the labeled data and input signals used by the existing system.
GenRec suggests that recommendation systems can shift from extensive manual feature construction toward careful context engineering, LLM post-training, and reward alignment. The approach is most promising when paired with catalog constraints, efficient serving, and objectives that reflect long-term member value rather than raw engagement alone.
The Science One Framework addresses a central weakness in autonomous AI research: polished papers can contain fabricated citations, unreproducible results, or methods that do not match the underlying code. Its Chain-of-Evidence (CoE) framework requires every claim to be linked to concrete evidence and introduces CoE Audit to test those links automatically. In evaluations, Science One produced fully verifiable papers while matching or exceeding baseline research agents and human performance on several benchmarks.
## Chain-of-Evidence for Verifiable Research
- CoE defines trustworthy research artifacts through two requirements:
- **Completeness:** Every claim has a recorded evidence chain.
- **Correctness:** The evidence genuinely supports the claim.
- Claims may include:
- Bibliographic references
- Reported scores
- Method descriptions
- Conclusions
- Evidence can include peer-reviewed papers, experiment logs, executed code, or result tables.
- Hallucinated citations, unreproducible scores, and discrepancies between described and implemented methods are treated as broken evidence chains.
## The Science One Framework
The framework builds evidence into the research process instead of attempting to verify a paper after it has been written.
- **Problem Investigator**
- Uses the Semantic Scholar API to construct a citation graph.
- Reads up to 100 full-text PDFs per topic.
- Produces a structured research brief.
- Restricts final-paper references to sources retrieved through the grounded API, avoiding citations generated from model memory.
- **Discovery Engine**
- Explores ideas through parallel explore-exploit branches.
- Each isolated cycle includes a Solver agent and a task-specific evaluator.
- High-performing solutions are iteratively refined.
- Raw evaluator outputs are preserved in strict, read-only records.
- **Paper Writer and Claim Verifier**
- Creates a structured inventory of factual claims.
- Attaches inline evidence tags linking claims to workspace artifacts.
- Checks each claim against its declared source.
- Rewrites unsupported claims conservatively rather than allowing them to exceed the evidence.
## CoE Audit Integrity Checks
CoE Audit is an automated, post-hoc forensic review of a paper, solution, code, and references.
- **Score verification:** Re-runs the submitted code independently and compares the result with the paper’s reported score.
- **Specification violation:** Checks whether the code solves the intended task without exploiting the evaluator or accessing ground-truth answers.
- **Reference verification:** Validates every bibliography entry against academic APIs.
- **Method-code alignment:** Compares the paper’s method description with the actual implementation using LLM-based judges.
## Evaluation Results
- The audit evaluated 75 papers across five systems-optimization tasks: Prism, Cloudcast, EPLB, LLM-SQL, and transaction scheduling.
- Science One led the evaluated systems on all four integrity checks.
- It had:
- Zero phantom references
- Perfect score verification
- The strongest method-code alignment
- Baseline systems hallucinated up to 21% of references and sometimes described advanced algorithms that were implemented as simple deterministic heuristics.
- Strong verification did not reduce performance:
- Science One matched or exceeded human experts on all five ADRS tasks.
- It achieved the best overall result on Cloudcast and EPLB.
- On additional MLE-Bench and Parameter Golf evaluations, the framework also demonstrated competitive performance, including two Gold Medals across five difficult Kaggle competitions.
The main recommendation is to design autonomous research systems around evidence generation from the beginning. Grounded retrieval, immutable experiment records, claim-level verification, and independent auditing can substantially improve reliability without necessarily sacrificing research performance.
LLMs handle public knowledge well but struggle with company-specific questions because relevant evidence is scattered across documents, code, meetings, and chat—and may be outdated or contradictory. The post argues that this is not merely a search problem: organizations need a shared layer for managing trustworthy context. Topic addresses this by extracting source-aware units, linking concepts and relationships, and verifying their freshness, consistency, and evidentiary support.
## Why Search Alone Is Not Enough
- Search retrieves relevant text but cannot determine whether it is current, authoritative, or consistent with other sources.
- A retry-policy example might contain:
- Documentation saying requests fail immediately
- A meeting discussion proposing three retries
- Code currently implementing two retries
- Agents must still determine:
- Whether the meeting produced a final decision
- Which source is newer
- Whether the code reflects an intentional change or an unfinished implementation
- Whether different sources describe the same behavior
- Topic provides a shared context layer so humans and LLMs use the same sources, relationships, freshness information, and conflict states.
## Six Dimensions of Trust
Rather than compressing trust into one score, Topic evaluates six separate dimensions:
- **Granularity:** Whether the context is a meaningful, independently manageable unit
- **Faithfulness:** Whether the source actually supports the claim
- **Staleness:** Whether the evidence remains valid
- **Canonicality:** Whether different names refer to the same entity
- **Consistency:** Whether sources are compatible
- **Coverage:** Whether important evidence or perspectives are missing
Different checks use different methods: rules and hashes for deterministic validation, LLMs for semantic interpretation, and humans for ambiguous or high-impact decisions.
## Ingesting Documents, Chat, and Code
Topic normalizes information into a common `ContentUnit` containing source type, unit type, original URI, content, hashes, timestamps, and source-specific metadata. It uses different boundaries for each source rather than splitting everything into fixed-size text chunks.
### Structured Document Sections
- Markdown documents are divided by heading hierarchy.
- Parent headings are preserved to retain context.
- Long sections are split only when necessary.
- URLs, document paths, and creation or modification times remain attached to the unit.
### Conversation Threads
- Entire messenger threads are treated as the semantic unit, not individual messages.
- Summaries preserve:
- Technical identifiers such as function names and file paths
- Questions, alternatives, and final outcomes
- Decisions versus unresolved issues
- The system avoids inventing consensus and ignores threads containing only casual conversation.
### Code Symbols and Semantic Cards
- Parsers extract functions, classes, file paths, line ranges, imports, and other symbols without using an LLM.
- Multiple symbols are then grouped into **code semantic cards** describing business behavior.
- Cards retain domain terms, code identifiers, source spans, and the relevant commit SHA.
- LLM-generated cards are checked against actual files, line ranges, supporting spans, and duplicate-card patterns.
- Cards are an intermediate layer for connecting code to business concepts, not a replacement for the code itself.
## Extracting Concepts and Relationships
- Topic extracts concept candidates and supporting evidence from each content unit.
- It preserves the relationship between every concept and its original evidence.
- Similar names are not automatically merged merely because they appear close in meaning.
- Concepts can be consolidated into canonical entities only when sufficient evidence exists.
### Human Review for Ambiguous Terminology
- Normalization and embeddings can identify obvious duplicates.
- Internal abbreviations and aliases may require organizational knowledge.
- Topic creates synonym proposals with their supporting context.
- Humans approve or reject ambiguous aliases; rejected proposals are remembered to prevent repeated suggestions.
### Typed Document–Code Relationships
Topic distinguishes among:
- `supported_by`: code behavior supports the document’s claim
- `contradicted_by`: code behavior conflicts with the document
- `mentions`: both refer to the same area, but support or contradiction is unconfirmed
Embedding search first narrows possible matches, after which semantic verification is performed. Low-confidence or failed checks do not create relationships; an absent relationship means “not yet verified,” not necessarily “unrelated.”
## Incremental Verification and Change Detection
- Stable identifiers and content hashes allow unchanged units to reuse previous extraction and relationship results.
- Deleted sources trigger cleanup of dependent relationships.
- Code anchors store the validating commit and span hash.
- If an anchor disappears, it is marked orphaned.
- If the span remains unchanged, semantic verification can be skipped.
- If the span changes, faithfulness must be checked again.
- Rule-based checks happen before LLM calls, reducing cost and limiting nondeterministic reasoning to cases that require it.
Topic’s practical recommendation is to treat trustworthy internal context as a managed system rather than a search result. Preserve source structure, keep evidence attached to every claim, use automation for deterministic work, and route ambiguous organizational judgments to people.
GitHub Copilot and direct model APIs serve different purposes rather than competing at the same layer. Copilot bundles model access with development workflows—repositories, editors, terminals, issues, pull requests, and organizational controls—while APIs give teams the primitives to build their own systems. The best choice depends on whether you want to own the surrounding infrastructure or use GitHub’s integrated tooling.
## Copilot as Development Tooling
- Copilot supports workflows from GitHub Issues through code changes, testing, pull requests, and review.
- Its value includes integration with:
- Editors and repositories
- Terminals and permitted commands
- Repository instructions
- Pull requests and organizational policies
- Paid plans include code completions and Next Edit Suggestions, while more intensive chat and agentic tasks consume AI Credits.
- Actual cost depends on context selection, input/output/cached tokens, tool calls, retries, and task complexity.
- Organization plans pool credits and provide budgets and usage tracking through the billing dashboard.
## Raw APIs for Systems You Control
- Direct API access is suited to product features, internal agent platforms, evaluation systems, and automation pipelines.
- Teams control prompts, retrieval, model routing, retries, logging, security, credentials, and billing.
- Production agents still require substantial engineering, including:
- Selecting relevant repository or document context
- Preserving instructions
- Handling failed tool calls
- Storing traces and audit records
- Defining data boundaries and approval points
- Agent SDKs can provide orchestration, tools, sessions, and streaming. GitHub’s Copilot SDK exposes the runtime used by Copilot CLI and can run with either a Copilot subscription or a provider key.
## BYOK: Keeping Copilot’s Workflow
- Copilot’s public-preview Bring Your Own Key feature lets teams use supported external models in Copilot Chat, CLI, and VS Code.
- Supported providers include Anthropic, AWS Bedrock, Google AI Studio, Microsoft Foundry, OpenAI, OpenAI-compatible services, and xAI.
- GitHub continues to provide the Copilot harness and integrations, while the customer pays the model provider directly.
- BYOK can preserve existing cloud contracts or provider commitments while maintaining a familiar Copilot workflow.
- Administrators can control which GitHub-hosted or BYOK models teams may use.
- Because BYOK is still in public preview, teams should consult the current documentation before making purchasing or architecture decisions.
## Choosing the Right Layer
- Choose raw API access when you need custom integrations, behavior, security controls, auditing, or billing.
- Choose Copilot when developers primarily need to work faster within existing repositories, editors, terminals, issues, pull requests, reviews, and security processes.
- BYOK is a middle option for teams that want GitHub’s development workflow but prefer to pay for models through an existing provider relationship.
The practical decision is not simply about token price. It is about whether your team needs to build and operate the surrounding AI system or wants an integrated development workflow managed through Copilot.
Hierarchical Interest Representation is an upstream embedding layer for Meta Ads that connects users’ inferred interests with advertisers, products, and services. It combines engagement data, multimodal content, graph learning, and hierarchical abstractions to address sparse deep-funnel signals and rare or unseen entities. The resulting universal embeddings and “Bag-of-Meaning” interest tokens could support retrieval, personalization, supervision, and ranking across Meta’s advertising systems.
## Purpose and Role in Deep-Funnel Optimization
- The system aims to identify people with genuine latent interest in an advertiser’s offerings.
- It is intended to complement systems such as Meta’s Generative Ads Model (GEM), Andromeda, and the Adaptive Ranking Model.
- It uses user behavior—including scrolling, engagement, and explicit “Interested/Not interested” feedback—to infer preferences.
- The broader goal is to improve discovery-oriented ad experiences and downstream conversion performance.
## Technical Challenges
### Sparse Engagement and Large-Scale Graphs
- Users, advertisers, products, services, and campaigns are modeled as graph nodes.
- Interactions and activities form edges in a graph spanning millions of advertisers, millions of ads, and billions of users.
- Deep-funnel feedback is relatively scarce, leaving many entities with limited direct evidence.
### Long-Range Relationships
- Useful signals may come from indirectly connected users and entities rather than direct interactions.
- Capturing these relationships at Meta’s scale requires memory-efficient sparse attention and high-performance graph-learning algorithms.
### Dynamic and Unseen Entities
- The ads ecosystem changes rapidly, while individual entities may have little historical engagement.
- Representations must generalize to rare and previously unseen businesses and products.
## Core Design Properties
### Dimension Reduction
- The raw graph is projected into a configurable “super-graph.”
- Learned latent interest primitives act as super-nodes.
- Sparse user-ad relationships become denser connections at the interest level.
- The primitive graph is more stable and stationary than the constantly changing ads vocabulary.
### Knowledge Enrichment
- Advertiser and product representations incorporate text, images, video, metadata, and catalog attributes.
- Vision and language models process this multimodal content.
- Content helps the system understand what a product or business is, not merely how users interacted with it.
- This enables better generalization to new or sparsely observed entities.
### Unified Relational Representation
- Users, advertisers, products, and latent interest primitives are embedded in a shared metric space.
- The system can estimate:
- Relationships between interest primitives
- Similarity between users, ads, and products
- A user’s proximity to particular interests
- Which interests an advertiser or product serves
- Embedding operations support both primitive-to-primitive and cluster-to-cluster relationship modeling.
### Multiple Hierarchical Granularities
- Coarse representations capture dense, stable, high-level interests.
- Finer representations capture sparse and specific deep-funnel intent.
- Cascading hierarchical layers allow the embeddings to serve different needs across retrieval, personalization, ranking, and supervision.
## Architecture and Training
- The architecture combines:
- An in-house transformer-based graph learner
- Bias-aware attention
- Self-supervised cross-view distillation
- Sparse attention for long-range graph relationships
- It combines real-world semantic knowledge with users’ temporal engagement histories.
- The model learns multi-hierarchical interest representations across a large graph.
- Training is performed end-to-end on real Meta Ads data involving billions of interactions.
## Outputs and Potential Applications
- Universal embeddings for users and ads entities.
- “Bag-of-Meaning” interest tokens representing latent interests at different granularities.
- Potential uses include:
- Ad retrieval
- Personalization
- Ranking
- Specialized ranking architectures
- Training supervision
- Cross-entity similarity and discovery
Hierarchical Interest Representation is best understood as shared infrastructure for Meta’s ads recommendation stack. By combining sparse behavioral evidence with multimodal world knowledge and hierarchical graph abstractions, it could make deep-funnel optimization more robust, especially for specialized, rare, or newly introduced products and advertisers.
Cloudflare argues that generative AI has rapidly replaced the traditional web model in which publishers traded content access for search referrals. With AI now driving much of online discovery and crawler activity, content is increasingly consumed without users visiting its source. The company says a new market is emerging in which transparency, access controls, scarcity, and licensing can help publishers regain economic value.
## AI’s rapid transformation of the Internet
- Generative AI adoption has reached more than 2.5 billion regular users—over 30% of humanity—in roughly 3.5 years, reportedly more than twice the adoption speed of smartphones.
- Users now spend only about 15 minutes on the open web for every hour spent searching for information.
- Instead of visiting and comparing multiple websites, users increasingly receive consolidated answers directly from AI systems.
- More than 50% of Internet traffic is now non-human, marking the arrival of what Cloudflare calls the “agentic Internet.”
## Crawlers are increasingly focused on AI
- AI training accounted for 52% of crawler requests in June 2026, up from 22% in spring 2025.
- Mixed-use crawlers, combining search, agent activity, and training, represented more than 36% of crawler traffic.
- Traditional search crawlers make up a smaller share of activity, even though they remain important for sending visitors to publishers.
- Mixed-purpose crawling makes it difficult for site owners to remain visible to AI-driven discovery without also giving away content for training without compensation.
## The traditional web business model is breaking down
- Historically, publishers allowed search engines to crawl their content in exchange for visibility and referral traffic.
- AI systems now answer questions, conduct research, compare products, and complete tasks without necessarily sending users to original sources.
- Content can therefore be crawled, indexed, and monetized by AI companies while the original publisher receives little or no traffic.
- News and media organizations experienced the disruption first, but retail, software, IT, finance, and other sectors are also affected.
- Some heavily crawled categories have seen human traffic fall by as much as 40% in under a year.
- Publishers are preparing for “Google Zero,” in which search referrals provide little meaningful traffic.
## The impact extends across industries
- Any organization publishing proprietary information online may need a strategy for AI access and monetization.
- The issue affects not only traditional publishers but also businesses whose websites contain valuable product, technical, financial, or industry knowledge.
- Cloudflare frames the sustainability of online content as an economic and public-interest concern because the Internet remains a major global information resource.
## Building a market for content
Cloudflare says Content Independence Day focused on three goals:
- Give site owners transparency and control over how their content is accessed and monetized.
- Create scarcity by allowing publishers to restrict or selectively permit AI access.
- Establish a marketplace where publishers and AI companies can discover, license, and price content.
According to the post, these efforts have helped create the early conditions for a monetized content market.
## Control and data create negotiating power
- Cloudflare’s attribution, business intelligence, and enforcement tools let publishers observe AI access at the network level.
- These tools provide stronger practical enforcement than voluntary mechanisms such as `robots.txt`.
- Publishers can identify:
- How often LLMs attempt to access their content
- Which competing AI systems are crawling their sites
- Which URLs are most in demand
- The relationship between crawling and referrals
- Restricting or controlling access creates scarcity, which gives publishers leverage in licensing negotiations.
- Better operational data reduces information asymmetry and allows content owners to negotiate with evidence rather than guesswork.
Ultimately, the post recommends treating online content as an economic asset rather than an unlimited free input. Publishers should measure AI consumption, control access, and pursue licensing arrangements so that the agentic Internet can support content creation instead of undermining it.
Cloudflare argues that the traditional exchange between crawlers and publishers has broken down as AI bots extract content without sending meaningful referral traffic. This creates lost revenue for publishers while increasing hosting costs, making granular traffic attribution essential. Its new Attribution Business Insights dashboard aims to help site owners identify which bots provide value and make informed decisions about access, blocking, and commercial relationships.
## The Internet’s Changing Economics
- Traditional search engines generally crawled content a few times for each visitor they referred.
- That crawl-to-referral balance supported advertising, affiliate revenue, subscriptions, and direct audience relationships.
- AI crawlers increasingly create a “zero-click” ecosystem by summarizing content without directing users to the original publisher.
- Cloudflare observed AI crawl-to-referral ratios ranging from 118:1 to nearly 50,000:1.
- Publishers face both reduced traffic-based revenue and higher infrastructure costs from unproductive automated access.
## Attribution Business Insights Dashboard
- The dashboard is available to Cloudflare Bot Management customers.
- It provides an immediate view of bot activity without requiring extensive manual analytics filtering.
- It measures:
- Human versus bot traffic to content pages.
- Overall and operator-specific crawl-to-referral ratios.
- Crawl-to-referral trends over 24 hours, seven days, or 30 days.
- Top bots by traffic volume, country, bandwidth usage, and current allow/block status.
- AI crawlers are classified by behavior:
- **Training:** collecting data for future large language models.
- **Search:** refreshing indexes used by retrieval-augmented generation.
- **Agent:** supporting automated interactions that return answers to users.
## Turning Traffic Data into Business Strategy
- Site owners can use high-level metrics to evaluate whether their content security policies are effective.
- More detailed operator-level data helps publishers understand how individual AI companies use their content.
- Comparing operators can support negotiations about:
- Blocking or allowing specific crawlers.
- Licensing content.
- Reconsidering existing commercial agreements.
- Prioritizing relationships with companies that provide meaningful compensation or referrals.
- The dashboard is intended to give publishers concrete evidence—such as comparative crawl volumes and referral performance—when discussing content access with AI companies.
Cloudflare’s recommendation is effectively to stop treating all crawlers alike. Publishers should use crawl-to-referral ratios, resource consumption, crawler purpose, and commercial value to decide which bots deserve access and under what conditions.
Frontend development is increasingly shifting from a coding problem to an orchestration problem. Requirements, designs, documentation, discussions, and existing code are scattered across tools, while LLMs can now connect these sources through repeatable workflows. The article argues that structured, reviewable workflows—rather than clever one-off prompts—are the key to scaling AI-assisted development and improving implementation quality.
## From Prompting to Repeatable Workflows
- A prompt may produce a useful result once, but it does not create a reusable process.
- A workflow defines a repeatable path from inputs to outputs:
- Collect context from Jira, Confluence, Slack, and the codebase.
- Summarize the actual requirements.
- Identify ambiguities and unresolved decisions.
- Propose an implementation plan.
- Wait for human review before modifying code.
- The LLM acts as the engine executing the workflow.
- LY Corporation’s Noah MCP connects systems such as Jira, Confluence, Slack, and GitHub, allowing AI agents to access real organizational context instead of relying on manually copied prompts.
- Once established, the same workflow pattern can be applied across many tickets, even when the specific inputs differ.
## Example: Planning a List Page
The example Jira ticket requests a list page with search, filtering, sorting, and role-based filter visibility.
- In the traditional process, a developer manually:
- Reads the Jira ticket and identifies missing details.
- Searches Figma for loading, empty, and no-results states.
- Finds role-based filter rules in Confluence.
- Searches Slack for prior decisions.
- Inspects the codebase for reusable hooks and components.
- Copies findings into notes and assembles an implementation plan.
- Implements the feature, resolves bugs and edge cases, and submits a PR.
- An AI workflow performs these steps systematically before coding.
- The generated plan identifies:
- A new `FeatureListPage` route and `FeatureList` component.
- Reuse of `useTableFilters` and `useUrlState`.
- Existing API support through `GET /api/<feature>`.
- URL synchronization for filters, sorting, and pagination.
- Role-based visibility using `useCurrentUserRole()`.
- Required loading, empty, and no-results states.
## Surfacing Hidden Requirements
The workflow improves quality by exposing information that might otherwise appear late in development.
- A Slack decision establishes that filter and sort state should use URL parameters rather than `localStorage`, enabling shareable and reloadable views.
- Existing hooks such as `useTableFilters` and `useUrlState` are discovered before new code is written, preventing unnecessary duplication.
- Unresolved questions are explicitly listed for human review, including:
- Whether filter and sort state belongs in URL parameters or `localStorage`.
- Which empty-state design should be used when Figma contains multiple variants.
- Resolving these questions early reduces rework during implementation or PR review.
## Closed-Loop Verification
The workflow should continue after coding rather than stopping when the first implementation is complete.
- The agent compares the implementation with the original plan.
- It runs:
- Type checks.
- Linting.
- Related unit tests.
- Relevant smoke tests or local verification flows.
- It reports:
- Successful checks.
- Failures that were fixed.
- Items that could not be verified automatically.
- UI screenshots or state notes.
- Remaining risks before opening a PR.
- This creates a closed-loop development cycle in which AI not only writes code but also validates its work against the intended requirements.
Teams should treat AI as a workflow and context-orchestration layer, not merely a code generator. The most effective process gathers information across systems, obtains human approval for the plan, implements with existing project patterns, and automatically verifies the result before review.
