Ai Search | Techlist.io

cloudflareJul 1, 20263 min readCurated summary

Making AI search smarter

Cloudflare argues that AI-powered search has broken the traditional bargain in which publishers exchanged crawl access for traffic and revenue. AI summaries increasingly answer users’ questions without sending them to source sites, leaving publishers to choose between reduced visibility and uncompensated content use. The company proposes rebuilding this relationship through smarter crawling and payments tied to actual content usage. ## Rebuilding the Bargain - Cloudflare’s responsible AI bot principles emphasize: - Transparency about a bot’s identity and purpose - Respect for site owners’ choices - Good-faith behavior - Blocking unwanted crawlers protects publishers but does not create a sustainable business model. - Cloudflare’s broader goal is to make AI search beneficial to users, AI companies, and content creators. ## Making AI Search Smarter - Cloudflare is launching a research program using signals from its global network, which covers more than 20% of the web. - These signals may identify: - Which pages are fresh or have genuinely changed - Which content attracts human and automated traffic - Which sources are high quality and relevant - Answer engines could use this information to surface better content and avoid repeatedly crawling unchanged pages. - More than 50% of traffic from legitimate crawlers reportedly goes toward re-fetching unchanged pages. - Reducing unnecessary crawls would lower: - AI companies’ compute costs - Publishers’ server load and bandwidth expenses - The program is intended to be neutral, limited to search, and will not share content or train foundation models. - Cloudflare plans to publish results and make the capability broadly available later in the year. ## From Pay Per Crawl to Pay Per Use - Cloudflare’s existing Pay Per Crawl model lets publishers charge AI companies for accessing their content. - Cloudflare says crawling is an imperfect measure of value: - A page may be crawled once but cited in thousands of answers. - It may also be crawled repeatedly without ever being used. - The company is therefore experimenting with Pay Per Use, where compensation reflects how often content contributes to search results or answers. - Early partners include Ceramic.ai and You.com. - Ceramic’s pay-per-query model pays publishers when their content appears in Ceramic search results. - Cloudflare’s network is intended to help AI companies scale these payment systems across millions of participating content owners. Cloudflare’s proposed model combines efficient, change-aware crawling with compensation based on actual content use. If adopted broadly, it could give publishers better control, reduce needless infrastructure costs, and create a more sustainable economic relationship between AI search services and the web.

Read original

Ai Search Cloudflare Bot Management Web Crawling+2

cloudflareApr 16, 20263 min readCurated summary

AI Search: the search primitive for your agents

AI Search is presented as a general-purpose search primitive for AI agents, handling retrieval across code, support documentation, customer history, and agent memory. It combines semantic and keyword search while providing built-in storage, indexing, and dynamically creatable search instances. The result is less infrastructure to build and the ability to maintain separate searchable contexts for agents, customers, languages, or other tenants. ## Why Agents Need Search - Agents often need access to information too large or dynamic to fit in a context window. - Common examples include: - Coding agents searching millions of repository files. - Support agents searching product documentation and ticket history. - Memory systems retrieving relevant past interactions. - Building this independently requires: - A vector index. - Document parsing and chunking. - An indexing pipeline that stays synchronized with changing data. - A separate keyword index and result-fusion layer if lexical search is also needed. ## Hybrid Search - AI Search runs vector search and BM25 keyword search in parallel. - Results are fused into a single ranking. - This supports both: - Semantic matches based on meaning. - Exact-term matches for names, identifiers, and technical terminology. - The blog’s own search is powered by AI Search. ## Built-In Storage and Dynamic Namespaces - New AI Search instances include managed storage and a vector index. - Files can be uploaded directly through an API and indexed automatically. - Developers do not need to configure R2 buckets or external data sources for every instance. - The `ai_search_namespaces` binding allows Workers to create and delete instances at runtime. - Instances can be created per: - Agent. - Customer. - Language. - Other isolated contexts. - Documents can include metadata used to boost rankings at query time. - A single query can search across multiple instances. ## Customer Support Agent Example - The example uses the Cloudflare Agents SDK and Workers AI. - A shared `product-knowledge` instance contains product documentation backed by an R2 bucket. - Each customer receives a separate instance such as `customer-abc123`. - After an issue is resolved, the agent stores a summary of the problem and its fix. - Over time, each customer’s instance becomes a searchable history of previous resolutions. ## Agent Tools and Retrieval Flow - The support agent extends `AIChatAgent` and uses Kimi K2.5 through Workers AI. - It defines tools for: - Searching shared product documentation and the current customer’s history in one call. - Saving a resolution after an issue is resolved. - The model decides when to invoke these tools based on the conversation. - Search results can prioritize recent documents using metadata, such as a descending `timestamp` boost. - The Agents SDK persists the conversation history across reconnects, while AI Search provides retrieval over larger knowledge collections. AI Search is recommended for teams that want agent-ready retrieval without separately assembling vector databases, keyword indexes, storage, and synchronization pipelines. Its dynamically isolated instances are particularly useful for multi-tenant agents and applications that need both shared knowledge and private, continuously growing context.

Read original

Ai Search Database Design Cloudflare Vector Db+4

figmaOct 24, 20242 min readCurated summary

How We Built AI-Powered Search in Figma | Figma Blog

Figma’s AI search emerged from a practical problem: designers often struggled to find existing work, sometimes relying on Slack to locate files from screenshots or vague descriptions. The team initially pursued design autocomplete, but research showed that designers more often reuse and adapt prior work than create from scratch. This led Figma toward visual and semantic search, launched in 2024, using AI to help users find and reuse relevant designs. ## The Problem of Finding Existing Designs - Designers frequently knew what they wanted visually but not where the source file was located. - At Figma, hundreds of Slack messages showed designers asking colleagues to identify files from screenshots or descriptions. - Traditional keyword search was insufficient when users did not know a component’s exact name or file location. ## From Design Autocomplete to AI Search - Figma began with a three-day AI hackathon in June 2023. - One prototype, design autocomplete, suggested likely next components—for example, a “Get started” button in an onboarding flow. - The team believed AI should handle repetitive tasks so designers could focus on higher-level thinking and user needs. - Internal testing and user interviews revealed that designers commonly revisit old explorations, reuse existing work, and build on prior designs rather than starting from nothing. ## Using Search to Improve AI Suggestions - Figma built search infrastructure alongside autocomplete. - Retrieval-Augmented Generation (RAG) could improve AI responses by supplying relevant examples from existing designs. - Finding designs similar to the user’s current work could make autocomplete recommendations more useful and context-aware. - As testing continued, the team recognized that locating and reusing existing work was a more fundamental need than predicting the next component. ## Visual and Semantic Search - **Visual search** allows users to search with: - A screenshot - A selected frame - A quick sketch - **Semantic search** interprets the meaning and context of text-based queries, even when users do not know the precise component name or description. - Search results can help users discover designs and components and then open, preview, or insert them into their projects. Figma’s experience shows that successful AI features often emerge through iteration rather than from an initial prototype. The practical recommendation is to start with real user behavior, test ambitious ideas, and use AI where it removes friction—in this case, helping designers quickly find and reuse relevant existing work.

Read original

Ai Search Large Language Models RAG Semantic Search+2