Bot Management | Techlist.io

cloudflareJul 13, 20263 min readCurated summary

Introducing Precursor: detecting agentic behavior with continuous client-side signals

Precursor is Cloudflare’s new client-side, session-based system for detecting bots and agentic automation through continuous behavioral signals. Unlike one-time challenges, it evaluates interactions across an entire user journey, making automation harder to imitate while reducing friction for legitimate users. It complements Turnstile by extending behavioral detection beyond sensitive checkpoints such as login, signup, and checkout. ## The Visibility Gap in Bot Detection - Cloudflare analyzes more than 1 trillion requests daily across over 20% of the web. - Turnstile runs nearly 3 billion times per day and provides risk-based verification at important application endpoints. - However, point-in-time challenges offer limited insight into how users and bots behave throughout the rest of a session. - Precursor addresses this gap by continuously observing interactions across an application. ## Why Session Behavior Matters - Modern bots can execute JavaScript, operate real browsers, and pass individual CAPTCHAs. - Sustaining believable behavior over a complete session is substantially more difficult. - Precursor uses accumulated behavioral signals to improve detection precision. - This can reduce unnecessary challenges for legitimate users while increasing the cost and complexity of bot operations. ## Human and Automated Interaction Patterns - Human mouse movement reflects physical and cognitive constraints: - Wrist and forearm movement produce curved paths. - Clicking involves a delay after seeing an interface element. - Hand tremors create subtle physiological oscillations. - Automated interactions often reveal: - Straight-line movement or mathematically ideal curves. - Unusually precise clicks. - Repeated velocities, origins, timing, and movement rhythms. - Individual actions may appear plausible, but patterns across a full session can expose automation. ## How Precursor Works ### Injection and Data Collection - Cloudflare dynamically injects a compact, obfuscated JavaScript bundle into HTML responses passing through its network. - No additional configuration, third-party embedding, or network connection is required. - The script observes signals such as: - Pointer movement - Keyboard timing and activity - Focus changes - Page visibility - Events are serialized, buffered in memory, and periodically sent for analysis. ### Edge-Side Evaluation - Cloudflare deserializes the incoming data and runs multiple evaluators against the behavioral streams. - Evaluators cross-reference signals to identify inconsistencies. - Examples include checking whether: - Pointer activity matches page visibility duration. - Keyboard events occur while a text field is focused. - The results are consolidated into detection signals that influence bot scoring. ### Session Integration - Signals accumulate across the session rather than resetting with a page refresh or new challenge. - Session metadata also supports: - Shadow-mode heuristics - Session analysis - Predicted-versus-actual completion comparisons - Session delinquency heuristics - These observations help improve detection and adjust a session’s bot score. ## Privacy by Design - Precursor collects only information considered necessary for detecting automation and abuse. - Keyboard behavior is recorded as timing and rhythm, not the actual keys pressed. - Signals are evaluated as aggregate behavioral patterns. - Data is used internally by Cloudflare’s detection systems and is not exposed in customer dashboards or linked to accounts, identities, or persistent profiles. ## Session-Based Analytics - Cloudflare is introducing session-oriented views in Security Analytics alongside Precursor. - These views shift analysis from isolated requests to complete visitor journeys. - The provided article excerpt ends while beginning to describe these dashboards. Precursor is best understood as a complement to Turnstile: continuous, privacy-conscious session analysis can detect automation that passes individual challenges while reducing reliance on aggressive interruptions for genuine users.

Read original

Bot Management JavaScript Bot Detection Behavioral Analysis+3

cloudflareJul 1, 20263 min readCurated summary

Making AI search smarter

Cloudflare argues that AI-powered search has broken the traditional bargain in which publishers exchanged crawl access for traffic and revenue. AI summaries increasingly answer users’ questions without sending them to source sites, leaving publishers to choose between reduced visibility and uncompensated content use. The company proposes rebuilding this relationship through smarter crawling and payments tied to actual content usage. ## Rebuilding the Bargain - Cloudflare’s responsible AI bot principles emphasize: - Transparency about a bot’s identity and purpose - Respect for site owners’ choices - Good-faith behavior - Blocking unwanted crawlers protects publishers but does not create a sustainable business model. - Cloudflare’s broader goal is to make AI search beneficial to users, AI companies, and content creators. ## Making AI Search Smarter - Cloudflare is launching a research program using signals from its global network, which covers more than 20% of the web. - These signals may identify: - Which pages are fresh or have genuinely changed - Which content attracts human and automated traffic - Which sources are high quality and relevant - Answer engines could use this information to surface better content and avoid repeatedly crawling unchanged pages. - More than 50% of traffic from legitimate crawlers reportedly goes toward re-fetching unchanged pages. - Reducing unnecessary crawls would lower: - AI companies’ compute costs - Publishers’ server load and bandwidth expenses - The program is intended to be neutral, limited to search, and will not share content or train foundation models. - Cloudflare plans to publish results and make the capability broadly available later in the year. ## From Pay Per Crawl to Pay Per Use - Cloudflare’s existing Pay Per Crawl model lets publishers charge AI companies for accessing their content. - Cloudflare says crawling is an imperfect measure of value: - A page may be crawled once but cited in thousands of answers. - It may also be crawled repeatedly without ever being used. - The company is therefore experimenting with Pay Per Use, where compensation reflects how often content contributes to search results or answers. - Early partners include Ceramic.ai and You.com. - Ceramic’s pay-per-query model pays publishers when their content appears in Ceramic search results. - Cloudflare’s network is intended to help AI companies scale these payment systems across millions of participating content owners. Cloudflare’s proposed model combines efficient, change-aware crawling with compensation based on actual content use. If adopted broadly, it could give publishers better control, reduce needless infrastructure costs, and create a more sustainable economic relationship between AI search services and the web.

Read original

Bot Management Cloudflare Ai Search Web Crawling+2

cloudflareJul 1, 20263 min readCurated summary

Your site, your rules: new AI traffic options for all customers

Cloudflare is replacing its broad “Block AI Bots” approach with finer controls based on what automated systems do: Search, Agent, or Training. The goal is to let website owners preserve discoverability and useful automation while blocking uncompensated model training and other unwanted access. These controls will be available to all Cloudflare customers, including Free-tier users. ## Why AI traffic needs more nuance - The traditional crawler exchange—content in return for referrals—has weakened as AI systems increasingly consume content without sending traffic back. - Website owners previously faced a binary choice: - Allow AI access to remain discoverable. - Block automation and protect content at the risk of losing visibility. - This tradeoff particularly harms small sites and can favor established search providers that use the same crawlers for search and training. ## A behavior-based AI taxonomy Cloudflare will classify automated traffic by its purpose rather than simply labeling bots as “AI”: - **Search** - Collects or indexes content to answer future queries. - Builds a database proactively. - Should generally provide referrals or other fair compensation. - **Agent** - Acts in real time on behalf of a person. - Includes chat-fetch bots such as ChatGPT-User and browser-use agents driven by Gemini or Claude. - Visits a site to complete a specific task for a human. - **Training** - Collects content to train or fine-tune a model. - Permanently incorporates data into the model’s underlying architecture. Bots may have multiple classifications. Cloudflare encourages operators to separate Search, Agent, and Training crawlers so site owners can understand and control their access more effectively. ## New controls for AI traffic - Cloudflare is adding separate controls for Search, Agent, and Training traffic. - These options replace the need for a single all-or-nothing AI blocking decision. - The controls will be available to all customers, including those on the Free plan. - Cloudflare will continue tracking other automated behaviors, such as ad verification, feed fetching, and agentic transactions. ## New default rules Starting September 15, 2026: - For new domains, **Training** and **Agent** crawlers will be blocked by default on pages displaying ads. - **Search** crawlers will remain allowed by default because they are more likely to send visitors back. - The policy treats ads as an indication that human attention—and therefore monetizable traffic—is the intended outcome. - Multi-purpose crawlers will be governed by all of their classifications, using the most restrictive applicable rule. - As a result, crawlers such as Googlebot, Applebot, and BingBot may be blocked when customers choose to block Training traffic. - Website owners can opt out of the new defaults through Cloudflare Security settings before September 15. Cloudflare’s recommendation is to manage AI access by behavior: allow Search when referrals matter, permit Agents when real-time user tasks are valuable, and block Training where content reuse is not adequately compensated.

Read original

Bot Management AI Agents Cloudflare Web Crawlers+3

cloudflareApr 21, 20263 min readCurated summary

Moving past bots vs. humans

The distinction between bots and humans is becoming too blurry to serve as the foundation of web protection. Browsers, accessibility tools, proxies, and AI agents can all behave differently while representing legitimate users, while human activity can also be malicious. Website owners should instead focus on intent, behavior, resource usage, and trust. ## The Web’s Original Balance - Browsers act as user agents, mediating between people and websites. - Websites rely on browser conventions to: - Present content correctly across devices. - Support purchases, logins, media, and accessibility. - Deliver advertising and control user experiences. - The web has historically balanced publisher interests with user freedoms through browser standards, extensions, and accessibility requirements. - AI agents disrupt this balance by fetching raw content without rendering pages like browsers. - Publishers often cannot tell whether a request supports one private summary or large-scale model training, making traffic and monetization less predictable. ## The Client-Server Model - Clients request resources from servers, which respond with the requested content. - Websites can scale through additional servers, caching, and CDNs. - The model’s openness allows many types of clients to interact with servers without requiring servers to understand their internal software. - That flexibility creates uncertainty: servers generally cannot see whether a response is: - Rendered for one person using a browser. - Automatically collected, archived, indexed, or reused by another system. ## Why Bot Management Exists - Websites must decide which requests they can afford to serve when capacity, CPU, or cost limits are reached. - Randomly dropping requests is possible but risks blocking legitimate users. - Access controls are also used to: - Separate attacks from normal traffic. - Manage non-malicious load. - Prevent data extraction and fake account creation. - Limit ad fraud and automated actions. - Web clients are unauthenticated by default, so services infer identity and intent from partial signals such as request volume and IP addresses. - A high-volume IP may indicate abuse, a VPN, or multiple users sharing one address, making simple bot-versus-human classifications unreliable. ## Toward Intent and Behavior-Based Protection - The important questions are whether traffic represents an attack, whether crawling is proportional to returned traffic, whether a login from a new country is expected, or whether advertisements are being manipulated. - “Bots” encompass two separate concerns: - Whether known crawlers should receive access when they provide little traffic or value in return. - Whether emerging clients behave unlike traditional browsers, affecting systems such as private rate limits. - Automation detection remains necessary, but protection systems should be designed for a future where automation is common among both legitimate and malicious actors. Website protection should evolve from identifying “bots” to evaluating intent, behavior, proportionality, and risk. The goal is not to determine whether a client is human, but whether its activity is expected, sustainable, and trustworthy.

Read original

Bot Management AI Agents Web Browsers Zero Trust Security+4

cloudflareMar 12, 20262 min readCurated summary

Announcing Cloudflare Account Abuse Protection: prevent fraudulent attacks from bots and humans

Cloudflare’s new Account Abuse Protection suite targets fraud from both bots and humans, focusing on whether activity is authentic rather than merely automated. It combines leaked-credential detection and account-takeover signals with new tools for identifying risky signups and suspicious identities. The capabilities are in Early Access for Bot Management Enterprise customers at no additional cost temporarily. ## Leaked Credentials and Account Takeover - Cloudflare reports that 41% of network logins use leaked credentials, with password reuse allowing old breaches to compromise valuable accounts. - Its leaked credential check compares hashed passwords against known breach data without storing or accessing plaintext passwords. - More than 60% of login-page traffic during the 2024 Black Friday analysis was automated, enabling attackers to test stolen credentials at scale. - Account takeover (ATO) detections identify customer-specific suspicious login behavior and expose attempted attacks in the Security analytics dashboard. - These detections caught an average of 6.9 billion suspicious login attempts per day across Cloudflare’s network during the referenced week. ## Fraud Requires More Than Bot Detection - Modern abuse combines automation, human fraud farms, device and location spoofing, and synthetic identities. - Attackers may use valid credentials, operate at human speed, or employ AI agents, making simple bot classification insufficient. - Common customer problems include fake users exploiting free trials, attackers logging in with correct passwords, and human-paced account draining. - Effective protection must evaluate intent, identity, and authenticity alongside automation. ## Detecting Suspicious Account Creation - Disposable email addresses allow attackers to create large numbers of accounts for promotions or other abuse without maintaining real email infrastructure. - Cloudflare’s disposable email check provides a binary signal that customers can use in security rules. - Organizations can block disposable addresses outright or challenge users who register with them. - Cloudflare also introduces email-risk assessment based on suspicious email patterns and infrastructure, helping identify potentially fraudulent signups. ## Privacy-Preserving User Identification - Hashed User IDs are per-domain identifiers created by cryptographically hashing usernames. - They help customers correlate suspicious activity and mitigate fraudulent traffic without exposing users’ original identifiers. - The feature is intended to identify risky account behavior while preserving end-user privacy. Cloudflare recommends enabling leaked-credential checks and using the new signup, identity, and behavioral signals together. This layered approach is better suited to fraud campaigns that blend valid credentials, human activity, and automated tools.

Read original

Bot Management Cloudflare Fraud Prevention Database Security+4