Cloudflare will begin blocking mixed-use web crawlers from ad-supported pages by default on September 15, 2026. The policy targets crawlers that combine traditional search indexing with AI training or agent activity without giving website owners separate controls for each purpose.
The new defaults will apply to new Cloudflare customers, new websites created by existing customers and existing free-plan customers that have not changed their settings. Website owners can still adjust the controls through their Cloudflare dashboard.
Cloudflare said search crawlers will remain allowed by default, while bots classified as AI training or agent crawlers will be blocked on pages that display advertisements. A crawler serving several purposes will be treated according to all its declared uses and blocked when those uses include restricted activity.
Cloudflare Pressures Crawlers to State Their Purpose
In its official announcement, Cloudflare said website owners often want their pages to remain discoverable through search while limiting the use of their content for AI training or automated agents. The company wants AI providers to operate separate crawlers so publishers can make different decisions for each use.
Cloudflare criticised large search providers that use the same main crawler for search results and AI-powered search features. It said the largest search engine has access to about twice as much online information as other AI companies because publishers may fear losing search visibility if they block its crawler.
Google offers Google-Extended, which allows publishers to restrict the use of their content for Gemini and some other AI products without affecting standard Search inclusion. However, Googlebot continues to collect content for Google Search, including AI Overviews and AI Mode.
Pay Per Crawl Expands Into Pay Per Use
Cloudflare is also developing its Pay Per Crawl service into a system called Pay Per Use. Instead of paying only when a crawler retrieves a webpage, AI companies could compensate publishers when their content is used to generate search results, answers or other products.
The company is initially testing the model with Ceramic.ai and You.com. Participating publishers can receive payments when their material appears in Ceramic’s AI search results or when You.com accesses premium content.
Cloudflare said more than half of AI crawler traffic involves repeatedly fetching pages that have not changed. Limiting unnecessary crawling could reduce bandwidth and computing costs for publishers while encouraging AI providers to use clearer identification and payment systems.
Chief executive Matthew Prince said the changes are intended to support website owners and AI companies whose crawlers clearly state their purposes. Cloudflare has also introduced tools for blocking AI bots, managing robots.txt files and charging crawlers that access publisher content.
Featured image credits: Wikimedia Commons
For more stories like it, click the +Follow button at the top of this page to follow us.