Cloudflare, a leading Content Delivery Network (CDN), has taken decisive action against AI firms by blocking their web crawlers from accessing content without authorization or compensation, effective July 1. This policy aims to alleviate the burdens many website owners, including those behind the site Practical Technology, have experienced due to overwhelming AI crawler traffic.
Website owners have reported that AI crawlers, such as OpenAI’s GPTBot and Anthropic’s ClaudeBot, can inundate their sites with an excessive number of requests, slowing down performance significantly. Some providers, like Vercel, have been cited for generating over 4.5 billion such requests monthly, causing frustrations to mount among webmasters.
Unlike traditional search engine crawlers, AI bots often operate more aggressively, revisiting the same pages multiple times a day or submitting hundreds of requests in quick succession. Although AI companies deny culpability for these disruptions, the mounting evidence suggests otherwise.
In response, Cloudflare is blocking AI crawlers for its new clients by default. This means that AI companies must now obtain explicit permission to access website content, clarifying their intentions, whether for model training or other purposes. Furthermore, Cloudflare is set to enhance its defenses against ‘shadow’ scrapers through advanced behavioral analysis and machine learning techniques.
The shift in policy comes from a broader concern among publishers, including notable entities like The Associated Press, Condé Nast, and Ziff Davis, which owns ZDNet. Many in the publishing sector accuse AI platforms of ‘strip mining’ the internet for content without proper compensation or regard for established protocols, such as the robots.txt directives meant to restrict crawling.
In an April 2025 lawsuit, Ziff Davis alleged that OpenAI infringed its copyrights during the development of its AI systems, highlighting the ongoing tensions between content creators and AI firms. Recent court rulings have favored companies like Meta and Anthropic, affirming that their actions fall under fair use, a decision that has not sat well with many authors, artists, and publishers who fear losing control over their work.
Amid this landscape, Cloudflare CEO Matthew Prince explained that the new measures aim to empower publishers and foster a more equitable economic framework that benefits creators as well as future AI developers.
In tandem with its crawler ban, Cloudflare has introduced a ‘Pay Per Crawl’ program, allowing publishers to set fees for AI companies wishing to scrape their content. This initiative, currently in private beta, utilizes an existing server response, HTTP 402, to signal that payment is required to access content.
Given Cloudflare’s significant role in internet infrastructure, this policy shift could ultimately deny AI companies access to a vast amount of web content unless they negotiate terms or pay licensing fees. Nicholas Thompson, CEO of The Atlantic, pointed out that until now, AI firms have faced no financial repercussions for using such content. However, this approach could compel them to reconsider.
As sentiment shifts, many AI companies have opposed the notion of paying for content. Meta’s Sir Nick Clegg recently asserted that requiring artist consent before scraping copyrighted work would threaten the viability of the AI sector. Cloudflare’s policy is a proactive response to the increasing intrusiveness of AI crawling, designed to curb the loss of traffic that would otherwise benefit publishers.
The surge in AI has already impacted traffic to news sites, with publications like Business Insider reporting a staggering 55% decline in views from April 2022 to April 2025. Thompson has warned that traffic from Google could dwindle to nothing if these trends continue unchecked.
As the situation evolves, it’s unclear whether other CDNs like Akamai will implement similar measures. For now, the era of unrestricted AI crawling is waning, particularly for the significant segment of the internet that utilizes Cloudflare’s services.
