Cloudflare will start blocking AI crawlers by default on pages with adverts
Cloudflare is to reverse its default settings for AI traffic on its network. From 15 September, crawlers that collect content for AI training will then be blocked by default on pages that display adverts. Search crawlers will still be allowed.
At the same time, the company is restructuring its payment model so that online publishers are paid according to how often their productions are used. Currently, it is still based on how often they are crawled. However, an item can be crawled once and used a thousand times. That gap is closed under the new rules.
End users will not notice anything of these changes. They all take place in the invisible domain of machine-readable content. Cloudflare, which claims to provide twenty per cent of the infrastructure behind the Internet, is working on formulating new rules of the game to rein in the aggressive raids of AIs.
In the current set-up, publishers are the losers. An AI visits a site, drinks up all the content, improves its language models with it and pays nothing. Cloudflare considers this unfair – all the more so because more than half of Internet traffic already comes from non-human users.
AI summaries erode the classic revenue model in which sites earn money from advertising. The more traffic, the higher the income. That logic no longer holds these days, however. Cloudflare points to research by the Pew Research Center from 2025: when Google shows an AI-generated summary, users click on a regular search result in only 8% of cases. Only 1% clicks on a link within an AI summary.
In the new situation, a publisher can indicate to Cloudflare for which purpose its content may or may not be used. The three scenarios are: Search (indexing content), Agent (AI reads the site on behalf of the user), and Training (collects content for AI training). According to Cloudflare, the site owner may request some form of compensation for each scenario. This can be as simple as a source attribution in the form of a hyperlink, up to financial compensation for supplying unique training data.
This new approach is being tested with Ceramic.ai and You.com.
Besides being Cloudflare’s CEO and founder, Matthew Prince (pictured) is also the owner of a local newspaper in a small town in his home state of Utah.
Emerce
Subscribers 0
Fans 0
Followers 0
Followers