The web business no longer works – AI bots replace search and advertising

Contents

Cloudflare CEO Matthew Prince has outlined his view of the “Internet after Search” in a presentation. His thesis: Classic web search via Google and other search engines is losing importance, while AI-powered agents and bots are increasingly determining traffic on the net. For website operators and publishers, this means fundamental upheaval.

In his presentation on YouTube, Prince presented data from the Cloudflare network, which, according to its statements, handles a significant portion of global internet traffic. Accordingly, the share of bot traffic has noticeably increased since the beginning of 2025. AI crawlers from companies like OpenAI, Anthropic, and other providers of large language models are systematically scanning the web to collect training data or generate real-time answers for their users. Prince spoke of a shift: where human users previously accessed websites via search engines, automated systems are increasingly taking over this role.

Bot traffic burdens infrastructure

This development has direct implications for infrastructure. When AI bots crawl websites, they generate server load without the affected operators benefiting economically. For web hosts, this means rising costs with simultaneously declining human traffic. Cloudflare observes that many website operators are now looking for tools to control or limit this bot traffic.

Prince argued that the web's business model to date – offering content for free, achieving reach through search engine traffic, and refinancing through advertising – no longer functions in this form. If AI systems directly summarize content and present it to the user, the click on the original source is omitted. Publishers thus lose both reach and advertising revenue.

Cloudflare positions itself as an intermediary

Cloudflare sees itself in a key position between AI companies and website operators. Prince referred to tools developed by the company that allow publishers to control which AI crawlers have access to their content – and under what conditions. At its core, it's about whether and how operators can license their content to AI companies. Some large international publishers have already reached corresponding agreements with OpenAI or Google, but for the majority of smaller and medium-sized providers, practical mechanisms are lacking.

Especially for the European market, regulatory questions also arise. The GDPR and the EU's AI Act set strict limits for data handling. Prince left it open in his presentation whether Cloudflare plans specific tools for EU-compliant AI crawler control. However, demand for them is likely to increase: German publishers, like major media houses, face the same decision as US publishers whether to actively license their content to AI companies or to block access.

A web without classic search?

Prince's vision of an “Internet after Search” is not synonymous with the end of search engines. Rather, he describes a transitional phase in which AI agents act as intermediaries between users and content. This fundamentally changes the power dynamics on the net: it is no longer the search engine that decides which content is visible, but the AI agent that compiles and processes information from various sources.

(fo)