AI labyrinth to ward off unwanted bots
More and more bots are ignoring the fact that they are unwanted. Cloudflare wants to put a stop to them with a labyrinth trick.
(Image: Cloudflare)
With an AI-generated labyrinth, the service provider Cloudflare wants to expose AI crawlers and other bots and prevent them from retrieving unauthorized content. The "AI Labyrinth" is designed to slow down and confuse unwanted crawlers that ignore "no-crawl" directives and waste their computing time. A developer had already presented something similar in January as a tool called Nepenthes.
According to Cloudflare, AI crawlers now make more than 50 billion requests a day on the Cloudflare network at –, accounting for just under one percent of all web requests. Identifying and blocking these bots individually has led to a kind of arms race that costs time and money, but doesn't really change the problem, according to a blog post.
Honeypot for AI bots
The AI labyrinth, on the other hand, is supposed to function like a kind of honeypot. If an unauthorized crawler is detected, the system does not redirect it, but instead links the requested page to a series of AI-generated pages. According to Cloudflare, these look convincing enough to entice the crawler to pass through. However, the content does not belong to the actual website, causing the crawler to waste time and resources. To avoid spreading false information, the content should be scientifically correct, but not related to the actual page accessed.
Videos by heise
The logic of the Cloudflare developers is that a real person would not navigate through four links in a labyrinth of AI-generated content. It is highly likely that this would indicate a bot, whose further details would then be used for better bot detection.
Integrated as hidden links
Cloudflare uses Workers AI with an open source model to generate the content. To avoid performance losses, the content is not created in real time, but generated in advance, checked for XSS vulnerabilities and stored in R2 for faster retrieval.
The pre-generated content is integrated as hidden links on pages. Each generated page contains meta tags to prevent indexing by search engines. Through targeted attributes and styling, these links remain invisible to human visitors. The new AI Labyrinth is available for all Cloudflare customers to choose from, the company writes.
(mki)
