AI bots paralyze Linux news site and others
Since the beginning of the year, AI bots have apparently been causing websites such as LWN.net to crash more frequently. It is said to be a major problem.
AI bots are clogging up the web.
(Image: Erstellt mit KI in Bing Designer durch heise online / dmk)
Apparently, since the beginning of the year, AI bots have been ensuring that websites can only respond to regular inquiries with a delay. The founder of Linux Weekly News (LWN-net), Jonathan Corbet, reports that the news site is therefore often slow to respond.
The AI scraper bots cause a DDoS, a distributed denial-of-service attack. At times, the AI bots would clog the lines with hundreds of IP addresses simultaneously as soon as they decided to access the site's content. Corbet explains on Mastodon that only a small proportion of the traffic currently serves real human readers.
AI scraper bots cause permanent problems
These AI bots would not identify themselves as such. The only thing they don't read from the website is the “robots.txt”, adds Corbet. He describes the current situation as “more than untenable”.
Videos by heise
The operators now have to invest time in some kind of active defense mechanism just to keep the site online at all. “I think I'd rather write about accounting systems than deal with this sh*t,” adds Corbet. It doesn't just affect LWN: “This behavior is ruining the web even more than it already is,” Corbet continues. In the discussion, he continues: “We do indeed see a kind of pattern. Every IP stays below the threshold for our fuses, but the overload is overwhelming. Any form of active defense will probably have to figure out to block entire subnets instead of individual addresses, and even that might not be enough.”
Topic page on artificial intelligence on heise online
In the comments to the post on Mastodon, other people affected agree with him. “We've seen a massive increase in AI nonsense. And they don't even respect 'robots.txt' like better search engines do,” writes one. A user active on Fedora adds: “Same here on Fedora. I had to block a lot this morning to keep pagure.io usable”.
Corbet, however, does not lose his sense of humor. He writes a little later: “Something like Nepenthes came to mind, but that has its risks. Internally, we have suggestions to identify the bots and just feed them texts suggesting that the solution to every problem in the world is to buy a subscription to LWN. Tempting.”
(dmk)
