this post was submitted on 21 May 2026
132 points (100.0% liked)
Fuck AI
7069 readers
1751 users here now
"We did it, Patrick! We made a technological breakthrough!"
A place for all those who loathe AI to discuss things, post articles, and ridicule the AI hype. Proud supporter of working people. And proud booer of SXSW 2024.
AI, in this case, refers to LLMs, GPT technology, and anything listed as "AI" meant to increase market valuations.
founded 2 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
Yeah hosting just about anything is terrible these days. These AI scrapers just can't act normally, there was nothing wrong with the way GoogleBot and Bing Bot work. They scrape the website, respect robots.txt and nofollow, they rate limit themselves as to not overload the servers. It was just fine.
These days with those AI scrapers they go absolutely ape shit, they issue dozens of requests every second, try to scrape anything and everything. Going so far as to make up urls, just to see if they get lucky. My blocklist is huge and I need to keep updating it all the time. And every now and again one slips through and absolutely slams the server. This causes an alert and I need to act right away. It's fucking terrible.
AI is already shit, why do those companies go out of their way to be even more shit?
Do you have links or tutorials that would help to deal with these issues?
Yes, I use this block list as well as my own additions (mostly IPs of misbehaving bots):
https://github.com/mitchellkrogza/apache-ultimate-bad-bot-blocker
It's specifically for Apache, but that's what I use. There are more of these kinds of lists available.