876
Codeberg: army of AI crawlers are extremely slowing us; AI crawlers learned how to solve the Anubis challenges.
(i.imgur.com)
cross-posted from: https://programming.dev/post/35852706
I feel like at some point it needs to be active response. Phase 1 is a teergrube type of slowness to muck up the crawlers, with warnings in the headers and response body, and then phase 2 is a DDOS in response or maybe just a drone strike and cut out the middleman. Once you've actively evading Anubis, fuckin' game on.
I think the best thing to do is to not block them when they're detected but poison them instead. Feed them tons of text generated by tiny old language models, it's harder to detect and also messes up their training and makes the models less reliable. Of course you would want to do that on a separate server so it doesn't slow down real users, but you probably don't need much power since the scrapers probably don't really care about the speed
The problem is primarily the resource drain on the server and tarpitting tactics usually increase that resource burden by maintaining the open connections.
The idea is that eventually they would stop scraping you cause the data is bad or huge. But it's a long term thing, it doesn't help in the moment.
The promise of money — even diminishing returns — is too great. There's a new scraper spending big on resources every day while websites are under assault.
In the paraphrased words of the finance industry: AI can stay stupid longer than most websites can stay solvent.