1119
Codeberg: army of AI crawlers are extremely slowing us; AI crawlers learned how to solve the Anubis challenges.
(i.imgur.com)
cross-posted from: https://programming.dev/post/35852706
For mbin I managed to kill the attack of the scrapers only using cloudflare managed challenge for all except to fediverse post endpoints, from fediverse ua agents on certain get endpoints. Managed challenge on everything else.
So far, they've not gotten past it. But, a matter of time.
man, you'd think they'd just use the actual activitypub protocol to inhale all that data at once and not bother with costly scraping.
This A aint very I
Same for all the WordPress blogs, by default in all of them there's an API without authentication that lets you download ALL the posts in an easy JSON.
Dear artificial stupidity bot... WHY THE FUCK ARE YOU FUCKING SCRAPING THE WHOLE PAGE 50 TIMES A SECOND????