this post was submitted on 18 Aug 2025
878 points (98.9% liked)

Technology

74193 readers
3880 users here now

This is a most excellent place for technology news and articles.

Our Rules

  1. Follow the lemmy.world rules.
  2. Only tech related news or articles.
  3. Be excellent to each other!
  4. Mod approved content bots can post up to 10 articles per day.
  5. Threads asking for personal tech support may be deleted.
  6. Politics threads may be removed.
  7. No memes allowed as posts, OK to post as comments.
  8. Only approved bots from the list below, this includes using AI responses and summaries. To ask if your bot can be added please contact a mod.
  9. Check for duplicates before posting, duplicates may be removed
  10. Accounts 7 days and younger will have their posts automatically removed.

Approved Bots

founded 2 years ago
MODERATORS
L3s@lemmy.world
enu@lemmy.world
technopagan@lemmy.world
L4s@lemmy.world
L3s@hackingne.ws
L4s@hackingne.ws
878
Codeberg: army of AI crawlers are extremely slowing us; AI crawlers learned how to solve the Anubis challenges. (i.imgur.com)
submitted 21 hours ago by Pro@programming.dev to c/technology@lemmy.world
125 comments fedilink hide all child comments
 

cross-posted from: https://programming.dev/post/35852706

Source.

you are viewing a single comment's thread
view the rest of the comments
[–] PhilipTheBucket@piefed.social 90 points 21 hours ago (19 children)

I feel like at some point it needs to be active response. Phase 1 is a teergrube type of slowness to muck up the crawlers, with warnings in the headers and response body, and then phase 2 is a DDOS in response or maybe just a drone strike and cut out the middleman. Once you've actively evading Anubis, fuckin' game on.

[–] traches@sh.itjust.works 14 points 21 hours ago (6 children)

These crawlers come from random people’s devices via shady apps. Each request comes from a different IP

[–] PhilipTheBucket@piefed.social 1 points 21 hours ago (1 children)

Is that really true? I guess I have no reason to doubt it, I just hadn't heard it before.

[–] sudo@programming.dev 7 points 19 hours ago

Here's one example of a proxy provider offering to pay developers to inject their proxies into their apps. ("100% ethical proxies" because they signed a ToS). Another is BrightData proxies traffic through users of their free HolaVPN.

IOT and smart TVs are also obvious suspects.

load more comments (4 replies)
load more comments (16 replies)