this post was submitted on 18 Aug 2025
878 points (98.9% liked)

Technology

74193 readers
3819 users here now

This is a most excellent place for technology news and articles.

Our Rules

  1. Follow the lemmy.world rules.
  2. Only tech related news or articles.
  3. Be excellent to each other!
  4. Mod approved content bots can post up to 10 articles per day.
  5. Threads asking for personal tech support may be deleted.
  6. Politics threads may be removed.
  7. No memes allowed as posts, OK to post as comments.
  8. Only approved bots from the list below, this includes using AI responses and summaries. To ask if your bot can be added please contact a mod.
  9. Check for duplicates before posting, duplicates may be removed
  10. Accounts 7 days and younger will have their posts automatically removed.

Approved Bots

founded 2 years ago
MODERATORS
L3s@lemmy.world
enu@lemmy.world
technopagan@lemmy.world
L4s@lemmy.world
L3s@hackingne.ws
L4s@hackingne.ws
878
Codeberg: army of AI crawlers are extremely slowing us; AI crawlers learned how to solve the Anubis challenges. (i.imgur.com)
submitted 21 hours ago by Pro@programming.dev to c/technology@lemmy.world
125 comments fedilink hide all child comments
 

cross-posted from: https://programming.dev/post/35852706

Source.

you are viewing a single comment's thread
view the rest of the comments
[โ€“] OpenPassageways@lemmy.zip 2 points 18 hours ago (1 children)

What the alternative?

[โ€“] sudo@programming.dev 7 points 17 hours ago

Not much for open source solutions. A simple captcha however would cost scrapers more to crack than Anubis.

But when it comes to "real" bot management solutions: The least invasive solutions will try to match User-Agent and other headers against the TLS fingerprint and block if they don't match. More invasive solutions will fingerprint your browser and even your GPU, then either block you or issue you a tracking cookie which is often pinned to your IP and user-agent. Both of those solutions require a large base of data to know what real and fake traffic actually looks like. Only large hosting providers like CloudFlare and Akamai have that data and can provide those sorts of solutions.