this post was submitted on 30 Mar 2025
0 points (NaN% liked)

Selfhosted

61755 readers
583 users here now

A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.

Rules:

Detailed Rules Post

  1. Be civil.

  2. No spam.

  3. Posts are to be related to self-hosting.

  4. Don't duplicate the full text of your blog or readme if you're providing a link.

  5. Submission headline should match the article title.

  6. No trolling.

  7. Promotion posts require active participation, with an account that is at least 30 days old. F/LOSS without a paywall has exceptions, with requirements. See the rules link for details. Tags [CBH] or [AIP] are required, see the links in Rule 8 for details.

  8. AI-related discussions and AI-involved promotional posts have additional requirements for tagging, as noted in Rule 7 and the AI & Promotional Post Expanded Rules post, and find example disclosures here.

Resources:

Any issues on the community? Report it using the report flag.

Questions? DM the mods!

founded 3 years ago
MODERATORS
 

Lemmy newb here, not sure if this is right for this /c.

An article I found from someone who hosts their own website and micro-social network, and their experience with web-scraping robots who refuse to respect robots.txt, and how they deal with them.

top 2 comments
sorted by: hot top controversial new old
[โ€“] F04118F@feddit.nl 0 points 1 year ago (1 children)

Interesting approach but looks like this ultimately ends up:

  • being a lot of babysitting / manual work
  • blocking a lot of humans
  • not being robust against scrapers

Anubis seems like a much better option, for those wanting to block bots without relying on Cloudflare:

https://anubis.techaro.lol/

Are there any guides to using it with reverse proxies like traefik? I've been wanting to try it out but haven't had time to do the research yet.