When you post on reddit, instagram etc. you're giving the company your valuable data. (And it often is valuable, you put your thoughts into it and all that). Consider if you take 15 minutes to create value content, you'd get paid $5 or sth if you were doing it commercially. Now all that value is donated to big tech companies. (that's the only reason why they have value at all.)
So what's the problem with that? The problem is that reddit and others can instantaneously put your years-long effort behind a paywall. I'm not joking. Look at these:
when i want to see content on reddit (for research) or instagram (for local community) they don't allow me to see the data that users have contributed. Reddit claims that it blocks VPNs for "security reasons", meanwhile i'm using VPNs in browser's incognito mode to not be tracked. Instagram wants you to sign in sothat they amass more users and can take leverage of the network effect even more.
I can understand restricting who can post to limit bot activity, but reading must always be possible. After all, it's our data. That's exactly what happens when you give your data to big companies to "communicate with other people": your data becomes a source of income for them, at our expense.
a lot of lemmy servers use anubis to hinder bots. it's quite effective actually. makes the bots do some hard cryptographic riddles before accessing data. insignificant for human users who only read 100 posts but significant for bots who want to read millions of posts.
What.
All you have to do is host your own lemmy instance and federation will give you all the data. Free of charge
yes i know. sadly a lot of big scrappers don't do that, instead they scrap the web interface. which is significantly inefficient so it slows down the service for everyone else.