I shared a link from NPR Tiny Desk but there was no image preview or video embed. After investigating this I learned that when you do that, each instance picks up the link and individually retrieves the link preview nearly simultaneously. For gargantuan sites like YT or Reddit, this is a trivial amount of traffic and not a problem, but for smaller sites like NPR this results in a sort of botnet DDOS attack on the site's servers. More here. Another example here . For that reason, NPR has blocked Fediverse servers like Mastodon.

I have experienced this on my personal server as well (DDOS), but at the time didn't know why, exactly.

This isn't the end of the world, but certainly is not great and hurts engagement with the content. Is there a solution for this? Perhaps using some sort of webtorrent service like PeerTube where they just fetch the information from another fediverse server instead of the source server? Maybe fetching embeds client-side? At least that would stagger the requests.

you are viewing a single comment's thread
view the rest of the comments

[–] khannie@lemmy.world 9 points 19 hours ago (2 children)

It was my first thought too but there are downsides to it. Trust is a small one given the nature of the system but compute and bandwidth costs could be an issue also, especially for larger instances.

The first solution is more fragile but could be CDN'd and those are cheap.

[–] Kangae_Hishiryo@scribe.disroot.org 4 points 9 hours ago (1 children)

WebTorrent kind of solution like PeerTube is the best, because it's highly scalable yet not that resource-intensive, because everyone (even users) provides their bit; also, as it's entirely P2P, it acts sort like a distributed and decentralized CDN, so it's snappier for everyone because it's more likely that you have a peer near to you like that.

[–] khannie@lemmy.world 5 points 7 hours ago (1 children)

Absolutely LOVE this idea but I feel like a solid implementation would be a pain in the fanny (and it also requires other peers to be online). I dip in and out over the day myself but I don't know what others use case is.

Also it is far less trustworthy and harder to block bad actors (V's removing a known bad actor from a relatively small peer group).

[–] Kangae_Hishiryo@scribe.disroot.org 2 points 7 hours ago

Idk, the part of uptime of peers is solved precisely by the fact that not every peer will be connected at the same moments that every other peer. If I'm connected at 10pm, it's highly likely that there will be at least another single peer at that hour that has what I'm pulling.

As for bad actors, that's something that at the end of the day is unavoidable. Though with some automated systems could be possible to make it more hard for them, but it's not that different from a centralised approach, and thus is better because is computationally more efficient.

[–] Ghoelian@piefed.social 6 points 16 hours ago

I don't think compute resources should be an issue, if anything it saves other instances resources.

As far as I understand, right now, all instances are fetching this data individually, so they all have the same compute load from this. Doing it on one and sending it along with federation should keep resource useage pretty much the same on the originating instance, and less for all others.

This would still increase bandwidth of course, but at least with all VPS providers I have used, bandwith is pretty cheap.