If both your Jellyfin server and your siblings are behind residential CGNAT, then high chance your connections are relayed through Tailscale's DERP servers. You can check with tailscale ping-ing your sibilings' nodes.
If this is the case, you may consider selfhosting your own DERP somewhere close to you, but I'd argue the performance gains are minimal compared to the extra costs. Another solution would be to enable IPv6 for both you and your siblings, skipping NAT traversal. I just hope both ISPs support it and support it properly in $CURRENT_YEAR.
This is all assuming you can direct play (i.e. not transcoding) your media. If you're transcoding, then it's good to look into hardware acceleration like the other comment mentioned, too
Hi, I think OP wants their sibilings to directly connect to their PC, skipping any relays, even if it's their VPS.
But if you are comparing setting up your own VPS instead of relaying through Tailscale's DERP, then the answer is... it depends on the distance and whether you can establish VPS->Local VM direct connections.
I found opening a specified port for Tailscale on the VPS to help with direct connections with CGNAT'd peers. I'm not familiar with Pangolin, but I think the same principle applies as long as at least one address:port combination is agreed between Wireguard peers.
If I'm being honest though, before doing all this, try asking your ISPs for IPv6 to avoid these cumbersome things together.