Docker docs:
Docker routes container traffic in the nat table, which means that packets are diverted before it reaches the INPUT and OUTPUT chains that ufw uses. Packets are routed before the firewall rules can be applied, effectively ignoring your firewall configuration.
Well, if you have the option you could set up a virtual network through the VPS and have a box with pfsense or something to route all traffic through. Take this with a grain of salt - I've seen this done but never done it fully myself.
I've just disabled all incoming connections (including SSH etc.) and access everything through WireGuard