Your services would first of all need some sort of integration to report failed authentication attempts to your firewall or you wouldn't have anything to act on to start the block. Sounds complicated edit: and also what fail2ban does by reading logs it seems.
If I were you I would ponder if it wouldn't be easier to just setup a headscale/wireguard/openvpn server and connect to your other services through that.
My favourite home firewall right now would be opnsense