Truthfully when I used to carry a wallet, I lost it and or my cards significantly more times than ive ever lost my phone.
But thats in part because my phone is usually linked to another device, so I'll notice if I get more than a few feet from it cause the other device's disconnect.
Both Chlorine and Salt Water can destroy a credit card with repeat or prolonged exposure.
So I definitely wouldn't recommend that, even if you carry a wallet.
Which is why id suggest a phone case with a credit card slot. Still have the physical card for an emergency.