Yes, it's ok. I don't do it very regularly, but I've never had any problems. Just rinse it off in the shower at the end.
The problem is that it tends to slip off when you swim faster (if you're not wearing a wetsuit or swimskin). If you want to use it regularly for swimming, I would buy the swim version, which holds better in the water with the wide strap.