Tap water? Noooo. Cozumel tap water is not potable, so unless you have a quality filtration system where you stay, don't drink it.
The water used for making ice, or drinking water in the restaurants & establishments is purified and safe and is kept in large 20-litre bottles. If you are immune compromised or simply want to be more careful, your consideration should probably be directed more towards raw products that have been washed clean before use, such as lettuce. In any case, it's best to ask your server or host if you have any concerns regarding the water used.
If you have concerns or are immune compromised, you may also consider using bottled water to brush your teeth.
Some homes on the island have reverse osmosis filtration - be sure to ask your hosts if you are staying at a rental house, Air BnB, etc. They can advise you if there is safe, filtered "tap" water to drink.
TIP: Restaurants may try to sell you a bottle of water, however technically they are technically obliged to provide you with a glass of water if requested.