Let's look at some cases where modesty culture isn't prevalent.
How about northern Europe where sauna culture is ordinary and mixed-sex nudity is common? Well, we don't see disproportionate rates of rape and sexual harassment in Sweden, Norway, or Finland, do we?
How about Polynesia where women traditionally dress for heat and humidity by baring a lot of skin? Nope, we don't see higher rates of rape in those places either.
Then there's gay culture, famously immodest. High rates of sexual assault among gay men? Nope, not at all, despite the fact that gay men in gay spaces often flaunt lots of skin.
So, if gay men, northern European men, and Polynesian men can control themselves in the presence of bare skin, why shouldn't other men be able to?
Methinks the answer does not lie with the bare skin itself.