I think it has to do with misogyny. Men who are perceived as like-women are valued less and derided more. Women get a little more leeway to be like-men, because being like-men is not to become less.

Few people get mad to see a woman wearing jeans and a sports shirt. But as is obvious from popular media right now, men in skirts are scandalous.