I think there's a difference probably between intentional and unintentional misgendering.

I think some people are probably calling you ma'am on purpose, because they intensely disapprove of transgender people, and they're shoving that in your face.

It astonishes me that that people even care about other people's personal choices like that, but they clearly do.

As to unintentional misgendering, I don't know. Something about androgyny might prompt them unconsciously to try to nail gender down, as if they're working to try to stop feeling uncomfortable.

Of course, this is a problem too because why should somebody else's gender presentation affect their comfort level?