I would add that “female” is clinical language, intentionally dehumanizing in its original usage.
Female is a scientific word, used to describe biology. Besides not being accurate, as you describe, because of its reliance on the binary, it’s been traditionally used in medicine and law enforcement.
Doctors and medical researchers have traditionally used the words male and female in writing about patients in an effort to distance themselves emotionally from cases. Here the dehumanization is intentional though not necessarily toxic.
But police and jailers took that same usage up starting in the mid twentieth century in a sort of pseudo-professional way, with the end being to de-emphasize the humanity of suspects and detainees.
She’s not a woman. She’s a female suspect.
He’s not a man. He’s a male prisoner.
In the ensuing decades, we’ve seen “female” enter the vernacular as a synonym for woman, but interestingly “male” has not been similarly adopted.
I propose that when we use female to mean woman, we retain some of that element of intentional dehumanization.
Whether we mean to or not, we’re speaking clinically in a way that demeans women.
At least that’s why it bothers me! Not to take away from your original point, which I don’t disagree with at all.