"I think the average person just lumps all of LGBTQIA+ together and thinks of us all as gay."

Absolutely, and education is probably only part of the answer. From your neighbor's reaction, he disapproves of gender and sexual diversity. Like many others, he really doesn't care what labels you choose for yourself. He doesn't like you because you deviate from gender norms. Whether you're violating those norms because you choose to have sex with people of the "wrong" sex or because you wear the "wrong" clothes, it's all much of a muchness to him.

If he ever figured out what non-binary meant, he'd dislike you just as much.

This is what people mean when they talk about queer "common cause." When discrimination against us is rooted in the same (or in a similar) cause despite our differences, then standing together is necessary for progress toward acceptance.

Back in the '70s, '80s and even through the early 90s, queer people often used "gay" as an umbrella term for all members of gender and sexual minorities. We don't do that anymore, but the public who are opposed to our existence really don't care. They just want us to conform.