Absolutely! I'm in my early '60s and I'm the right age to have lived through the same thing. I lived in New York City at the height of the AIDS crisis, and while we did have sex-positive clubs and parties, they were underground, and you had to know how to look for them. They could not be advertised openly, so people found out about them by word of mouth and guerilla-type flyers and cards passed around hand to hand.
I used to go to this party called Hell that was advertised that way. It was not focused on older guys like you're describing, but I never once witnessed anyone complaining that somebody shouldn't be there, as long as they were being respectful of the event and the people partying there.
We had more pressing matters on our minds than being exclusive! Like the fact that nobody wanted the party to be allowed to happen in the first place. Like we were technically breaking the law just by being there.
Anybody who could jump through hoops to find the place and come party was presumed to be a fellow traveler and welcome.
Sounds like that one jerk at your party could use a history lesson.
I'm glad he didn't ruin your night!