edit: I am a man and the only man in this scenario
This happened some time ago. I was driving and stopped on a gas station. probably for coffee and had to go pee. I saw a long queue of around fifteen to twenty exclusively 30-something girls, definitely bit older than me. They probably were a sports team, since there was a hired couch bus waiting outside and they were mostly wearing sweats. There were separate stalls for men and women (one each), so I went for the men's room. It turned out locked, so I stood just outside it. One of the girls in the queue said that the back of the queue is "back there". I replied "sure, but I'm going to the men's room", understandably assuming they were queueing for the ladies room. To which she said "yeah but there's one queue for both". I am familiar with the concept of shared queues, but mostly from supermarkets or post office, where you would queue for several checkouts and just go to the first one that is free. Never encountered shared queues for gender-separated toilets, so I said "but the toilets are separate, I'm going to the men's room and you can queue for the ladies room" and simply went in without any more protest from them when the men's room emptied (and it was another girl in there).
Were I the asshole?
what sort of privilege were I enforcing?
The privilege of taking a restroom for yourself while everyone is waiting for its turn. When you have a right that other people don't have, it's a privilege.
wouldn't you say then that the girls were enforcing the privilage of using men's toilet due to no men at the station (before I arrived)?
That's not what a privilege is.