How It All Starts
Most people have had this experience at some point. You are hanging out with a group of people, chatting and trying to have a good time, when some asshole starts looking over his shoulder while saying, "Hey, I've got a joke."
With everyone's attention on them, the asshole starts hedging their responsibility, "Now, I'm not (insert: racist, sexist, homophobic, etc.) BUT..." What follows is a joke purpose-built to express the exact attitudes that the teller just said they didn't hold.
Some people might laugh, some people might uncomfortably look at their toes, and you are standing there, and even if you want to say something, maybe you have reservations.
If you are in a situation where you feel safe and empowered to air your grievances with the asshole, the defense is always the same. The asshole appeals to the few people who laughed (whether or not they were faking it) by saying something along the lines of: "God, it's just a joke. Have a sense of humor."
The Fight Against Argumentum Ad Fuddy-Duddem
So now you are being framed as the fuddy-dud (also known as the spoil-sport, Debbie-downer, killjoy, etc.) because you didn't want some asshole to think it's okay to spew that hate around you. The thing is, even if many people in the group agree with you, the fuddy-dud
argument seems to undercut your position.
And while this is a fallacy, it is an effective fallacy (I like to call it
Argumentum Ad Fuddy-Duddem). It equates someone not laughing at a specific joke as a lack of any sense of humor. In fact, a sense of humor is what allows you to examine the joke and see what logic is working underneath and discover that this asshole's "joke" does not exist to show an unexpected break in logic or humiliate something worth hating (like the boss) but, in fact, serves to justify a logic of oppression and humiliate an oppressed group.
Fuddy-duds do exist, and certainly some leftists are fuddy-duds, but confronting "jokes" that serve oppressive social structures does not make one a fuddy-dud.
And let's face it, why should you be the bad guy? It was the asshole and his "joke" that killed the fun.
Considerations When Confronting Assholes
Of course, in any fluid, live experience, you want to be discerning when telling someone that they are wrong. Some groups and situations might make it unsafe to speak up, or you might be at risk for being fired from a desperately needed job.
Another thing to consider is
how you confront. If the person is misguided and doesn't genuinely have these hateful beliefs (and maybe even if they do) you might catch more flies with honey. Pulling them aside in a discrete manner, from a place of concern and forgiveness, can get your point to sink in, whereas too much confrontation might make the person defensive.