"Not a culture fit" could easily mean something like "this person and I have a very fundamental (probably irreconcilable) disagreement about something important". Fundamental disagreements are probably going to result in disagreements about things that emanate somehow from the more fundamental thing.
For example, some people seem to think that when they go to work, they do so voluntarily as part of a mutually beneficial arrangement with their employer. Some people think something more like that they are forced to work and are essentially a slave. These two people will probably have disagreements about ideas that are further from the root than their idea about whether they are essentially free or slaves.
If I were looking for someone to work with, I would be looking not just for the necessary skills, but for deep compatibility. You seemed to be saying "not a culture fit" is just a way of rejecting people without seeming racist/sexist, but I think it's pretty clear that's not true.
But the example you cited is not a legitimate legal reason to dismiss a candidate in the United States. It is a philosophical belief and thus protected by the Equality Act of 2010.
I wholeheartedly disagree and do not believe you have done anything other than support my thesis.
Fundamental disagreements about something important are the very definition of political / philosophical belief.
If someone does not possess the technical skills to do the job, they can be dismissed from consideration, but "compatibility" with your preexisting beliefs is in no way a valid reason.
Could you legally disqualify me from employment because I disagree with you on this very point? No. But you seem to be saying that you would anyway.
Your original contention was that only racists/sexists/whatever-ists hide behind "not a culture fit". That wasn't true. Now you seem to want to talk about some legal point. As I said, I don't want to discuss legality, that's some other topic.
Refusing to defend your statements is not a compelling way to win an argument.
If you believe that you can dismiss a candidate because you have "fundamental disagreements" about politics you are incorrect.
My contention is not that "culture fit" is used exclusively to bar candidates because of race/sex/etc, but that it is used in that way and that it is impossible to know what culture fit even means because it means different things to different people.
To some people, such as yourself, it means something that is not a valid reason to dismiss a candidate.
If nothing else, please respond to this question: If my company does not have a dress code, but the "culture" of the place is that everyone generally dresses in button down shirts and slacks every day. Furthermore, I believe fundamentally that people work more productively when they dress nicely, by my definition of what nice is, and that there is greater team cohesion when they do so.
Then a fully qualified candidate walks into my office wearing clean, but baggy jeans, a hat with a sticker on it, tilted to the side, sneakers, and a t-shirt. He actually mentions in the interview that his style is important to him and part of what defines him.
Can I dismiss this candidate on the grounds that he is not a culture fit for my organization?