That's because this view is actively harmful to your colleague who does not consider their gender to be binary. This isn't the equivalent of a colleague telling you they believe gender is non-binary, it's equivalent to someone telling you "I know you think you're a man, but my opinion is that in fact you are not a man". Adding "that's just my 2 cents though, and I believe in your right to (wrongly) believe you're a man!" doesn't help -- you've still just told someone that you don't respect them or how they think of themselves, and that your political stance is that they shouldn't have the right to make that decision.
No one is arguing that the OP shouldn't be allowed to _hold_ this opinion, privately and in his own head. I do argue that the _act_ of sharing the opinion with a colleague at work is harmful, since by doing that the OP is revealing that he does not respect the identity of his colleagues, and further strongly implies the OP will do whatever is in his power (by way of voting for political candidates, for example) to limit the ability of his colleagues to have rights based on these identities.
I don't think you have to physically attack someone to be actively harmful to them and their existence.
