I find that separating connotations from the core meanings of words leads to clearer arguments. In your position, if someone calls you sexist and you deny that, it's not clear if you're denying treating someone differently based on their sex or if you just don't think you're mistreating the person. My stance is more likely to lead to a response of, "Yes, I'm being sexist, but not in a wrong way."

We generally agree.