This doesn't make sense to me; it sounds like you're suggesting I can't make my own moral judgment regarding your actions – that if you think what you're doing is okay, I have to think it's okay too.
Even if morality is only about answering the question, "how should I conduct myself", and not directly about how anyone else should conduct themselves, if one of my values is alleviating the suffering of others, I will need to recognize when other feeling things are being hurt. If you're the one causing the hurt, I suppose you could say that I'm not making a "moral judgment" about your actions (but rather some other kind of assessment), but regardless I will feel a duty to intervene.
Even if morality is only about answering the question, "how should I conduct myself", and not directly about how anyone else should conduct themselves, if one of my values is alleviating the suffering of others, I will need to recognize when other feeling things are being hurt. If you're the one causing the hurt, I suppose you could say that I'm not making a "moral judgment" about your actions (but rather some other kind of assessment), but regardless I will feel a duty to intervene.