At that point the abusers who move in will realize you can't see them and will harass your members with impunity. That doesn't seem like good community management. Actually it seems like throwing new members into the deep end (exposing them to every bad actor you already knew about) and at some point I suspect the only people who join your site will be malicious.
That is flat out not true. Someone constantly attacking your character to others is going to have an effect. Those other people may be influenced by what that person says, and that will affect how they perceive you, which will in turn affect your involvement in the project.
