If you go back to Popper's paradox of tolerance, it was predicated on protecting oneself from those who used "fists or pistols" rather than words to argue.