A similar argument I've heard before used states rights as a defense.

You might have said if states were forbidden from holding slaves then that "exception" could be used to ban absolutely anything.