Not at all. Libel can punish you for speech. Defamation can punish you for expression. If you break a law, you no longer can vote in some states (I heavily disagree with this, just stating a fact) or own a gun. Etc... Etc... This is very normal.