"Freedom of speech means the government can't legally punish you for expressing your views."
Well, if the government is illegally punishing you for expressing your views, that seems like it's a free speech issue too. Individuals changing their reaction to you, not so much though - especially when that reaction is further speech.
It doesn't mean that nobody is allowed to criticize you or your views.
In order for speech to mean something at all, it has to have consequences.