To your point, rights only exist if people in society uphold them for you. There isn't anything magical about calling something a right.
Speech can be both politically incorrect and incite flamewars or violence. At some point, public health and safety is the concern beyond politics. I think we want politically incorrect speech to have a space on Twitter so that it can be publicly challenged but it should stop short of encouraging violence like we saw in Charlottesville.