The social contract concept is over-used by people who try to make it cover too much. It becomes a one-sided contract of adhesion which you’re assumed to have agreed to simply by existing. This, however, is simple reciprocation—it’s more like a truce than a contract. It would be unreasonable to expect tolerance from others while refusing to grant the same tolerance to them.

Of course there is no obligation to be intolerant just because the other person is; you are free to make a better choice.