Whiteness is a symptom of colonialism. That particular distinction was invented in response to Bacon’s Rebellion, which had rebels of all ethnic backgrounds. The (bourgeois/feudal lord) response was to invent the category of blackness, as opposed to whiteness, as a legal category that would be subject to the mistreatment that led to Bacon’s Rebellion. Everyone else would now be lumped together despite existing ethnic distinctions between English, Dutch, German, etc and provided with little titles and cutouts so that they could see a path for themselves beyond that of their black previous compatriots. That worked fairly well and reinforced itself over time, becoming the legalistic political basis of chattel slavery.

To be clear, settlers were already colonists that murdered indigenous people, stole their land, and dehumanized them. But it wasn’t based on whiteness in particular, as the category hadn’t been invented.