That's like, the difference between one ship full of Pilgrims and every subsequent ship, or between one wagon train and every subsequent wagon train, or between the people on the Mayflower and their kids (who showed up in a society that already existed).
The people who draw the distinction generally don't like the pilgrims settlers because of what they did when they got here. In contrast to immigrants, who did not "settle" anyone or any land. You can see a lot of this in today's politics as well.
It's more than a gradient, it's like counting up ones to make 1,000,000. Which one is showing up to a small number and making it big, and which ones are showing up to a big number and hanging around?