That's the old "there is no morality without God" argument. Just because rights aren't handed down on stone tablets (i.e. natural rights) doesn't mean that there are no rights. In the English tradition, rights arise from the long-standing practices and beliefs of a society. Thus, unless we have a society that embraces savagery and murder in general, it's not "just fine" for the majority to cause the genocide of a minority.

Also, what's worse: a system in which the rights of minorities may occasionally be trampled, or a system in which a minority of philosopher kings by design circumscribes the political self-determination of the majority?