My view is this: once you bind everyone under the monopoly on violence that is government, its wrong to not let the people who are bound decide what the government should do. That's the fatal flaw of non-anarchist libertarianism to me. Its some cabal getting together to decide what "natural rights" should be then using the monopoly of violence to protect what may be their minority viewpoint. Appealing to "natural rights" is no better than appealing to "divine law." Its self-serving hand waving.
That's the old "there is no morality without God" argument. Just because rights aren't handed down on stone tablets (i.e. natural rights) doesn't mean that there are no rights. In the English tradition, rights arise from the long-standing practices and beliefs of a society. Thus, unless we have a society that embraces savagery and murder in general, it's not "just fine" for the majority to cause the genocide of a minority.
Also, what's worse: a system in which the rights of minorities may occasionally be trampled, or a system in which a minority of philosopher kings by design circumscribes the political self-determination of the majority?
