I read your comment as arguing that people who structure their social systems based on conservative principles e.g. we like our neighborhood a certain way so we resist rapid change (barring explicit racism, of course) are a tax on society.
I agree with your example and agree that people wanting an explicitly racially segregated community are cynical and racist. But an similarly conservatively principled person may simply prefer a low-density lifestyle and act in ways that resist change for that reason. There may even be scenarios where we can look back and correlate responsible, well meaning, application of certain principles with undesirable social trends, but I don't think that alone invalidates the principle.
I don't know if I misread your comment, but it was odd to see a broad critique on conservatives and their principles:
> My point is just that your logic can be (and is) used to defend any number of despicable causes.
I could use plenty of sound logic to defend despicable causes, but fact that the cause is despicable does not alone invalidate the logic if the logic is sound. When sound logic argues for despicable things (which it often does) we introduce ethics and morality to judge whether we should actually act on the logical conclusion.
I think a better way to phrase your comment would be, "We shouldn't have to put up with despicable causes, period". I'm being a little pedantic, yes.
