Only if you define them such that you exclude lab-made diamonds. But I agree that we can like things for multiple reasons and that rarity is a part of the demand for diamonds.
In any case, I believe people will eventually forget about "unnatural" diamonds. People will be more creative with thier works if a failure does not cost so much, and the average size will be larger. Consider that modern farmed perls were once considered unnatural. But they are not only cheaper but more uniform than wild ones and now nobody buys those.
It is ad hoc, yes, but is not the only justification.
Diamonds are also rare (unlike olivine) and clear diamonds are pretty (unlike ones with lots of inclusions). We can like things for multiple reasons.