> The problem with slavery is that there's no coherent argument that differentiates slaves from masters at all

As I see it, the problem is that there was lots of such argumentation - https://en.wikipedia.org/wiki/Scientific_racism

And an even bigger problem is that this seems to be making a comeback