It eventually still boiled down to some kind of forced labour. Feeding those laborers for decades consumed ressources, which the rulers couldn't just magically conjure out of thin air... so e.g. farmers had to probably suffer from that.