That's oversimplified. First robots usually did not make economic sense. After they were made useful and sold in quantities, this made them worth their initial investment.
The problem is simply fricking hard. Bricks need to be split into pieces (half-bricks, third-bricks etc) to accommodate windows, doors and edges. Due to real-world plot plans, there will be plenty of non-right angles. And yes, there's laying mortar which is an art form :D
There's a surplus of cheap labor in the world, so it doesn't make financial sense to bear the upfront costs.