There have been attempts over the centuries. Rammed earth walls attempt to do away with the "dealing with lots of relative small pieces" problem (adobe bricks). But they require form building, which adds a significant labor component that isn't there for the "lots of small pieces" approach.