It's not just the networking. Which would be awkward enough. It's also power and IO/Storage.

Just take a look at the PI Clusters people have built, the volume is a few times that of the boards alone.

Also, the CM4 is a bit cheaper than a comparable "complete" SBC, though I don't know if you'd come out ahead with the price of the board.