You're thinking in terms of space, now consider the same concept in the dimension of time.

Branch misprediction is a permanent loss. You will never get those nanoseconds back.

Yes, you have to wait longer.

Space inefficiencies require you to install more memory.

If you exhaust memory, it gets paged to disk, increasing access latency. I.e. You have to wait longer.