You will still typically achieve a better result (take less time building the software, produce software that works better and runs faster) if you "optimize later" than if you try to make everything fast from day one.