You should be using LTO where incremental build times are a concern, i.e. for development builds.

And for realease builds link time is hardly a concern.