Yes, you need to set the RPATH correctly when building ELF executables and shared objects. You would only not know this if you were only ever building things to install into /usr with dependencies on things in /usr.
> Curious observation: dynamic linking on NixOS is not entirely dynamic. Because executables expect to find shared libraries in specific locations marked with hashes of the libraries themselves, it’s not possible to just upgrade .so on disk for all the binaries to pick it up.
Dynamic linking is not just about being able to upgrade without re-linking. Dynamic linking is not even primarily about that, not anymore, if it ever was.
Dynamic linking is more than anything about semantics that no one has bothered to add to static linking!
Static linking for C is stuck in the 1970s.
Dynamic linking for C makes C more like C+ -- a different language.
Specifically:
- with static linking symbol conflicts are a serious problem
- with dynamic linking symbol conflicts need not be a problem because with direct binding (Illumos) or versioned symbols (GNU), you get to resolve the bindings correctly at build-time and have them resolve correctly at run-time
- at build time you get to list just direct dependencies, and the linker does the rest -- compare to static linking, where you have to list all dependencies only in the final link-edit and then you must flatten the dependency list into some order, and then if there are conflicts, you lose.
For all those who keep harping on how static linking is better than dynamic linking, what I would suggest is that what must be done to make static linking not suck is to enrich .a files with the kinds of metadata that ELF adds to shared objects so we can get the same "list only direct dependencies" semantics when static linking as when dynamic linking. And I would note that libtool does this, just... very poorly.
What I would do to fix static linking:
- have `ld` add a .o to every .a that includes the `-L`/`-R`/-l` arguments given when constructing the .a (normally one does not do this when linking statically!)
- have `ld` look in every .a found when doing a final link-edit to recursively find its dependencies, and, most importantly,
- provide the same direct binding / versioned symbol semantics as in dynamic linking so that external symbols in dependents are resolved to the correct dependencies in the same way as in dynamic linking.
Notionally this is quite simple. But adding this to the various ld implementations would probably be rather a lot of work. Still, if people insist on static linking, this work should be done.
libtool is a build tool made primarily to make your life hell if you do anything that the libtool authors did not plan for. Like who thought it would be a good idea to silently drop unknown linker-driver flags.
It writes metadata into .la files, which are text-based adjuncts to .a files. It's meant to give you a common and portable interface to static and dynamic linking.
But libtool is written in POSIX shell, it's not part of the linker, and it is a bit of a disaster.
> For all those who keep harping on how static linking is better than dynamic linking, what I would suggest is that what must be done to make static linking not suck is to enrich .a files with the kinds of metadata that ELF adds to shared objects so we can get the same "list only direct dependencies" semantics when static linking as when dynamic linking.
It can't. The problem is that .a files do not record their direct dependencies, unlike ELF objects, and instead depend on the final link-edit having the full tree of dependencies provided, but flattened into a list. That flattening loses critical information needed to correctly resolve conflicting symbols.
It absolutely does. When you use cmake, if you link against the target foo which is a static library and itself was marked as linking to bar, then your final executable will have -lfoo -lbar.
Of course this information isn't stored in the .a files, but in cmake's FooConfig.cmake files - who cares as long as it works ? The vast majority of libraries now have those even if they don't use cmake as build system as they are fairly easy to generate.
Those '*.cmake' files only work with CMake, which is an issue if you try using any other build system. There is also pkg-config and the `*.pc` files, which are doing essentially the same job without depending on a build system. But it's really all just patchwork, the files are frequently missing and the whole setup is extremely brittle if you hop between OSs, different library versions or just different CMake versions (e.g. '*.cmake' files exporting different targets and variables). That CMake has to provide those files themselves for a lot of libraries (all those not using cmake) is another issue.
> Those '*.cmake' files only work with CMake, which is an issue if you try using any other build system.
At least meson can parse those. Making every other build system in existence add support for those is definitely less work than changing the format of .a and will yield exactly the same result (not that it matters much, cmake being the standard c/c++ build system for years now)
Also .pc files are near inexistent on the most used desktop OS.
> When you use cmake, if you link against the target foo which is a static library and itself was marked as linking to bar, then your final executable will have -lfoo -lbar.
And that is bad. `-lfoo -lbar` loses important information. It is the linker-editor that needs to have this, and not just the build tooling layered above it. libtool, cmake -- it doesn't matter, they're all broken for static linking as long as static linking is broken in this way.
> Of course this information isn't stored in the .a files, but in cmake's FooConfig.cmake files - who cares as long as it works ?
If the only way to make it work is to adopt a particular build system, then no thanks. But again, it can't actually work -- it can't solve the problems that are in the link-editor itself.
> they're all broken for static linking as long as static linking is broken in this way.
The main app I work on links against
- Qt
- LLVM
- libclang
- ffmpeg
- my app which is itself ~40 libs
- a dozen others
which combined account for multiple hundreds of static libraries on Linux, Mac and Windows and things just work. I could maybe bog my head against theoretical link-time problems which I don't experience or make things better for my end-users and just target_link_libraries(myapp Qt::Core), what do you think is the reasonable option ?
> Curious observation: dynamic linking on NixOS is not entirely dynamic. Because executables expect to find shared libraries in specific locations marked with hashes of the libraries themselves, it’s not possible to just upgrade .so on disk for all the binaries to pick it up.
Dynamic linking is not just about being able to upgrade without re-linking. Dynamic linking is not even primarily about that, not anymore, if it ever was.
Dynamic linking is more than anything about semantics that no one has bothered to add to static linking!
Static linking for C is stuck in the 1970s.
Dynamic linking for C makes C more like C+ -- a different language.
Specifically:
- with static linking symbol conflicts are a serious problem
- with dynamic linking symbol conflicts need not be a problem because with direct binding (Illumos) or versioned symbols (GNU), you get to resolve the bindings correctly at build-time and have them resolve correctly at run-time
- at build time you get to list just direct dependencies, and the linker does the rest -- compare to static linking, where you have to list all dependencies only in the final link-edit and then you must flatten the dependency list into some order, and then if there are conflicts, you lose.
For all those who keep harping on how static linking is better than dynamic linking, what I would suggest is that what must be done to make static linking not suck is to enrich .a files with the kinds of metadata that ELF adds to shared objects so we can get the same "list only direct dependencies" semantics when static linking as when dynamic linking. And I would note that libtool does this, just... very poorly.
What I would do to fix static linking:
- have `ld` add a .o to every .a that includes the `-L`/`-R`/-l` arguments given when constructing the .a (normally one does not do this when linking statically!)
- have `ld` look in every .a found when doing a final link-edit to recursively find its dependencies, and, most importantly,
- provide the same direct binding / versioned symbol semantics as in dynamic linking so that external symbols in dependents are resolved to the correct dependencies in the same way as in dynamic linking.
Notionally this is quite simple. But adding this to the various ld implementations would probably be rather a lot of work. Still, if people insist on static linking, this work should be done.