You make it sound simple. Bazel's docs are minimal and what exists is often wrong. For a C or C++ toolchain you might turn to a contributing project such as toolchains_llvm, but even those experts are still figuring it out, and they give you little guidance on how to assemble your sysroot, which is the actual issue in the article. And, the upstream distributions of LLVM are built under certain assumptions, so unless you know exactly what you are doing, you probably also want to build LLVM instead of using their archives. If you figure out how to build LLVM in a completely consistent and correct way with respect to the STL and support libraries, you will be perhaps the 3rd or 4th person in history to achieve it.
You include the target glibc as part of your toolchain
And, I always prefer to actually pick my glibc independently from the host os and ship it with the binary (generally in an OCI image). This way you can patch/update the host without breaking the service.
no, you just need a compatible starlit/glibc. you pick an old version (e.g. RHEL8) or otherwise compile with multiple tool chains if you need newer glibc features
Whether you use Bazel or not, this is a well-known issue with an equally well-known solution. There’s no need for such a lengthy write-up: just use a consistent sysroot across your entire build environment.
If you can’t create your own sysroot image, you can simply download Chromium’s prebuilt one and configure your C++ compile rules correctly. Problem solved.
We also have an dockerfile for clang/LLVM in that repo so the whole thing is hermetic. It’s a bit of shame Bazel doesn’t come with stronger options/defaults here, because I feel like I want to reproduce this same toolchain on every C++ project with Bazel
> the service crashes with a mysterious error: version 'GLIBC_2.28' not found
Mysterious?
This is by the way why many binary Python packages use https://github.com/pypa/manylinux for builds: if you build on an old glibc, your library will still (generally) work with newer versions.
A heterogenous build cluster with non-hermetic builds and shared caching. The fact that this is only a glibc symbol versioning problem and not something far more severe is, well, a blessing.
At the bare fucking minimum, I would expect the CI builds to have a homogenous configuration and their own cache, not writable from outside CI. If you’re lazy, just wipe the cache every time you upgrade the CI cluster. Maybe I’ve just been living too long in environments where we care about artifact provenance and I’m blind to the whims of people who don’t care about provenance.
I want to feel sympathetic, because I know Bazel is a pain in the ass to learn, but it sounds like the author was juggling knives in a windstorm and caught a few pointy ends.
You don’t need containers. Just hermetic builds. Ideally every byte read during the compilation process comes from a file under your control, that you version and you test. That includes the compiler, glibc, and all of your dependencies.
Ambient, implicit dependencies are the devil’s playthings.
Right. But you need tooling to manage those dependencies, and an ecosystem of content to avoid reinventing the wheel. That's what containers provide. Since Bazel predates the modern container ecosystem, it had to invent its own tooling. The user experience is very rough, and the ecosystem tiny. As a result most Bazel users do not run their builds in a controlled environment - see the parent blog post.
Adopting containers would solve this, but it seems to be a major blind spot for the Bazel community. My theory is that for them to adopt container technology, they will have to reinvent it themselves, hence my tongue-in-cheek comment.
Containers tend to be coarse-grained. For example, maybe you are writing a program, so you put the entire build environment in a container. If you have lots of programs with different dependencies, do you need lots of containers? Do you need to rebuild lots of containers when dependencies change? Bazel is much more fine-grained, and in Bazel, each individual build action gets its own build environment.
By default, that build environment includes the host (build system) glibc, compiler, system headers, system libraries, etc. If you want repeatable builds, you turn that off, and give Bazel the exact glibc, compiler, libraries, etc. to use for building your program.
You get the isolation of containers, but much faster builds and better caching.
One of the niceties of Zig's build system is its batteries-included support for targeting an explicitly provided glibc version. No need for alternate sysroots or dedicated old-distro containers like you would with the traditional C/C++ compilers, all you have to do is append it to the target triplet, like so:
I'm in the middle of submitting PRs to multiple projects because they are compiling on ubuntu-latest and forcing a glibc 2.38 requirement. These are multiplatform projects where most or none of the devs use Linux.
The first project I was able to change their workflow to build inside a 20.04 container. The other project uses tauri and it requires some very recent libraries so I don't know if an older container will work.
Do you have any documentation or generic recommendations for solving these issues caused by blindly using GitHub Actions for all compilations?
> The first project I was able to change their workflow to build inside a 20.04 container.
This approach does _not_ work because you end up with the `node` that runs GitHub Actions not being able to run, certainly this will happen if you end using a sufficiently old container.
> Do you have any documentation or generic recommendations for solving these issues caused by blindly using GitHub Actions for all compilations?
where you replace `DEBIAN_RELEASE` with the release you want to target, and then
- configure your project's build to use that sysroot.
That's it.
If your project does not support sysroots, make it do so. In general compilers will support sysroots, so it's just a matter of making your build configuration facility support sysroots.
