Here's a hypothetical but a realistic scenario that not having reproducible builds is a major issue:
1. I create an app for a client.
2. Client installs the app on their 10,000 enterprise mobile devices and trains the users.
3. Month passes. I've changed everything in the app.
4. Major security or outage event happens. I need to change a line in the app and those 10,000 devices need to be updated ASAP.
5. If I can't checkout the old version, change the line and ship it without wondering "what else will change in this build" - then both me and my client are going to have a bad time.
It will take two weeks for a full qa/audit of the app and there's no time for that.
I've a feeling there's multiple definitions of build reproducibility going on here. I'm guessing you mean that it's not important to be have something byte for byte identical, but more to ensure that exactly the same build steps were run with the same source code?
For most of us, that's what build reproducibility means, but I guess for a subset of users it means producing an identical binary.
> but I guess for a subset of users it means producing an identical binary.
Whenever I hear people talk about the problems of creating reproducible builds, I often hear stuff about timestamps or other metadata inserted by the compiler that would "break" the reproducibility (under the stricter definition).
Having your own source code versioned and dependencies version-pinned (and pretty high confidence that the dependency package foobar-1.12 stays the same over time) seem just like old fashioned "good practice".
The looser definition would imply that all versioned software without external dependencies (or the source of the dependencies manually included in the repository) is reproducible?
Yeah, true. I was thinking of doing release builds in containers via the CI/CD pipeline, keeps the environment pretty static, but not completely static of course.
But further: All of these things would still not be enough for the strictest definition (exact same binary), at least with normal compiler defaults afaik?
Byte-for-byte identical builds is useful mainly because format-agnostic diffing tools are a lot easier to use. There are quite a lot of cases where something is not byte-for-byte identical merely because of a timestamp that affects nothing.
That said, I think it was (and perhaps still is, to many people) surprising just how many sources of irreproducibility exist. Timestamps and absolute build locations are obvious sources, and to some degree, generally don't have an effect. Iterating over file inode order (i.e., "for each file in directory {}") is usually innocuous, but it can cause link order issues and change ordering of static constructors--which can have drastic effects (both in terms of performance and actual functional changes) on the resulting binary.
But if your build process is going to unexpectedly change the encoding of text files [1], that's actually pretty terrifying. There are also cases where the compiler just seems to randomly choose how to optimize code [2]. Note that randomness isn't coming from an obvious "if rand() % 4" check here, but perhaps from something more subtle such as "we're iterating over a map whose keys are addresses of internal data structures, and we stop optimizing after hitting 1000 entries as the function is too big, and the addresses change because link order or ASLR."
You would still have to do it with reproducible builds. It's only reproducible if you have the same code. If you change something there is no reasons to get the same result, a bit like you would expect with a hash, it is deterministic, but that's all.
I recently took a look at the top 20 reviewers, and based on some personal knowledge of what the items that buy 5-star reviews are in the current market, at least 3 of the top 20 people are doing it. These are people with thousands of reviews. It's rotten all the way to the top.
