As an intensive (and reluctant) user of GNU Make, I keep looking for a more modern replacement. Redo is not it.
Make is really a complicated combination of these components:
- a dependency definition language with in-place evaluation and half-assed wildcards
- A functional language with extremely limited datatypes (effectively just space-separated strings)
- a clunky macro system
- A "distributed" topological execution engine (did you know Make can coordinate job slots across recursive executions of Make? It's impressive! And shouldn't have to exist!) [1]
All the alternative build systems wisely choose to tackle only some of these features. Redo is interesting in that it's a minimal system that leans on existing Unix tools to fill the gaps, but I can't say I'm impressed with using Shell as the scripting language, though it is arguably more useful than Make's home-grown language that too few people know. Edit: actually, redo's doesn't enforce Shell, it can be any executable. That's more interesting, maybe (flexibility also introduces complexity!)
The most interesting development in this space is the paper "Build Systems A La Carte" [2] by Mokhov, Mitchell, & Peyton Jones, who did a great job of breaking down build systems to their fundamental concepts, and along the way found one particular point of the design space that wasn't fulfilled (Table 2, p79:16). Unfortunately, I fear it'll be some years before something production-ready emerges from that research, (and I'm sure I'll be grumpy about some of their implementation details anyhow! Build systems do not a happy developer make)
Not just a functional language -- I found that my ability to write maintainable (and performance) Makefiles drastically improved when I began to think of Make as a Prolog/Datalog variant with particularly heinous syntax and very limited constraint resolution.
I really like the declarative style of make, to the extent that I've been abusing it as an automation tool. Doing things like checking if a server is alive or setting up a serial port. But due to limitations like depending on mtime and issues with special characters in targets, I've switched some of my more intensive Makefiles to prolog using this: https://github.com/webstrand/robo.
When I find the time I hope to write bindings for netlink for prolog, so that I can declaratively create and configure network namespaces. Complete declarative system configuration using prolog would be a dream come true.
It's funny you mention that because I've wondered for a while what it would be like to have a Prolog-based build system (or even just as a CI config target instead of abusing yaml). I just lack the technical nous (and time) to try and pull it off.
I second Bazel. People keep on mentioning how steep the learning curve is, but the conceptual model is really simple, elegant, and intuitive.
What is steep is the technical know-hows:
1. When things don't work as expected. For example, while it worked flawlessly with languages that it natively support such as Java, that wasn't the case for other languages such as Javascript or Python.
2. When you have to do something advanced such as building custom Bazel rule. You'll need to understand Bazel internals and unfortunately the documentation isn't very intuitive and also somewhat sparse.
The http_archive has a sha256-attribute to deal with the "random". You could add multiple urls to point to your local mirror as well. It's a way to pin external dependencies and get reproducible builds without checking in a big blob into your git. pkg-config in my experience is less reproducible and tends to pollute and be affected by system environment, unless you have some build guru who can setup chroot in your project.
Oh I didn't mean "random" as in "could change". In fact "random" was about the unspecified .so blob, not the http archive.
> pkg-config in my experience is less reproducible
But this is what I mean, you only want full reproducibility in a "corporate" environment. In a FOSS desktop environment you often specifically want to use whatever the system has.
I suspect the experience varies pretty widely based on what languages you use and what you’re trying to do with it. If you’re just building Go binaries then I’m sure Bazel is great, but if you’re doing anything with Python 3 or if you’re doing something a little off the beaten path like code generation, it’s probably a frustrating experience.
Lots of things can be complicated in Bazel, but I would not at all consider code generation to be among them. You just write a genrule, and wrap it in a macro if you want to do the same thing more than once. (It can be more complicated than this, of course, but then code generation isn't the hard part of the problem.)
> If you’re just building Go binaries then I’m sure Bazel is great
Compared to other things in Bazel maybe, sure. But if you're just building Go binaries you don't need Bazel. The standard Go tooling is beyond sufficient: it's hard to beat.
> I suspect the experience varies pretty widely based on what languages you use and what you’re trying to do with it.
I second GP's assessment that Bazel is pretty developer-hostile if you are not Google or have Google's resources to maintain Google's build systems.
Case in point: get Bazel to consume system libraries or vended/third-party dependencies in a C++ project. This is a pretty basic usecase, but it was left as an afterthought in Bazel, and it's an uphill battle just to get Bazel to work with them.
I don't doubt that Bazel can be usable if your entire world is locked tight in repos you control and were all forced to migrate to Bazel. It's the same thing with GYP, another developer-hostile build system spawned there. However, that is not how the world outside of Google works.
The only way I see Bazel become relevant and usable for non-Google/FANG-size orgs is if a higher level build system like CMake supports generating Bazel code, so that all these shortcomings are brushed under the proverbial rug. However, if we get to that, are we really using Bazel, or is Bazel relegated to an implementation detail?
The very fact that this occupation exists pushes me away from Bazel, having never used it. Makefiles are horrible, but they are well documented and I've been able to google my way out of every problem that I've either run into, inadvertently caused, or had to debug from somebody else's mess (it's always a mess).
I don't actually want a better build tool. I want a tool that I spend as little time as possible thinking about.
But that’s not really how it works. Build systems exist in a complex domain — whether we like it or not, that complexity can’t really be hidden away, it will leak one way or another.
The complexity need not "leak" - it could be an explicit feature of the tool. Make kinds sorta does this, as the learning curve is steep from the beginning but stays constant. Tools that are easier to use at first but then smack the user with "leaked complexity" set the user up for situations he is not experienced enough to handle.
That might be fine in an enterprise setting where the user likely has a mentor or support contract to turn to. But it's not what I want at home.
The problem with Bazel has nothing to do with learning curve. It's a pretty simple model as far as build systems go.
It's that it requires a lot of boilerplate and has a very rigid nested structure... which compounds boilerplate, unless you venture into custom plugins/build rules.
Some of the basic ideas are absolutely right, e.g. separating the resolution of dependencies from the build, and purity, but it's just soooo much boilerplate.
I think many of the ideas/techniques behind Bazel are excellent: hermiticity, distributed caching, dependency analysis, massively parallel execution, etc. There are so many things every other build system out there can learn and apply from Bazel that could make them 10x more effective.
But IMHO Bazel's fundamental flaw is that it tries to be a one size fits all solution that's supposed to work for all languages and ecosystems. That pretty much guarantees that it will have more need for boilerplate/configuration, be less performant, have more bugs, and at the end of the day just offer a worse user experience than a build system that applies all of Bazel's underlying ideas/techniques while being designed solely for a single language/ecosystem and more deeply integrated into it.
I think the eventual best case scenario for Bazel is a jack of all trades, master of none, but in its current state, for most of the languages/ecosystems it claims to support, it would be way too generous to even call it a jack of the trade in terms of overall UX.
I see GNU Make as sitting on the opposite end of a spectrum than Git. Git is fascinating as a sophisticated, featureful system emerging from a simple fundamental concept (Merkle tree of file hashes), whereas Make is a mish-mash of concepts cobbled together to expose a particular set of user features (describe and execute a graph of executions in a factorized way).
Git emerged decades after other version control systems were first invented, and found a neat abstraction that covered the space.
That's why I have such hope for the BSalC paper and what can emerge from it.
I think not adding an arbitrary new declarative language is good design feature of redo. I think redo has other issues but personally redo with bash as the underlying glue works a lot better than make (in terms of maintaining more complete DAGs). Now I'm not saying it wouldn't be nice to get a declarative language which works WITH redo, but it would be important to keep it separate.
One thing redo is missing for me is support for building things in a different directory. It's a nice-to-have feature but it's difficult to envision how such a redo would work. Really overlays actually solve this problem, although a fuse based overlayfs would also be nice (so you can use it without needing to escalate privileges).
Have you given bazel a serious look? It hits quite a few of the points you mention. There's a steep learning curve and you kind of have to go all in with it, but once you're there it's quite nice.
Bazel is alright (read: generally better than the competition) as long as you're not doing something uncommon with your toolchain or hitting one of its many shortcomings, otherwise it can quickly become a nightmare. It doesn't even make it easy to access the darn executable you just compiled (!) which always baffles me—if I compiled a binary I don't want it hidden away in some random folder, I want it right there so I can distribute it. And with Python, etc. you hit more obstacles. Make degrades more "gradually" in a sense: it's easy to do arbitrary things in a 1-line recipe, but then it becomes harder to generalize and make it modular. Overall, Bazel definitely wins as the scale of your project grows, and it has tons of useful features, but it's got some ways to go before I'd regard it as "nice".
Bazel does not put outputs in random folders, it puts them in folders that are a deterministic mapping from the build target plus relative path to the thing being built in the source directory. It is well-suited if you have a huge amount of source code with things being output, but one among many speed bumps when getting started.
(also depending on what you're doing, `bazel run <foo>` will invoke the binary directly, and `bazel build <foo>` will, if there's an output artifact, usually print its location)
I skimmed the paper. I use Make, extensively, but in a way I generally don’t see in the wild (for C/++):
1. I convert all paths to abspaths;
2. All intermediate products go into a temp dir;
3. All recipes are functions;
4. The set of objects are defined by a “find . -name ‘…’” for C sources;
5. Every C file depends on every header file in the repo;
6. Use ccache; and,
7. Deterministic build.
My big work project is ~20mm loc, including ~65 FW images (embedded OSes). Incremental compilation is ~300ms. Clean rebuilds are 30-40s; scratch builds are 8m. (All of this on ~4 year old MBP.)
Obviously, there are some design tricks to get raw compilation speed up — naive C++ only compiles at ~3kloc/s; it’s possible to get that up to 100kloc/s.
> Every C file depends on every header file in the repo.
So change in single header file always becomes a full build? Do you use pimpl or other tricks to avoid frequent changes to h-files, otherwise class members usually end up in the headers for simplicity and to avoid incomplete type problems on non-pointers unfortunately. Was this C or C++?
What's the difference between a clean rebuild and a scratch build?
Yes: touching any header file causes a full rebuild. Remember that ccache is "under the hood" doing memoization. So, a clean rebuild could still hit ccache, whereas a scratch build starts from an emoty ccache.
"As an intensive (and reluctant) user of GNU Make, I keep looking for a more modern replacement."
Unless I am mistaken, the author of https://cr.yp.to/redo.html never implemented the idea. He has since used only the Bourne shell and standard UNIX utilties as a build system instead of make. For example, consider the "do" script in this project: https://nacl.cr.yp.to
The "do" script was written circa 2011. The idea for "redo" was published circa 2003. The idea for "make" dates back to 1976. GNU Make was written around 1989.
For the avoidance of doubt, I am not suggesting any of these build system ideas or implementations are better or worse than the others. That is for the reader to decide. I am only pointing out which one is the most recent, or most modern, if we use the word "modern" according to its dictionary definition.
I’m quite happy with ruby’s rake for my needs. I mostly use it to define pipelines, often involving data downloads and transformations into datasets for ML. It works well when you work with files, preferably containing a single or few entities each. Have been meaning to wrap some tools to make it handle databases into an extension.
I don't like using tabs in Makefile syntax. Tabs are in many cases (especially on print) indistinguishable from spaces.
Also, any "make" program for C should automatically detect dependencies between files because this information is present in the code and it doesn't make sense to duplicate it.
make doesn't really understand C or C dependencies, but the compilers usually do. Here is a minimal example that will take care of dependencies by passing -MD -MP to gcc or clang. The dependencies (*.d files) get generated as part of the normal gcc/clang invocations. There is no separate dependency generation step.
Actually, the compilers do not and you are doing less than half of the job there. They don't emit information about the non-existent files that would change the build if they were to suddenly exist, such as an "x.h" in one of several standard search directories in that example.
Also they don't add an empty rule for the source file (like they can do for headers), so if you rename from e.g. foo.c to foo.cpp, which still produces a foo.o, your incremental build fails because the dep file still has the dep on foo.c which doesn't exist anymore.
The original sin here is generating the dependencies from the source files in the same step that was used to generate the object files. The only way to ensure that Make observes structural changes in the dependency DAG is if the graph is rebuilt.
> Also, any "make" program for C should automatically detect dependencies between files because this information is present in the code and it doesn't make sense to duplicate it.
This requires that either you implement or shell out to a C/C++ parser as part of the dependency resolution step (which has to happen prior to the actual compilation). This will slow down your compilation a lot unless you're able to parse C/C++ as fast as your make/bazel/whatever language.
The idea behind redo, in contrast, is that "dependency resolution" is not a separate step. The dependencies can be generated at the same time as the compilation, by the compilation in a perfect world; through some post-processing in a less than perfect one.
Unfortunately, current C and C++ compilers do not emit all of the information that they possess internally, only less than half of it (the positive information about files found, not the larger amount of negative information about non-existent files that were searched and would change the build if they were to exist).
I guess my point is that in the case of some target a that depends on some target b existing (I'm not sure what the best example is here, I know that bazel takes advantage of this to parallelize compilation, but I'm not intimately familiar with c++ compilation internals, so it might be clearer with a generated file example).
You have a.cc that depends on b.cc, but some tool generates b.cc, so it may not exist (but `b.do` does). If I invoke `a.do` to compile `a.cc`, and don't directly declare b.cc as a dependency in the .do file, the theoretical compiler will resolve that it depends on b.cc, and note that that doesn't exist, and then what?
In the bazel world, since you define the dependency graph explicitly before invoking any compiler, you can't run into this issue, but in situations where you need to get dep info from the compiler...what happens with stuff like this?
Gradle is a good make replacement as well. The kotlin DSL is much more readable and easier to follow than makefiles and has a lot of additional features.
Except when stuff goes wrong. Then gradle is a right PITA because debuggability is essentially zero. And periodically a new version of gradle will break something non-obvious deep in your dependency tree.
> All the alternative build systems wisely choose to tackle only some of these features.
But why should all of these features be tackled by a single piece of software? Especially seeing how Make, in itself, has proven wildly insufficient already... what, 30 years ago, when GNU autotools came onto the scene?
More specifically, why do you find combinations such as meson + ninja, or CMake + ninja, or even CMake + make as superior to just-Make?
Please someone mention Tup and how it can be a reasonable build system. I’ve heard the Fuse dependency is not ideal, though I felt it had a nice UX experience with the Lua config.
Plus it’s worth considering that you can potentially use Fennel to configure the builds (since Fennel compiles to Lua).
I have encountered Tup in the past and could not figure out how to define a "generator". As in: a function that defines how some output can be generated from a certain input running multiple commands. I don't want to copy those commands for every input I need to process.
Edit: Generator is a term typically used in CMake and Meson for this, in Make I'd use a pattern rule mostly.
Not sure how to exactly to build a "generator" either, but it seems that it would be a build rule that generates multiple outputs from multiple inputs right? If that's the case there's a `foreach` function for convenience, but it doesn't seem to have something for multiple commands. Though there's this Lua example on their website that leaves me wondering if what you want is possible, here's the code (since the documentation seems offline atm):
If you want to have a way to specify rules that use the same command, but still specify the rules manually, look up "!macros" in `man tup`. If the issue is that the commands need to write and read some temp files, then note that you can write temp files in a tup command (they need to be placed in $TMPDIR iirc). Note that the tup command can be calling a script that you place alongside, in case this makes specifying your sequences of commands easier. You can define the macro in top-level Tuprules.tup and use include_rules in all the Tupfiles to get it included.
Thanks, the `!macro` thing seems to be what I was looking for. Having intermediary temporary files is of course fine.
The use-case at that time was to convert a LaTeX file containing a TIKZ image (therefore having a .tikz file extension) to an SVG which required compiling the LaTeX code to PDF, cropping the PDF and converting it to SVG. Since there were multiple occurrences of this across the project, I wanted to have that as a reusable "function" in the build system. I did not want to write a dedicated script for this because of portability issues (Shell vs. Batch).
How does !macro rid you of that portability issue? One way or another, you'd need to have a command that does that whole sequence of operations. If it's inline, I'd expect same portability problem (`&&` in Bash corresponds to something else in `.bat`).
What you could do is use Tup's LUA support (you can write tupfiles and tuprules in LUA, and you can define and use LUA functions in them): have each rule handle just one step of your process, generate the whole necessary DAG of rules, encapsulate all of that in a LUA function defined in tuprules.lua and call that function in tupfile.lua for every diagram you want to generate. One thing that you lose is the ability to use `foreach` with your construction (other than by creating a variant of your constructions that embeds foreachness, or by using tup.glob and iterating).
Chris Forno (jekor) made an implementation of djb's redo in Haskell. The videos of him live coding it are on YouTube, and I recommended it for anyone wanting to learn Haskell.
Not impressed by shell incantations. What would sell such a tool to me is a feature to replace those with new and more intuitive syntax. And a terser one, too.
Holding on to how things are done in the shell is not a thing to be proud of. I think a lot of us around here stopped counting the times we got tripped by globbing, forgetting or misplacing one special character in a ${} block, or quoting.
Let those monstrosities die already. Please.
There's this tool -- https://github.com/ejholmes/walk -- that is pretty good and I liked it but dropped it for the same reasons: it leaves the heavy lifting to you and it depends on your mastery in the black arts.
Now obviously I'm not managing huge projects but nowadays https://github.com/casey/just serves me just fine for everything I need.
Nobody forces you to use any kind of shell with redo. Its .do files can be anything you want: ELF object files, shell scripts, Perl, Python, whatever, just make it executable and obey trivial simple rules (stdout and three arguments).
And again noone forces you to use multiple files too :-). You can literally have just single default.do file for the whole project. apenwarr/redo somewhere explicitly noted that. Separate .do files are only for convenience and ability to automatically depend on target's build rules independently from others, that Make just do not do at all.
Did you try https://pydoit.org/, it's a build tool in similar spirit, which allows the actions to be either python-functions or external programs. Try to look past their tutorial 1, I don't know why they mixed in module_imports there, making it look a lot more complicated than what it really is.
Nice idea, I like it. I have no use for Python functions however, and the startup time of the interpreter will likely get in the way since I need a tool I can invoke many times in scripts as well.
Will try it for sure though, thank you for the link.
Just like tup, redo keeps popping up as a nice conceptual thing, but none of these take off in practice.
For most projects the winning build system is not the most conceptually interesting one, nor the most flexible one. It's the most "convention over configuration" one. This is why Meson has taken the freedesktop/gnome/etc world by storm.
It's not hard to "take over the freedesktop/gnome" world by storm since it's basically the same small group of people who have any power in that world.
File system time stamps are an unreliable way to decide if a target needs to be rebuilt. It doesn't work with distributed builds, it doesn't allow instant rebuilds to help with bisections, various file systems might not have not store the same time attributes, even on the same OS, etc. In my experience (large, mono-repo builds) hashes are the way to go and offer the most portable way to map an artefact to a tree of dependencies.
Another problem which is not discussed and is a very important feature for a modern build engine is "sandboxing" - forcing a build rule to declare all inputs, all outputs and to prevent any other kind of FS access and network access during the run of that command. BuildXL from Microsoft does that. Without sandboxing it's pretty hard to enforce a sane set of rules for building your large project.
The sandboxing restriction is one that merits care for (or not supporting) some use cases. Tup requires declaring all outputs (but not all inputs), which can be inconvenient when a build process creates intermediate or derived files that are tedious to anticipate. [0]
Using filesystem timestamps is a misfeature of THIS particular redo implementation. Almost all the other redo implementations I know of (which are generally also much faster than apenwarr's redo) use a hash if the timestamp has changed.
Also, redo only lets a given .do file produce ONE output from a set of specified inputs. If you want to enforce this, nothing stops you from writing some declarative tool on top of redo (meaning you would replace the shebangs) which implements the sandboxing. Other than that, it's outside of the scope of concerns of a tool such as redo, and adding it would be a misfeature in my opinion (since it's not directly related to handling of the dependency graph).
1. How does redo compare with lower-level build tools such as ninja?
2. Why is it important/worthwhile to create a simplified/elegant version of Make, when for a few decades already, it is customary to write something higher-level that generates Makefiles (or builds in ways other than through Make, like scons or again via ninja)?
3. Is there a redo generator for CMake, and does it have advantages over Makefile generation?
0. I would recommend you actually read about redo yourself. Redo is not just the implementation posted here (which personally I don't like for multiple reasons). Redo is a very simple idea envisioned by DJB a while ago which was never implemented by him properly. You can read more about it here: http://cr.yp.to/redo.html it won't take long.
1. Ninja is not meant to be hand written, redo can be hand written. Ninja verbosely describes your build as flat file containing the description of a DAG and the rules required to traverse it. Redo is a recursive definition, intended to be mostly hand written, which describes your build process. Redo can and should be split up across multiple files.
2. It may be customary to you, but not everyone agrees that the correct solution to building a project is to have a gigantic software suite consume a confusing and abstract configuration file and spit out a bunch of buggy makefiles. There are in fact plenty of people who hand write makefiles, redo fits in that same space. It's easier to hand write, it's easier to fully describe the DAG of your project, it's easier to reason about the build process and it's easier to make sure it is correct.
3. No. It would make no sense to have a redo generator for CMake as it would be severely missing the point of redo. The only reason CMake has generators for anything but ninja is because some people worry about portability to other systems and as such having make is better than nothing.
Not all redo implementations treat $2 the same. Specifically JDEBP's redo differs in that $2 is the extension rather than the part with the extension removed (it's unspecified what $2 is for non default*.do files).
I feel like DJB's ideas behind redo are actually some of his most poorly thought-out ideas. It seems that he thought, 'Make has <problem>. To solve <problem>, a new build system should have <solution>.' In other words, he seems to just used additive solutions instead of subtractive ones. [1]
Most build systems seem to be like that, adding features on top of features. The end result is usually complicated and finicky.
I think that's why build systems are among the software developers hate most. And I do mean hate. (The first time and only time I saw an acquaintance, a normally calm person, have an outburst was talking about a build system.)
I think the ideal build system will be subtractive, not additive.
So I guess this is as good a time as any to talk about my current project, a build system. :)
(Most of this will be copied from two comments I made on lobste.rs.)
First, there is a paper that already lays out the features needed for an optimal build system: A Sound and Optimal Incremental Build System with Dynamic Dependencies. [2] The features are:
1. Dynamic dependencies.
2. Flexible "file stamps."
3. Targets and build scripts are Turing-compete.
That's it. Really.
The first item is needed based on the fact that dependencies can change based on the configuration needed for the build of a package. Say you have a package that needs libcurl, but only if users enable network features.
It is also needed to import targets from another build. I’ll use the libcurl example above. If your package’s build target has libcurl as a dependency, then it should be able to import libcurl’s build files and then continue the build making the dependencies of libcurl’s build target dependencies of your package’s build targets.
In other words, dynamic dependencies allow a build to properly import the builds of its dependencies.
The second item is the secret sauce and is, I believe, the greatest idea from the paper. The paper calls them “file stamps,” and I call them “stampers.” They are basically arbitrary code that returns a Boolean showing whether or not a target needs updating or not.
A Make-like target’s stampers would check if the file mtime is less than any of its dependencies. A more sophisticated one might check that any file attributes of a target’s dependencies have changed. Another might hash a file.
The third is needed because otherwise, you can’t express some builds, but tying it with dynamic dependencies is also the bridge between building in the large (package managers) and building in the small (“normal” build systems).
Why does this tie it all together? Well, first consider trying to implement a network-based caching system. In most build systems, it’s a special thing, but in a build system with the above the things, you just need write a target that:
1. Uses a custom stamper that checks the hash of a file, and if it is changed, checks the network for a cached built version of the new version of the file.
2. If such a cache version exists, make updating that target mean downloading the cache version; otherwise, make updating the target mean building it as normal.
Voila! Caching in the build system with no special code.
That, plus being able to import targets from other build files is what ties packages together and what allows the build system to tie package management and software building together.
I’ll leave it as an exercise to the reader to figure out how such a design could be used to implement a Nix-like package manager.
(By the way, the paper uses special code and a special algorithm for handling circular dependencies. I think this is a bad idea. I think this problem is neatly solved by being able to run arbitrary code. Just put mutually dependent targets into the same target, which means targets need to allow multiple outputs, and loop until they reach a fixed point.)
So how would this build system fit into the table (page 27, Table 2) of the "Build Systems a la Carte" paper?
It fits 8 of the 12 slots.
This is done with another feature: power-limiting. It will be possible to tell my build system to restrict itself, and any attempt to go beyond would result in an error. (This is useful in many situations, but especially to help new people in a team.) My build system can restrict itself along three axes: dependencies, stampers, and code.
To implement dynamic dependencies, you need a suspending scheduler, so obviously, my build system
The stampers are actually what defines the rebuilding strategy (and it can be different for each and every target), so there could be stampers for all of the rebuilding strategies.
This, technically, my build system could fill all four slots under the “Suspending” scheduler strategy in the far right column in table 2 on page 27.
In fact, packages will probably be build files that use deep constructive traces, thus making my build system act like Nix for packages, while in-project build files will use any of the other three strategies as appropriate. For example, a massive project run by Google would probably use “Constructive Traces” for caching and farming out to a build farm, medium projects would probably use “Verifying Traces” to ensure the flakiness of mtime didn’t cause unnecessary cleans, and small projects would use “Dirty Bit” because the build would be fast enough that flakiness wouldn’t matter.
This will be what makes my build system solve the problem of scaling from the smallest builds [3] to medium builds [4] to the biggest builds [5]. That is, if it actually does solve the scaling problem, which is a BIG “if”. I hope and think it will, but ideas are cheap; execution is everything.
Additionally, you can turn off dynamic dependencies, which would effectively make my build system use the “Topological” scheduler strategy. Combine that with the ability to fill all four rebuilder strategy slots, and my build system will be able to fill 8 out of the 12.
Filling the other four is not necessary because anything you can do with a “Restarting” scheduler you can do with a “Suspending” scheduler and Turing-complete code. (Just use a loop or a stamper recheck as necessary to simulate restarting.) And restarting can be more complicated to implement. So in essence, my build system will fill every category in the BSalC paper.
And I could do that because I took things away instead of adding more.
Aside from the fact that DJB barely specified redo to the point where you could make accusations claiming it did or did not do something, I think there are implementations of redo which are capable of fitting the requirements you specify.
1. Redo doesn't give you the tools to have dynamic dependencies for example, but it also doesn't concern itself with those and doesn't prevent you from having them. I have a small project which has dynamic dependencies (not system dependencies but just configuration options) and handles them gracefully with redo (without ever needing a make clean).
2. Flexible file stamps are something which you can implement on some redo implementation using a combination of redo-always and redo-stamp.
3. The targets and build scripts being turing complete is something which redo doesn't specify but that is the intentional use of redo.
That being said, I think there is a place for something even more BASIC than redo which can be used more directly to implement what you've described while also being flexible enough to build redo on top of. It has given me things to think about.
> Aside from the fact that DJB barely specified redo to the point where you could make accusations claiming it did or did not do something, I think there are implementations of redo which are capable of fitting the requirements you specify.
You're not wrong, and I think DJB did get there. However, there's a difference between the Unix way of designing a system out of disparate pieces working together (something DJB is good at and tends to do) and designing a cohesive whole system.
Obviously, redo is the former. My build system will be the latter, and I am doing it that way (despite being a fan of the Unix way in general) in an attempt to 1) hopefully make a build system that the masses actually like, 2) make it easier to make it work on Windows from the start, and 3) a cohesive whole build system is a much better platform for implementing hermetic builds.
(There are a bunch of smaller reasons, such as separating file targets and non-file targets, but all of those reasons are details.)
So yes, you are absolutely correct that redo can do everything I mentioned, and I guess that's an L for me. I should have mentioned other features as well.
> 1. Redo doesn't give you the tools to have dynamic dependencies for example, but it also doesn't concern itself with those and doesn't prevent you from having them. I have a small project which has dynamic dependencies (not system dependencies but just configuration options) and handles them gracefully with redo (without ever needing a make clean).
That's not quite what I envision when I say "dynamic dependencies" (my build system will also have configuration, but the config changing will not use dynamic dependencies), but yes.
However, at least in the TFA redo, dynamic dependencies (or simulating them) can be awkward. [1]
> 2. Flexible file stamps are something which you can implement on some redo implementation using a combination of redo-always and redo-stamp.
Correct. And I should have mentioned that. But again, DJB did that in an indirect way, making the system "finicky" (in my opinion).
> 3. The targets and build scripts being turing complete is something which redo doesn't specify but that is the intentional use of redo.
Yeah, no argument there.
> That being said, I think there is a place for something even more BASIC than redo which can be used more directly to implement what you've described while also being flexible enough to build redo on top of. It has given me things to think about.
Well, I don't think going even more basic is good here. Hermetic builds are just too good to pass up, and that requires some complexity.
>make it easier to make it work on Windows from the start
The problem with working on windows is that there's no shared standard (unless you count whatever irrelevant part of the POSIX spec that windows follows a shared standard). On windows there is no bash or posix shell, there is no gcc or a compiler which takes gcc-like options, there's no posix compliant or gnu compliant make. It's all just wrong. If you plan on making a build system, it has to take a lot of things and provide an abstraction layer over them to make them work for windows. I say forget it, it's a waste of effort, just make it work on linux and by extension (through WSL or mingw or whatever) you get windows. The alternative is worse as you have to add support for every special language to your build system. One of the best tests of a build system I've found has been to get a build system to build an executable and run it to produce an output which is then consumed as part of the build process. With pure make, this is purely awful and requires either generating a bunch of rules or lots of boilerplate, and generally it's easy to get wrong. With redo I got this working with relatively little work and got it to work with cross-builds.
> cohesive whole build system is a much better platform for implementing hermetic builds
In my adventures with redo, the biggest obstacle to hermetic builds that I've found aside from just the environment (env vars) has been compilers. Compilers suck at a: telling you what they're looking for and b: where they're looking for it. The best solution seems to be to strace gcc and look for calls to fstat or something like that to figure out which candidate header files it's looking for. And I hope you agree, that's an awful solution. Surely a better build system would need some form of overhaul to the interfaces provided by compilers? (Or, some re-implementation of some compiler features so you can spoon-feed a compiler the exact flags to ensure it behaves predictably even if the environment changes.)
I guess unifying things could solve the environment problem, but I think I can solve it without unifying anything.
>However, at least in the TFA redo
TFA?
>[1]
Okay, so you mean dynamically generated .do files. I think in most cases you can avoid them, but I agree that having every depend-er explicitly depend on the .do file before depending on the file it's intended to redo is really error prone and annoying. There's at least one way to retrofit this into normal redo: by having some default.o.redo file or something and wrapping redo-ifchange and checking if any dependencies have a default*.redo file and depending on the corresponding .do file, although kludgy this at least avoids the possibility of errors if you add future depend-ers to the dependency. I agree redo is missing something here though.
>Correct. And I should have mentioned that. But again, DJB did that in an indirect way, making the system "finicky" (in my opinion).
Why finnicky? In the usual case you DON'T want anything other than just timestamps and hashes, it works for all cases, albeit slowly for some. If you NEED to customize it, you have to pull out the special tools but then you just wrap that in some little utility and mostly forget about it.
> Well, I don't think going even more basic is good here. Hermetic builds are just too good to pass up, and that requires some complexity.
What do YOU think are the obstacles to hermetic builds? I already know what I've found but I'm always looking for more input.
I'm currently working with a few people to try to make redo work for strictly hermetic builds, or alternatively to design a better redo (or alternatively something else) which solves all the build problems I think exist. Unlike you, I don't really care if people like it or like using it, I've resigned myself to the fact that almost nobody cares about simple unix-esque software anymore and that if I want something like this, I'll have to design and write it myself.
Part of the language builtin to my build system provides ways of running commands without a shell.
> there is no gcc or a compiler which takes gcc-like options,
No, but there is MSVC. You just need to abstract the compiler and compiler flags. CMake did it, so why can't I?
> there's no posix compliant or gnu compliant make.
I don't need a make, and neither should redo. My build system will not be like CMake in that it will generate build files for something else; it will execute them itself. No make necessary. In fact, I could implement a make easily with my build system; just a parser would be necessary.
> If you plan on making a build system, it has to take a lot of things and provide an abstraction layer over them to make them work for windows. I say forget it, it's a waste of effort, just make it work on linux and by extension (through WSL or mingw or whatever) you get windows.
I don't think it's a waste of effort. People on Windows deserve a good build system too. In fact, not having a good one means that they are dependent on Microsoft. I'd like to help them reduce that dependence.
> One of the best tests of a build system I've found has been to get a build system to build an executable and run it to produce an output which is then consumed as part of the build process. With pure make, this is purely awful and requires either generating a bunch of rules or lots of boilerplate, and generally it's easy to get wrong. With redo I got this working with relatively little work and got it to work with cross-builds.
It's not hard to get make to do that. See [1] and [2].
> In my adventures with redo, the biggest obstacle to hermetic builds that I've found aside from just the environment (env vars) has been compilers.
Yes, but however it's done, it's still easiest in a self-contained system, IMO.
> Surely a better build system would need some form of overhaul to the interfaces provided by compilers? (Or, some re-implementation of some compiler features so you can spoon-feed a compiler the exact flags to ensure it behaves predictably even if the environment changes.)
The build system should carefully control the environment. In fact, it should make sure the environment is empty. (There are details, but in general...) That's the biggest thing to make sure the compiler does not act weird.
You should also do the build in a separate directory and copy needed files into the directory. Then check all commands and their arguments for possible files before running them. This is one reason why depending on an outside shell is not ideal.
Hermetic builds won't ever be perfect, but you can make it 80% of the way with only a few things.
> TFA?
"The Famous Article". It refers to the link under discussion on HN. In this case, it refers to the apenwarr redo.
> Okay, so you mean dynamically generated .do files.
Not just that. Those were just an example of how dynamic dependencies are already awkward in redo. There are countless other uses for dynamic dependencies. To give you a taste, my build system will actually be able to build a distributed system, i.e., it will be able to be used for DevOps deployment, and it will be easy.
Surely redo can as well, but as I've thought about how to do that, it seems awkward. Perhaps you can prove me wrong?
> Why finnicky? In the usual case you DON'T want anything other than just timestamps and hashes, it works for all cases, albeit slowly for some. If you NEED to customize it, you have to pull out the special tools but then you just wrap that in some little utility and mostly forget about it.
Finicky because as far as I can tell, the build database in redo is modified by multiple commands over a build. This can cause problems should a problem occur with the database mid-build. In a self-contained system, the database is in-memory during the entire build and only written out to disk once the build is done, which can then be used as a sort of "commit" of the build, marking it as finished.
There are other reasons, but that's the first that comes to mind.
> What do YOU think are the obstacles to hermetic builds? I already know what I've found but I'm always looking for more input.
I think a good paper to read would be the Dolstra thesis on Nix. [3] If you consider Nix a build system (not just a package manager), it goes further than even Bazel. The thesis should be a good place to start.
I didn't meant to imply that you or redo needs a make. Just that for something like autotools or cmake, there's a lot of abstraction required just to get things to work. Or you can bundle things or you can write your own things but the point is, it's a lot of work.
>I don't think it's a waste of effort. People on Windows deserve a good build system too. In fact, not having a good one means that they are dependent on Microsoft. I'd like to help them reduce that dependence.
Right, but as I point out, microsoft already took care of that by giving windows users WSL.
>It's not hard to get make to do that. See [1] and [2].
Yes I know what it takes, but if you don't think this is horrifically over-complicated then I don't know what to tell you. Instead of using implicit rules with target specific overrides you've made it slightly less boilerplaty by making it less scalable and expanding the entire build command for a specific target. I still think this is too much boilerplate and noise.
>Yes, but however it's done, it's still easiest in a self-contained system, IMO.
I think it may be easiest to implement (just like any complicated solution is easier to implement than a simple one) but I don't think it's a good goal for a build system. Complexity isn't free, even if it has a cheaper initial cost.
>The build system should carefully control the environment. In fact, it should make sure the environment is empty. (There are details, but in general...) That's the biggest thing to make sure the compiler does not act weird.
I disagree, you would also need to compile the compiler (because compilers like to take compile-time configuration). Moreover, you're going to have to have specific code for handling every different compiler with all its quirks, you're also going to have to spend a lot of time figuring out all the quirks. And you're going to have to spend a lot of time duplicating the code which implements those quirks.
Surely getting compilers to support some standard hermetic building facilities would be easier and better in the long run than maintaining a complex database of compiler quirks to handle for every specific compiler version etc?
> Hermetic builds won't ever be perfect, but you can make it 80% of the way with only a few things.
Yes but I can already get it 80% of the way with redo.
> Surely redo can as well, but as I've thought about how to do that, it seems awkward. Perhaps you can prove me wrong?
I'm still not sure we're both on the same page for what a "dynamic dependency" is but with redo I've solved the problem of having a dynamic .do file by simply depending on a dynamically generated .rc file: i.e a file containing sourcable shell variables (including bash arrays), generated using declare -p or things like the @Q parameter transformation.
>Finicky because as far as I can tell, the build database in redo is modified by multiple commands over a build. This can cause problems should a problem occur with the database mid-build. In a self-contained system, the database is in-memory during the entire build and only written out to disk once the build is done, which can then be used as a sort of "commit" of the build, marking it as finished.
There are no concurrency issues with the build database in any redo implementation worth it's salt and realistically there's nothing stopping a redo implementation from implementing its database the way you describe. It's also an implementation detail which I've never had to interact with. Not sure how that makes it finnicky.
> I think a good paper to read would be the Dolstra thesis on Nix. [3] If you consider Nix a build system (not just a package manager), it goes further than even Bazel. The thesis should be a good place to start.
From what I've seen, nix is too coarse grained for being a build system, which is where I've seen bezel recommended, but both of these fail to provide a general purpose build system which can be used outside a tightly controlled environment to build a piece of software hermetically using existing system dependencies. At the end of the day, I want to be able to build software with existing system dependencies provided by my package manager with the assurance that if I hit build, if those system dependencies have changed, the relevant parts of my code will be forced to rebuild and I will be told about it. I can see the value of totally hermetic builds, but a build system capable of solving the problem I describe should easily be extendable to solve the general problem of hermetic builds. In short, I want to be able to compile my code with my existing compiler, and if my compiler is updated, I want to be able to hit redo and have everything rebuilt correctly without worrying that something didn't get built.
At this point, it seems to me like you have already decided I am wrong, no matter what I say, and have no desire to accept what I say that is meant to help you. So I'll just leave the conversation where it is.
I will say, though, that if you're not hearing about my build system in a few years, you can safely assume that you were right and that I was wrong.
> At this point, it seems to me like you have already decided I am wrong, no matter what I say, and have no desire to accept what I say that is meant to help you. So I'll just leave the conversation where it is.
It seems a bit defeatist to say this. Nobody on the internet, or the world for that matter, (unless they're agreeable and are just pretending to agree with you) is going to agree with you after the exchange of a few messages.
I should point out that I never disagreed with your fundamental claim that it would be easier to make a hermetic build system IF it's all encompassing and complicated. It's for example why systemd is easier to write if it's all tightly coupled versus if it was trying to have minimal coupling with minimal simple APIs and minimal simple tools building it up.
It's easier to write complex and tightly coupled software, it's why everyone does it instead of following the unix philosophy. I was simply pointing out that the unix philosophy was successful (for a while) for a reason, not because it was easy to follow it, but because the difficulty of making things simple paid off in the long run.
Please read what I wrote, I never made any hard claims about you being wrong, I've been mostly asking questions to get you to elaborate your stance. The sheer fact that I'm willing to entertain such long-form conversation should be enough proof of the fact that I haven't completely mentally dismissed your arguments as outright wrong.
> I will say, though, that if you're not hearing about my build system in a few years, you can safely assume that you were right and that I was wrong.
I don't think popularity (which is what you appear to be aiming for) is a good metric of the quality of software. Especially in the modern day and age where one of the most popular methods of shipping cross platform software is to ship it with its own instance of chromium and to write it as a local webapp. I think this is at least because modern day humans have become a lot more impatient and a lot more focused on short term convenience over long term benefits. But this discussion is outside of the scope.
Make is really a complicated combination of these components:
- a dependency definition language with in-place evaluation and half-assed wildcards
- A functional language with extremely limited datatypes (effectively just space-separated strings)
- a clunky macro system
- A "distributed" topological execution engine (did you know Make can coordinate job slots across recursive executions of Make? It's impressive! And shouldn't have to exist!) [1]
All the alternative build systems wisely choose to tackle only some of these features. Redo is interesting in that it's a minimal system that leans on existing Unix tools to fill the gaps, but I can't say I'm impressed with using Shell as the scripting language, though it is arguably more useful than Make's home-grown language that too few people know. Edit: actually, redo's doesn't enforce Shell, it can be any executable. That's more interesting, maybe (flexibility also introduces complexity!)
The most interesting development in this space is the paper "Build Systems A La Carte" [2] by Mokhov, Mitchell, & Peyton Jones, who did a great job of breaking down build systems to their fundamental concepts, and along the way found one particular point of the design space that wasn't fulfilled (Table 2, p79:16). Unfortunately, I fear it'll be some years before something production-ready emerges from that research, (and I'm sure I'll be grumpy about some of their implementation details anyhow! Build systems do not a happy developer make)
[1] https://www.gnu.org/software/make/manual/html_node/Job-Slots...
[2] https://www.microsoft.com/en-us/research/uploads/prod/2018/0...