Bazel may be 1.0 and the internal abstractions of dag solving etc are rock solid from years of use inside Google, but the ecosystem around Bazel is quite bad at the moment. They've been breaking backwards compatibility constantly, so rules authors have struggled to keep up (hopefully done now that it's 1.0).
The reason the ecosystem maturity is so important for bazel is because its design encourages complete reimplementations. The rust packaging rules for bazel reimplement a subset of cargo's features from scratch. The docker rules reimplement image building from scratch (they don't actually use docker). The (Google maintained) python rules shell out to pip, but it's got a ton of problems, some packages are unbuildable, and it busts the cache constantly and redownloads everything.
If you're using C++ or Java, I can heartily recommend Bazel as an improvement to your current tool. If you're using anything else, hold off until the ecosystem is more mature.
I can believe all of these reimplementations are worth it for hermeticity, but I seriously doubt most shops need the extreme hermeticity Google does. It's a thing that really matters at gargantuan scale.
> They've been breaking backwards compatibility constantly
I maintain a ruleset and have been on top of Bazel upgrades for a few months. My experience hasn't been so bad. There definitely are a lot of deprecations, and some code changes required, but it was pretty manageable IMHO.
With that said, I agree that some stacks are more first class citizens than others. But I feel this is more due to some stacks' departure from the C++ style of build management than an issue of Bazel changes. At our company, the Go org has had a pretty good experience with migrating from Buck to Bazel. For our web org, the main challenges have been that Bazel doesn't really lend itself well to dealing with codebases where inputs and outputs co-exist in the same filesystem tree (and this is incredibly common in the web world with things like babel transpiled files, yarn offline caches, lerna, etc) or where the ecosystem has a bad relationship with symlink handling (e.g. jest, flow, webpack, --preserve-symlinks, etc)
> The reason the ecosystem maturity is so important for bazel is because its design encourages complete reimplementations.
My experience from before Bazel is that reimplementations were the way to go most times you had projects with multiple languages, because the alternatives lead you down dark paths—calling into a separate build system suffers from the same problems as recursive make.
What excites me about 1.0 is the possibility that the third-party rules can mature. Its not too bad if you only need a couple sets of third-party rules but my experience is like yours-when a new Bazel release came out, you had to figure out when to upgrade based on your dependencies on third-party rules. If you had several of these, it made upgrading Bazel infeasible.
Amazon’s build system handled this gracefully. Native build tool chains were always used, but wrapped in a relatively thin facade. As long as the buildtool could respond to a `release` target and put artifacts in a well known location it didn’t matter what the underlying implementation was.
The biggest drawback that I could see was all dependencies and tool chain needed to be modeled, but once they were, incremental builds, dependency rebuilds, etc. all Just Worked™.
Getting to that point requires a lot of upfront work, however, which can be hard to justify in smaller groups.
Can it do "bulletproof", repeatable incremental builds? I'm asking because I don't see how it would be able to, unless the toolchain is wired into your build system _at least_ as deeply as it is in Bazel.
Slightly OT, but Tup[1] handles this by sanitizing the environment and building in a FUSE filesystem to track reads and writes. It's bulletproof except for system header files (or more properly anything outside the root of your build, which usually is just system header files).
Nix[2] accomplishes bulletproof repeatable builds by sandboxing everything, but I don't believe incremental builds are a design goal (though if you modularize everything into different nix expressions you will in fact get annoyingly hard to use bulletproof incremental builds).
So bazel and brazil solve similar but very distinct problems. bazel is solving the problem of fast, repeatable, "bulletproof" builds for (very large) monolithic repos, whereas brazil is solving for fast, repeatable, "bulletproof" builds for (very large) collections of repositories or packages.
brazil is EXTREMELY agnostic to what happens within any given package build, with the associated tooling primarily focused on dependency artifact discovery. The (IMO) most important component of brazil is the version set, which is a versioned dependency closure that allows for one-off builds (so your auto-build application packages) as well as merges from upstream version sets (where library vendors release their code). they are the glue that makes the distributed, manyrepo software development model at Amazon feel safe and consistent.
If I had my way, I'd combine the single codebase UX of bazel with the multi-code base dependency tracking of brazil, because i think they solve complimentary problems extremely well.
Google just tests the living Jesus out of everything, and only versions a few core packages such as protobufs, grpc, and other packages used by pretty much everybody (this is called the "crust").
Everything else is tip of tree, and things are automatically re-tested using checked-in tests if they are affected by your changelist. You basically can't submit if you break anything. So in a way, Google doesn't need "versioning". Whatever is currently checked in is good to go.
Tests are required, of course, and a Google reviewer won't let you submit anything if your tests suck. This, obviously, precludes the use of such a set-up in, shall we say, "more agile" orgs which don't have good test coverage.
Blaze (at Google) is also not just a build system, but also an interface to a much larger distributed build and test backend, which lets you rebuild everything from the kernel upwards in seconds (by caching petabytes of build products at thousands of possible revisions), serves up source code views for developer workstations (code is not stored there either), and sustains the scale of distributed testing needed for this setup to work. As a result, nobody builds or tests on their own workstation, and there's close to zero (or maybe even zero, period) binaries checked into Google3 monorepo. If you need a Haskell compiler and nobody used it in a while, it'll be rebuilt from source and cached for future use. :-)
Fundamentally, I think Google got things very, very right with Blaze. Bazel is but a pale shadow of what Blaze is, but even in its present state it is better than most (all?) other build systems.
> Blaze (at Google) is also not just a build system, but also an interface to a much larger distributed build and test backend, which lets you rebuild everything from the kernel upwards in seconds (by caching petabytes of build products at thousands of possible revisions), serves up source code views for developer workstations (code is not stored there either), and sustains the scale of distributed testing needed for this setup to work. As a result, nobody builds or tests on their own workstation, and there's close to zero (or maybe even zero, period) binaries checked into Google3 monorepo. If you need a Haskell compiler and nobody used it in a while, it'll be rebuilt from source and cached for future use. :-)
The distributed, cached builds are effectively the same thing for Brazil (at the package level), with applications, libraries, system packages, all the way down to compilers being built from source. Brazil doesn't have the level of file system abstraction that Blaze/Bazel does, and prefers towards local development using overlays version sets for certain core build tools.
>Fundamentally, I think Google got things very, very right with Blaze. Bazel is but a pale shadow of what Blaze is, but even in its present state it is better than most (all?) other build systems.
I think Bazel gets things amazingly right for a single monorepo even w/o all of the fanciness that comes from internal systems and services, but the mechanisms for managing out of source dependencies is just flat out clunky. I suspect this is because Blaze never had to really solve that problem, and I think Brazil is a much better tool for solving it because it's what it had to do.
>> mechanisms for managing out of source dependencies is just flat out clunky
I'm not sure I agree. You can reference arbitrary, versioned out of tree deps via WORKSPACE file. You can pull from git at tag or revision, you can pull from cloud storage, maven, pypi, etc. Just about any realistic scenario is supported, and those that aren't yet are easily scriptable using a restricted python-like extension language.
Perforce effectively provided this interface as long as you stuck to packages within the same depot. Multi-package reviews and commits were very natural. The git migration eliminated that and moved the concern up to build time selection of versions. Not terrible, but definitely less convenient.
It would be neat if all build systems had a standardized way to return their dependency graph (like gcc -MD) so they can be used recursively by other build systems. I'm not aware of any that does and there's far from a standard which allows easy chaining of multiple different sub build systems. Which unfortunately either leads to recursive make, remodeling the dependencies in two places or forcing everything into the same build system.
Agree. I've been dealing with Python rules for Bazel at work and the default implementation of re-downloading everything with pip became a real pain real fast. Decided to generate the third party py_library with a separate tool instead and much happier for it. As an aside for those looking to start with Bazel, it's worth checking out the BazelBuild Slack -- https://slack.bazel.build/
What are the reasons to prefer Bazel to Maven or Gradle foe Java? I know it is supposed to be faster but on the other side last time i checked it it was much more complex and time consuming (expecially compared to Maven). Are there other good reasons?
I've worked with both and while Bazel definetely has a learning curve, mostly because is very different than Maven and Gradle, I actually find it considerably simpler than those two. Maven dependency management is really complex, and I'm tired of dealing with dependency hell. And Gradle doubles down on Maven complexity where you stop having declarative XML config and instead have actual Groovy/Kotlin programs.
So, please consider the difference between complex and unfamiliar.
Speaking as someone who just got bitten by Maven’s dependency resolution now only building broken artefacts. (Sub-dependency’s dependency is loose enough to pull in a version built with an incompatible JDK) I prefer Bazel
my organization's goals for java builds are consistency (we want to be able to build deterministic jars), reliability, and playing nicely with IDEs via tooling.
given that, as far as i can tell, there's nothing so compelling that it's worth switching over your project or organization. the tooling around maven is so good, and the inertia so strong (for my organization anyway), that there's no real incentive to switch.
we've found that non-maven developers grumble a lot when forced to use it but then find the system, while verbose, is really solid. we are certainly not willing to throw that all away to play with google's latest cool thing.
I grumbled a lot against Maven when my employer at the time decided to throw away our Ant build for Maven, considering that:
- We actually made good use of Ant tasks and macros for having convention over configuration build infrastructure
- Maven was still on 1.0 beta releases, with incomplete support for the plugins we required
However this was more than 10 years ago, and nowadays I am yet to find something that beats Maven eco-system, specially given that I don't suffer from XML allergy, rather it is my favorite configuration format.
I cry a little beat every time I have to wait for Android Gradle builds.
Guess what, the upcoming Android Summit is enjoying yet another Gradle performance improvements talk. They can't get enough of them.
I don't work in Java, but just so you know, you can define Maven deps in your WORKSPACE file, and Bazel will automatically download and cache them (and their dependencies) much like Maven would. The rest of the build is trivial to set up.
I think for Java projects Bazel is a very bad choice. Last time I've checked it was not even able to resolve dependencies transitively. And I doubt that it's faster than recent versions of Gradle.
For a mono-repo setup I find Bazel much better than those two. If you have few hundred projects or more, Maven is a pretty bad choice. Gradle is slightly better, but still much slower than Bazel.
But it's not just speed, Bazel does do true CI: you make a change in a a lib and then you can immediately compile and run tests on anything that depends on that directly or indirectly. I have not been able to do that with Gradle, forget about Maven.
Finally, if you use other languages, with shared artifact such as protobuf files, Bazel is simply amazing: change a proto file and recompile everything that depends on that on every language.
It's not an absolute, but a contextual thing. For a clean build, on a single machine, under normal circumstances, gradle will often outperform bazel, because bazel pays a cost for isolation, parallelism infrastructure, and trying to max out your machine. OTOH, the abi-based compilation-avoidance, higher sensitivity to caching opportunities, and parallelism can make incremental builds much much faster on Bazel. Add in test-result-caching for non-flaky tests, and it's really a solid win for builds that don't invalidate most of the build graph.
So it depends on (a) the shape of your graph, and (b) the nature of the change being built (e.g., does it invalidate a core library and all its downstream, or is it close to the app?).
The other thing is remote build execution - the high focus on hermetic builds and parallelism makes building on a cluster of build workers extremely powerful, reducing build times, reducing load on developer machines, etc. And there are out-of-the-box solutions, initially from google on GCP to let you do this yourself.
So... is bazel faster? Hell yeah, in some situations. Kinda in others. Not in some others. You need to think through what your development model is, your scale, how you construct (or want to construct) your build graphs, isolation of components, etc.
They're really excited about 1.0 for this reason, they're tired of breaking other people's builds too. The goal is to limit that with the 1.0 release, so expect this to get a lot better.
I follow some of the relevant repositories, and there has been a big effort over the last few months to get many many loose ends cleaned up, toward a stable set of tools that will change in a predictable cadence without undue breakage.
They’re underinvesting in their Python story in my opinion.
To my knowledge rules_python still doesn’t support pip3 installs, and from browsing Github, Twitter, and the Bazel Slack it seems everyone is reimplementing their own package management integration because the default is so undercooked.
Right now rules_pygen is the best open source option, but it isn’t fully featured and was broken by Bazel 0.28.
About the container stuff specifically, I think the different solution provided here versus off-the-shelf Docker is not merely to be compatible with Bazel. Rather it looks like, from having used it modestly, an intentionally different way of thinking about container builds, aiming toward different performance characteristics that scale etc.
I can't think of any reason you couldn't call the docker tools in a genrule() though, if you prefer.
Docker's build system is really strange. The build cache assumes that your dependency tree is linear, and it will invalidate the build cache if any file dependency changes, but a `RUN apt install foo` doesn't invalidate the cache--it's just assumed that every RUN statement is idempotent. Further, the Dockerfile's COPY is close to worthless--if you want to import all of your repo's Python packages, you have to put them all under a single directory or you have to copy them one by one (or you copy the whole repo and deal with the fact that every JS and README change will trigger a complete rebuild). Also, everything depends on a running Docker daemon and last I checked, the performance degraded if you tried to build multiple images in parallel.
Yeah, I actually like the speed with which it can build images. But it's a risk that it's an entirely different implementation. If you have a team of people to deal with maintenance, then it's worth it. If you don't, then when an incompatibility crops up, you're stuck until you can hack around it. Having a really well maintained docker ruleset is crucial if you don't have that team
> but the ecosystem around Bazel is quite bad at the moment
I've noticed this as well. I'd love to use Bazel for our Python 3 project, but as far as I can tell, the advertised Python 3 support doesn't actually work. There are several issues filed for this, but apparently no progress since 2018. Some people reported workarounds, but none that I could reproduce.
I believe the core is solid, but language support feels alpha at best.
Python support is very much alpha. We use rules_python and the “reported workarounds” include forking rules_python, reimplementing a Bazel pip package manager integration, and adding a few twists to your Python codebase to handle Bazel idiosyncrasies.
It’s dozens of hours of work, and even then you quite easily can find yourself not enjoying Bazel’s caching features because of pip wheel being non-deterministic. It’s been a hard road.
That makes sense. What doesn't make so much sense is that the documentation indicates that Python 3 is supported, but everything I've tried and everyone I've spoken to have indicated that it's just broken. In any case, I hope I can use it eventually.
Any idea if deterministic wheels is on the Python project's radar?
I haven't been super happy with the prioritisation of that issue in rules_python.
In our Java+Scala+Python monorepo we've got our Java+Scala tests caching so that part of our test suite gets a low as 2 minutes in CI. The Python part of our codebase is maybe 5-10% of the codebase but its tests take ~5 minutes in CI because of no caching.
The reason the ecosystem maturity is so important for bazel is because its design encourages complete reimplementations. The rust packaging rules for bazel reimplement a subset of cargo's features from scratch. The docker rules reimplement image building from scratch (they don't actually use docker). The (Google maintained) python rules shell out to pip, but it's got a ton of problems, some packages are unbuildable, and it busts the cache constantly and redownloads everything.
If you're using C++ or Java, I can heartily recommend Bazel as an improvement to your current tool. If you're using anything else, hold off until the ecosystem is more mature.
I can believe all of these reimplementations are worth it for hermeticity, but I seriously doubt most shops need the extreme hermeticity Google does. It's a thing that really matters at gargantuan scale.