But on earth, we have seen now several instances where teams have moved from polyrepo setup to a monorepo. Although "monorepo vs polyrepo" is always a debated topic, and it's hard to scale a monorepo, large companies like Stripe, Canva, Cruise, Doordash have been able to manage monorepos by building strong tooling and automation to handle the scale.
We're excited to announce "The Hangar", a community created specifically for software engineers working in developer experience space to come together, and learn from each other.
The Hangar already boasts a community of dev-productivity teams/ dev-experience teams from leading companies like Figma, Cruise, Square/Block, Slack, and Netflix. The group features open discussions around best practices, useful tools, relevant industry news and career tips.
At The Hangar, we host monthly “off-the-record” zoom hangouts where you'll find detailed insights on topics such as migrating to a self-hosted CI system or setting up Bazel correctly. This is a space where vetted experienced professionals can exchange ideas, share hard-earned wisdom, troubleshoot issues, and ultimately help each other in their projects and careers.
I've also had terrible experience posting on SO lately. Even though I have been posting on it for several years, they keep flagging my questions. Plus none of the questions these days get answered without a bounty. So, now I have to wait over 2+ days to get an answer.
Dev productivity metrics have come a long way from the time when one use to measure the number of lines of code written or the PRs created. I think Dora metrics is still a step up from that as it focuses on measuring team productivity vs individuals, and provides some indicator of the true bottlenecks for productivity.
For everyone who complains about monorepos, remember some of the most forward thinking engineering companies like Google and FB also use monorepos. All the arguments that people make in favor of polyrepo are making so because of lack of strong tooling for monorepos. That's also why Google and FB would not have scaled if they were using GitHub / GitLab but had to build their own. Also Google's original source control was built on top of perforce!
The issue with building a custom monorepo system that can handle Google's and Facebook's scale is that it fails to scale down, even to moderately large project and organizational size. It's expensive (think at least 7 figures opex) and not what most people should be doing.
git, for all its issues (and I'm a git-hater), scales down to an individual coder and scales up (with a lot of hacks, the hacks being used varying depending on whether you're taking a poly or mono approach) to companies that employ thousands of developers.
Polyrepo scales to thousands coders without anyone even noticing, and that's the beauty of it.
Just look at the size of node_modules in an average project. You stand on the shoulders of thousands of other engineers and you don't even think about it. That's your polyrepo at work.
Now imagine that every time a dependency wants ship a new version, their maintainers have to update all of the dependents. That's your monorepo.
It is quite obvious which one is more scalable.
The only real benefit monorepo has is that every dependency is always at it's latest version. But the cost to achieve that... let's just say there's a reason you mostly hear about monorepos from Google.
For anyone who complains about dictatorships, remember some of the most resource-abundant countries are dictatorships.
Those companies should be one of last places to look for good software development practices. They have absolutely no incentive to recognize their own mistakes.
a) There is no correlation between mono/poly repos and your ability to scale. There are many examples of successful companies using either approach.
b) As a general rule people should be cautious about adopting approaches and technologies from Google, Meta without a clear understanding of why they need it. What works at their scale doesn't always apply to smaller teams.
> remember some of the most forward thinking engineering companies like Google and FB also use monorepos
As a counterpoint though I’d say that the issues Google and FB face, particularly in terms of the sheer scale of the work they’re doing, is pretty unique.
Google literally invents programming languages for domains it feels needs them, I’m not about to blindly follow that practise either.
They way at least FB is using a monorepo is very different than any kind of monorepo most people imagine. It's not just about tooling, git itself could never handle it. I am all about using a monorepo but Google and FB having one is not the an argument for it.
IIRC originally neither mercurial nor git would scale enough, but mercurial was much more willing to accept scalability-related patches, so long as they didn’t harm the more common small-scale use-cases. After a couple of years of submitting patches for moderate improvements, meta wanted to make some more controversial large-scale changes like dropping support for sequential commit numbers, and ended up hard-forking and breaking compatibility to do that. That incompatible-but-better-performing fork then stayed internal for a while, before recently being released as Sapling.
As a bonus, part of the rewrite of the internal storage engine involved creating a storage engine abstraction layer, which in turn made it easy to add Git as a backend :D
GitHub actions has support for this, not sure about Circle, but I think they do iirc. Still doesn't help if the hosted service is down. If your hosted trigger connection is down, no amount of instances is going to help
That's very interesting feedback. We certainly don't have a way to force simulate failure.
A related capability we are working on is to also rerun the identified flaky tests X times so they pass. This depends on the capabilities of the test runner, so it will work with specific ones first (cypress, pytest, etc). That way you still make sure that flaky tests pass instead of supressing.
Curious, how often do you see a flaky test in your system? In my past experience at one of the mid-size startups, we used to get a new flaky test almost on a weekly basis in a monorepo. We started the process of actually flagging them as ignored (we created a separate tag for flaky tests), but later realized that the backlog of fixing flaky test never came down.
In another case observed, devs just got used to rerunning the entire suite (the flakiness here was about 10-20%)
Haha great point. Well from what we have learned from our users is "fixing" test typically end up with "delete most of them". Fixing tests can be time consuming effort.
Another way to think about it is, whether Flaky tests are worth keeping? At some point if the tests fail often, do these really add value. And we think - it does. If you are able to identify flakiness from real failure and reduce noise, you can still avoid real failures.
Wow. That works like really poor technical leadership. Fixing flaky tests (as opposed to deleting them) is indeed time consuming, but it is a far cheaper choice than getting to the point your test suite is untrustworthy.
There may be a point where the cost of ownership for a specific test exceeds its utility, but the way to resolve that is usually to reevaluate your code and supporting tests. Suppressing flaky tests seems a very unwise choice.
Perhaps under extreme circumstances and with unhealthy code bases there may be a case for this, but I struggle to imagine it.
That is a fair argument. Not all organizations have the bandwidth to measure and manage stability of builds. Some companies build internal tools / dev productivity team for this purpose. There are always right intentions to comment out the flaky test with the mindset of coming back to it, but it is also a very low priority item in most cases when you have to ship new features.
Fixing flaky tests can very commonly take longer than writing new tests.
But on earth, we have seen now several instances where teams have moved from polyrepo setup to a monorepo. Although "monorepo vs polyrepo" is always a debated topic, and it's hard to scale a monorepo, large companies like Stripe, Canva, Cruise, Doordash have been able to manage monorepos by building strong tooling and automation to handle the scale.