> Then one day, our designer broke the build in the middle of the night. Everyone came in the next day and couldn’t work until they figured out what had happened.
I've heard this [campfire?] story before. A bad commit happens and now no one can do any work. And I don't understand... Is it really that big of a deal to have to revert to an older commit or comment out the broken code until it gets resolved? Sure, I can see it being annoying, but not bringing an entire group of developers to a standstill. What am I missing?
Stateless code makes for boring programs, but stateful code can be difficult to roll back.
As a toy example, imagine a database with columns for `first name` and `last name` and an update that combined them into a single `full name` column. That's going to be tough to revert for users signing up with just their full names.
While that would certainly impact production, it would be a bit strange for development environments to be stateful in the same way. Those not immediately involved in dealing with the stateful issue seemingly should be able to continue with their work away from production systems just fine.
Although in this case the build broke because the last committer didn't have authorization to deploy to production. The "fix" was giving him permission to deploy in the future. It is not clear why the developers going about their regular day wouldn't naturally resolve the broken build.
It's not a big deal usually to revert a commit. Sometimes nobody has no idea it was a commit done in some other code repo in some other distant team. It's a problem of visibility; it just turns out the cause was a commit somewhere. Figuring that out can be non-obvious.
I actually saw this happen at Google: no one could pinpoint the cause of the broken build for a while, and then it turned out to be a UX designer who broke the build. He was apologetic, like McFunley's example. But unlike the example, this guy never made another commit.
> Then one day, our designer broke the build in the middle of the night. Everyone came in the next day and couldn’t work until they figured out what had happened.
I've heard this [campfire?] story before. A bad commit happens and now no one can do any work. And I don't understand... Is it really that big of a deal to have to revert to an older commit or comment out the broken code until it gets resolved? Sure, I can see it being annoying, but not bringing an entire group of developers to a standstill. What am I missing?