Not sure if this story is true: Okta used to have a monolith architecture on AWS. All their 300 engineers used to work on the same service, so they had to run all their tests for every commit. The cost? $60 per commit.
When your system is critical enough, those kind of prices are cheap for what you get.
I can't tell you about Okta, but there used to be a (now apparently gone) article by Nelson Elhague about their test parallelization infrastructure. Many thousands of pretty slow test ran for every Jenkins build, massively parallelized across machines to get an acceptable response time. At the time he wrote this, there was no modularization to speak of, so every test really had to run for every build. Very expensive, but there's a lot of value on getting the equivalent of integration-level tests for almost every critical system automatically.
As a monorepo grows, the technologies used have to change to make its disadvantages spiral out of control: Serious modularization of codebases, something like Bazel to make sure code isn't overbuilt, and a testing strategy that finds the right balance of finding problems and keeping expenses in control. Every company with a monorepo ends up walking that road, but the good part is that by the time you have a problem, you definitely have proof that your organization is in a situation where the problem is worth solving.
This is very different from, say, what I saw at a company that had Fred George as CTO, and was going all in on microservices with one customer and about a dozen engineers, all of which wrote in the same JVM language. At that point, all the work that avoids running too many tests is probably going to be a net loss, and even more so when one considers opportunity costs.
And there you have it...$60 per commit means slowing down your commit velocity! Which means its longer before changes get tested or reviewed or deployed.
The cost (money and more importantly time) of running tests fundamentally changes the development process.
You only need to run CI on Production, and you can squash commits. Generally a release is once per week, and at Oktas level you have a release engineer.
300 developers' work being crammed together into a weekly release without having their contributions pass CI first sounds absolutely hellish for whatever poor bastard has to track down who broke the build.
There's tooling like bors that will rollup commits together and run CI on them before merging to master. If it fails then the commits will be retried individually.
I suppose so, but that cuts down on the number of CI runs when merging to master because the vast majority of commits being merged to master don't fail the build.
In a typical bors workflow, that holds, because you do a CI run before attempting to merge, so commits don't break the build except in the rare case that two commits that are individually okay combine in a way that breaks.
If you get rid of the individual run before merging, your combined runs will hardly ever pass, so it won't cut down on the number of runs.
Yes, I suppose I could be taking the anecdote too literally. It's much more likely they are only running all test on push to main / against pull requests / on code review.
Sure, $230K a year is on the high side, but Okta is headquartered in SF, so it is plausible.
I cannot find a documented source for overhead, but in private conversations multiple people told me that 100% is pretty reasonable amount, and often is even higher.
> so they had to run all their tests for every commit
Did they _have_ to? They could have run tests for affected transitive dependencies, cached build artefacts, randomly sampled integration tests on master. There's plenty they could have done to reduce the bill.
