High level architects should focus primarily on creating a system architecture that is designed to incrementally and composably add functionality.
This often means adopting some sort of overarching architecture that nudges people towards composable implementations. For write-heavy, highly stateful systems with complex business logic, this means using something like a workflow engine where you can simply declaratively add tasks and conditions to pre-existing DAGs while being confident the existing workflow will not fundamentally change. To create new functionality, it's often enough to use existing functionality as a drop-in template. Duplicate and add / remove - no need to worry about what's there because you're not touching it.
For read heavy systems thinking about composition at the very highest levels is also very important. This allows engineers to easily add functionality without being concerned about breaking what's already there.
Always favor additive models over models where updating existing functionality is the norm. This means junior engineers can "color within the lines" so to speak, and the risk to the rest of the system is low.
Would also emphasize that while unit testing is often useful for the individual developer working on a piece of code, but as far as ROI, end-to-end integration testing has the most bang for the buck. If you have the full endpoint tested, for instance, from end-to-end, including database writes, your confidence level goes up by an order of magnitude when you have to modify that functionality. If you have to choose between investing in extensive unit testing and extensive end-to-end API or contract tests, always choose the latter.
> Would also emphasize that while unit testing is often useful for the individual developer working on a piece of code, but as far as ROI, end-to-end integration testing has the most bang for the buck. If you have the full endpoint tested, for instance, from end-to-end, including database writes, your confidence level goes up by an order of magnitude when you have to modify that functionality. If you have to choose between investing in extensive unit testing and extensive end-to-end API or contract tests, always choose the latter.
It's not so clear-cut, unfortunately. Extensive end-to-end tests can be very hard to setup, can be flakey, can take forever to run (especially if you do it on every change), etc.
I agree you should have some layer of automated end-to-end testing (and not enough people do), but in the end, I think you have to work towards making the system as compositional as possible, so you can also test things extensively in isolation - the end-to-end tests then serve to make certain that the individual pieces fit together, but don't hit all the edge cases.
I can think of few better investments than to have a reliable suite of end-to-end tests running as part of your merge pipeline even if they're difficult to set up. Sleep quality improves once you have this in place. Your tests don't have to be brittle or flaky. If your tests are brittle it is definitely a good investment to fix whatever's causing the brittleness rather than accept it as a fact of life.
Having the tests run against an isolated data and infrastructure environment without additional noise from shared activity is a good first step.
Distributed systems automatically bring challenges that makes it very hard to create reliable test suites. You can reduce the flakey-ness, but I don't think you can ever completely eliminate it.
Beyond that, you haven't addressed the fact that a comprehensive end-to-end test suite in a complex system is really, really slow.
Just to clarify, I'm talking about integration testing the service itself - posting a payload, saving to the database, producing messages to a mock queue, etc. Test the entire service in isolation from other services and validate that it is behaving correctly. Not end to end across services. You should be mocking out all service dependencies and testing against the contracts for those systems.
Our API tests run flawlessly every time because they write against an isolated database with well-defined endpoint and messaging contracts. They also execute all remote operations against a mock API that conforms to those contracts. This is perfectly achievable.
I agree that this is achievable but it seems to contradict your original assertion. Mocking out other services doesn't give you the same assurances as end to end testing the whole stack.
To me it's a classic tradeoff: the more you integrate in your tests, the more meaningful they are - but also the harder to write and maintain.
This often means adopting some sort of overarching architecture that nudges people towards composable implementations. For write-heavy, highly stateful systems with complex business logic, this means using something like a workflow engine where you can simply declaratively add tasks and conditions to pre-existing DAGs while being confident the existing workflow will not fundamentally change. To create new functionality, it's often enough to use existing functionality as a drop-in template. Duplicate and add / remove - no need to worry about what's there because you're not touching it.
For read heavy systems thinking about composition at the very highest levels is also very important. This allows engineers to easily add functionality without being concerned about breaking what's already there.
Always favor additive models over models where updating existing functionality is the norm. This means junior engineers can "color within the lines" so to speak, and the risk to the rest of the system is low.
Would also emphasize that while unit testing is often useful for the individual developer working on a piece of code, but as far as ROI, end-to-end integration testing has the most bang for the buck. If you have the full endpoint tested, for instance, from end-to-end, including database writes, your confidence level goes up by an order of magnitude when you have to modify that functionality. If you have to choose between investing in extensive unit testing and extensive end-to-end API or contract tests, always choose the latter.