Tests are good, I would even go as far to say I love tests.
However tests are often used or at least attempted to be used as a crutch for poor language/tooling/architecture decisions.
The path to code that breaks less is in good decision making (ideally early in the process) to ensure as much as possible the code you write is "obviously correct". Part of this is languages that protect you from silly mistakes, i.e statically compiled, strongly typed, etc and tools/libraries that help you write good code and make use of constructs that are easier to make "obviously correct". i.e sequential/blocking code styles vs async/callback styles. (obviously there is no such thing as "obviously correct" in an absolute sense, hence the air-quotes but some approximation thereof)
Similarly architecture plays a big role. If you concoct a pile of separate micro-services that then all need to be integrated tested together because you didn't settle on an IDL and contract between them etc then you are in for a world of hurt.
Some of it just comes down to principals also. Prefer idempotency. Try design around streaming systems instead of job queues if you can so you get replayability out of the box. Focus on data first, a good schema and using your DB correctly can save you thousands of lines of codes and all the tests that go with them by preventing bad states from ever possibly existing, etc.
If you don't give yourself a good foundation and try to "fix" it with tests and "process" you are doomed to a bad codebase that will just get worse with a test suite that seems to grow forever that either doesn't meaningfully reduce defect rate or if it does then it drastically reduces velocity.
The golden happy path has tests and process but only just enough of each to keep things in check, the good decisions are meant to carry most of the weight.
> make use of constructs that are easier to make "obviously correct". i.e sequential/blocking code styles vs async/callback styles.
I agree with the overall sentiment but I don't think this is a good example. If `await foo(); await bar(); await baz();` is subtly wrong, then `foo(); bar(); baz();` isn't obviously right. Continuation passing style is a minefield, of course, but callback hell is a historical artifact, not an intrinsic feature of asynchronous code.
Probably should have been more specific that I was talking about the API styles rather than the runtime. Namely callbacks and CSP which you mentioned but also rx which probably wasn't that popular outside of JVM land but still haunts me.
If you `await` every async call then sure, at that point you have made the code sequential within that logical thread of execution, if anything proving my point that it makes it easier to be sure it's correct.
I would say that once you aren't doing that, i.e doing real concurrency within a single "thread" of execution then that is going to be less obviously correct then simply not being concurrent.
That said, broadly speaking async/await is sequential/blocking enough for my taste.
There are runtime/language specific problems with it that reduce it's obvious correctness IMO (unhandled promise rejection when not immediately awaiting on Node for example) but it's not an API problem.
If you can unit test something as a unit then do it... the moment you put a MOCK in that code is waste.
That means to test your system you need to go end to end. API -> processing -> storage -> response.... Write the records and read them back. That also means your UI needs to do it to! JS keeps adding features in the browser, and none of them result in reliable testing. We need to do better.
But end to end testing is hard, our environment is complicated, it takes too long. Fix it (its fixable)! Bound your test system to run from nothing to complete in under 20 mins. Treat failed tests and long runs like production outages.
Write your end to end testing as stand alone. That means if you move your code base to a new language, your tests should still be valid and running.
Does this mean dont write unit tests? NO you should have some unit tests. Things that matrix out well (think email validation), are the softballs that should have some unit tests. Highly contentious code that business "logic" that is a dense ugly switch... unit test that, build the mocks, because if you have to touch that you want every confidence that it will pass your E2E tests.
Watch where you duplicate: Its the job of the front end to fully exercise its code. Its up to API devs to fully test their API's, to hit all their boundary conditions. Will there be duplication, sure, but its going to make clear where breakages are happening between layers.
> the moment you put a MOCK in that code is waste.
I disagree. I mock the interfaces to other components that have clearly defined output in a clean architecture to properly unit test (i.e. a use case). Everything else causes components to be mixed and things to be tested again and again, causing complex and glued together tests.
It all depends on proper architecture, which is hard to get right though. We're using kotlin (sealed classes ftw), hexagonal arch + ddd (use cases) and graphql/rest on top. Everything split up by gradle multi modules with architecture tests to verify that modules have not been mixed. We also have a domain module with plain kotlin that implements business logic free of implementation details. With this high separation of concerns unit testing is the only thing you need to do (automated).
Just a fun anecdote on end-to-end testing I did 20 years ago while working on the compression of images.
Some parts of the algorithm were easy to test, others very hard. On the easy part was loading and saving the file. I tried to test as best as I could, but the lossy part was not really tested.
Then time to benchmark the algo on some images.
The results were kind of OK, but some images compressed much better than others at low bit-rate, while the content did not look much different.
On square images, everything was fine.
On rectangular ones, at high-bit rates it was somewhat fine, but strange patterns like ghosts of another part on the image started to appear for lower and lower bit rates.
Of course, I searched on the mostly algorithmic parts what was going on, and failed.
Only after a holiday break and fresh eyes did I found the root cause: inversion of height and width in some part of the loading AND saving.
I had "end to end" test of the loading and saving, but the error was symmetric, so undetected. Young engineer error...
IMO/IME, mocks in unit tests can be extremely dangerous when used alone, in non-static-typed languages.
Changes in interfaces or behaviour of a module won't break the unit tests of modules that depend on it. So you must rely on either typechecking (not available in all languages) or integration tests (which will then trigger a change that will cause the unit-test-with-mocks to break).
If there is neither to help, you code just breaks in production.
Not only that, but this can cause some design ossification: classes that have too many dependents become very difficult to change, since you now also have to modify all mocks of it.
In Python one could use auto specs to be sure test would be broken on calling mocks with changed API[1]. I don't think it's hard to implement (if already not) similar feature for other dynamic languages.
I would almost agree, but what do you do about external systems? Take postgres for example - yes, you can easily set one up. But how do you test concurrency issues due to timing?
This can be necessary but very tricky to achieve without mocks/fakes or whatever it's called.
> But how do you test concurrency issues due to timing?
You don't.
1) The outcome of the tests will be:
0.1% There is an issue.
99.9% Maybe there's an issue.
2) Finding concurrency issues needs fine-grained targeting. You can't just write a "does anything race?" test. Testing won't reveal the location of the races. You already need to know about the race condition to be able to write a test for it. And once you know it's there, what are you testing? Are you making sure that the system does the wrong thing?
Say there is some weird behaviour and you suspect that it's due to a race condition. Maybe someone forgot to do "select for update" and you want to write a test to confirm. How to do that?
Even if you know 100% that it's a race condition, you still might want to add a test to prevent it from happening again no?
I've had great success with using testcontainers and a self-contained module interfacing with that. As we use hexagonal architecture, these are adapters.
We developers write code. That code can be application/library code or test code. The overall goal is to deliver the application/library.
As with most things in life, I find the best approach is seldom at an extreme. The time spent on writing tests should only be spent if those tests help you achieve the overall goal better than spending that time writing application/library code. And vice versa.
With that in mind I feel I get the most bang for my buck primarily writing higher-level tests like integration tests. These will test that the core functionality works, and cover larger parts of the code-base.
Like Lego, the pieces might be perfectly in spec and the error might be in how they're put together. Thus unit testing will only get you so far, and I mostly reserve unit testing for smaller pieces of library-style code, especially if there's tricky edge cases involved.
I would also add that I prefer higher-level integration tests because I want the tested code to be malleable in case I need to refactor in the future. To me unit tests take away some of this malleability.
I write unit tests for the same cases you do, however I also tend to write mostly unit tests for regression testing, which sorta breaks my own rule.
I've seen a lot of terrible tests over my career and some of the most egregious examples seem to have one thing in common: mocking frameworks. By definition the test code becomes very highly coupled to the implementation details of the code under test. Rather than being a "safety net" to enable fearless refactoring, they become a huge impediment.
I now lean towards only public API testing too, but still monitor test coverage. Sometimes this means I write only integration tests, but I'm fine with that. Test coverage is usually still very high. Tests are more meaningful, and usually simpler. If code can't be hit through the public API, then it's candidate for deletion.
So much of our profession seems to be built on unjustifiable dogma, and I think it's great we're starting to see some thoughtful challenges to this. (The recent pushback against thoughtlessly reaching for microservices as a default option is another good example).
Related, but writing a regression test for every little bug you fix isn't always worth the time either, like when the test is complex to write, the bug isn't likely to happen again, and the consequences of the bug aren't costly. It's rare to write a test to confirm a typo in some copy/text was fixed for example.
Sounds obvious, but I've been on teams where you're seen as a cowboy if you even try to discuss the downsides/costs of testing.
I agree that testing for a typo is silly, but the consequences of a repeated bug aren't technical, they are about customer service. If your customers see the same bug repeatedly occur on multiple releases, they lose trust in your product. That is a huge consequence. And one easily remedied by a test for that bug - sure, it offers little technical value, but is a solid guardrail against, "Didn't you guys just fix this??", for the same surface-level symptom that could be caused by multiple technical root causes.
After all, if a customer sees the same widget break the same way 3 releases in a row, the last thing they want to hear is: "Well, actually, that just looks like the same bug, but really..."
> If your customers see the same bug repeatedly occur on multiple releases, they lose trust in your product. That is a huge consequence.
I mentioned to weigh up how likely the bug will happen again and the consequences of that specific bug. A typo isn't likely to impact trust nearly as much as a crash.
In what scenarios do you not write tests? Have you been in a situation where writing the test would be a lot more work than it's worth?
I think you make the assumption that the person writing the test will be able to write a test that will trigger only when the same bug would happen. This could happen for some very narrow unit tests (as in the examples), but in real life there will be many tests involving multiple functions (to reproduce the same situation/code path) and then the tests can prevent other bugs.
(put it another way: if the programmer introduced a bug by mistake, probably by writing a test it will test more than wanted by mistake...)
> I think you make the assumption that the person writing the test will be able to write a test that will trigger only when the same bug would happen.
I'm aware of this, it still comes down to weighing up the costs vs benefits. It's not practical to write a test for every small behaviour of a complex app.
I am confused about the example with the 'Greeting' function breaking the test. Did the author see this as an issue with testing?
If the functionality is changed, the tests will have to be changed to reflect that. One feature of tests is that they are are documentation for the expected functionality of code, so naturally it needs to be updated when functionality changes. It is a feature in my opinion.
Yeah, the idea isn't that this type of code should never be tested but that it's a higher burden to test, and maybe less valuable to test, compared with other code. And that should be factored into the decision (and there should be a decision) on whether or not to test it
>Now, suppose that the product manager comes to us one day and says, "We have a new design, we'd now like it to say "Greetings Brandon, welcome to our website!"
I have been writing YAML based tests which rewrite themselves based upon program output in these cases.
Change the code -> run one script and now the test is updated. There is near zero test maintenance cost for surface level changes like this message or a tweaked JSON output from an API.
I call it snapshot test driven development.
I agree with the overall sentiment - testing ought to considered an investment that provides a recognized ROI.
There's a similar concept of "snapshots" in UI testing, which I maybe should have included in the post
I think snapshots definitely lower (but don't eliminate) the cost of a certain class of tests, and can sometimes tip the scales from "not worth it" to "worth it"
It's almost the same thing, I think. I do the same thing with text artefacts and screenshots.
There is a hidden cost to snapshot testing - theyre highly susceptible to flakiness. The tests have to be hermetic and the all the code has to be deterministic or it doesnt work.
In my experience preconditions are invaluable but fine grained post conditions within a function can get unwieldy really fast. I tend to move my post conditions outside to the caller. So, they end up in my calling modules and integration tests which call the logic directly.
So, responsibility for determining the correctness of the return value and what to do next (return error, retry, ..etc). falls to the caller.
That sounds wrong to me because then you will have duplications of the postconditions after each call. I haven't experienced any unwieldiness of having them at the end of a function.
It is more along the lines of when do you have post conditions that make more sense without the calling context.
I write mostly small pure functions. In those with proper preconditions it is impossible to have invalid output since the output for a given set of inputs is always the same. So, if the inputs are valid then the output must be valid as well.
Where I can’t write pure functions the post conditions are more along the lines of evaluating status codes and/or response bodies. Those I have found make more sense to allow the caller to evaluate and choose what to do in the context of the calling action. It isn’t that I am duplicating post conditions but that it is really better handled with more context as to what is being done since different callers may have different definitions of a correct response.
It is entirely possible though that in different domains or languages that post conditions make more sense. I am not saying their bad by any means. It just in what I work in currently I tend to just turn those post conditions into preconditions for the next action or gating some recovery logic.
But, I have tended to prefer flat call structures and keep retries and error handling logic at the application/service entry points if possible.
This is just a programmer musing about output checking. Testing is not merely output checking.
Programmers mis-using the word "test" is one of the great forces against good testing. If you have any intention to test your products, programmers, then you must investigate your products. This does not mean setting up an "optimal" number of automated output checks any more than doing police work begins and ends with setting up CCTV cameras.
Testing can be pretty contentious until you live through a few weeks or months of bad deployments and delays. As an application matures and complexity increases, you'll need to ensure that a benign change in one component doesn't cause something else to fail. The only way to do this is introducing testing to the ci/build process and letting the developer know what needs to be fixed before their work can be integrated.
Younger me hated tests. More experienced me sees tests as a necessary part of building high-quality software. Younger me didn't like tests because:
* Writing the test can be more difficult and time consuming than writing the feature you are testing.
* Tests redefine what "done" means for a new feature by adding "tests pass". That means it will take more time and attention to detail to finish.
* Finally, tests introduce a new constraint: you have to take tests into account when you change code and that slows down development, too.
All of the above said, there's definitely a sliding scale of too little/too much when it comes to tests that ranges on complexity and the developers working on the software.
As programmer you should be confident that your code works but for an business you need more than that, you need to prove the code and the assumptions you make with it so that it is safe to rely on.
Good programmers understand that tests allow them to move on safely to the next thing, as professional if you are not doing this you are creating technical debt.
It feels like your analysis is based on having to add tests without enough support from a skilled tester who is empowered to drive testing, if the tester is just told to rubber stamp code without delaying deployment they wont be able to do much. Thats not really the fault of the tester that is a choice to deprioritise test.
1) Figure out where the critical boundaries of your system lie. Typically they will coincide with business/commercial boundaries. For example this may be at the user/browser interface as they use your website/service. It may also be at an API that is exposed to customers for integration with their systems.
2) Understand what is on the other side of that boundary (human? machine?) and attempt to replicate as far as it practicable.
3) (optional, experts only) identify internal boundaries that may benefit from testing in this manner. Beware that artificially erecting such boundaries in inappropriate places may seriously compromise system development.
The thing about test cost isn't only an engineering question. If business critical functionality is broken customers will silently leave without converting and after awhile the engineer will be without a job.
The thoughts here are a bit too focused on the costs of testing. Although deeper questions are asked about your own judgement of whether to test something or not, it doesn't consider the benefits. I wrote about this the other day in another post:
- Test give the ability of other developers to be productive on the project faster. Having tests tells other developers the intended behaviour and notifies them when they have broken it.
- Flow on effect of slow or large test suites. It demotivates developers to write new tests, run existing tests and accept failures if there are too many. This then leads to a lack of confidence, trust deteriorates between team members and development pushes towards going faster than being stable. Eventually, once enough bugs occur or a large incident happens, then you go back to looking at tests again.
If you have fast and reliable test suites, a developer wants to run and add to them. Developers feel that this is a high quality project that needs to be well maintained. This culture then permeates into other areas of your business.
(Sorry for the self plug) This is why I created Data Caterer (https://github.com/data-catering/data-caterer) to try provide a fast and reliable tool to help with end to end testing.
Do we even have to read skepticism of tests, is it like a religion or some god you can see? Don’t believe in jeotests witnesses? Do you go back to retest the whole application manually on each deploy? Do you manually retests all past bugs in each deploy? What kind of cowboys are getting into software development
Skepticism doesn't mean throwing the baby out with the bathwater. Just because someone is skeptical of extreme points of view doesn't automatically put them in the other extreme. Quite the opposite, actually.
One should test all functionalities, happy paths unhappy paths and bugs, is that an extreme view? What should not be tested? What functionality has been naughty and doesn’t deserve test? What does it mean extreme view? How do you ensure correctness of naughty testless parts on deploy?
Also the article isn't talking about "testing functionalities", it briefly talks about code that is redundant to test in certain ways (eg: unit testing), not "functionalities". There's way more nuance to that.
An example: you could even have 100% coverage via integration or end-to-end tests, but still have zero unit tests (EDIT: or the inverse, you could have zero integration tests). The discussion is not just about the what but also about the how.
unit tests also work to ensure maintainability, decoupling, cohesion and cleanliness, other than correctness, so it's not redundant, I'd suggest to read something from Bob C. Martin or Martin Fowler instead of randomers through the internet to see what are the unredundant role of different kind of tests
it is more likely to easily write a unit test for a piece of code that is well architected and written using TDD and also to safely modify it, than to modify or write test for a 50 lines of code beast self-contained, undecoupled method
Writing unit tests should not take time, should be part of the time needed to write the method, if it's hard to write unit tests for a unit of code, then it's most likely that the code needs refactoring
So let me ask then, if there's 0% unit tests, how do you ensure maintainability, cleanliness, decoupling and cohesion of code? By indenting the assignment columns? :D
I don't think anyone in this discussion is claiming that unit tests are redundant. Nor is anyone arguing that we should have 0% unit tests.
Please don't interpret examples of extreme hypothetical scenarios as suggestions or recommendations.
EDIT: I guess the broader point of my comments is that you shouldn't assume that others are taking extreme positions. Again, as an example, it is also possible to have 100% coverage with only unit tests! But that's just an example of what's possible, not a recommendation or prescription. And I wouldn't recommend extremism in either direction. As always, "it depends".
I don't understand this idea of considering opinions extremist is coming from. I'd say, it's about needs, I think it's either needed and make sense to adopt a kind of tests or not, otherwise I stand by my past questions, if a unit test ensure maintainability etc. of a piece of code, what is the rationale for which it's not needed for a piece of code, if functional tests ensure correctness of behaviours, what's the rationale for which we don't need a functionality / piece of code ensured for correctness? I cover by unit / functional my code bases completely, because I value what they add to an application, and can't think of a piece of code that I don't need to be maintainable or a piece of functionality that doesn't need to be ensured to be functional, is that extremist? It depends on what, in your opinion?
EDIT: I can only think of reasons beyond technical, in case of limited time to market or tight deadlines, but in that case it's tech debt tracked by tickets and docs, still going to be subject to test-writing
Even in ideal conditions, it depends on the project, the language, the methodology, the architecture, on the stance on mocking, on the engineering philosophy used. It depends even on what a person considers a "unit test". For all I know, by "unit test" you might also mean what people call "integration test".
> So I think it's fair to say that (despite certain engineering books and management directives) the optimal number of tests isn't as many as we can possibly come up with.
Are there really (mainstream, well-regarded) books which promote the idea of endlessly ideating and writing tests? This isn't a narrative I'm aware of?
I have seen this sentiment in people (including my younger self) that have never felt the reassuring feeling of tests really having your back: the ability to make a change to a code base, and being certain the public API of the code continues to work as expected is an incredible benefit to maintainability.
Having a comprehensive test suite doesn’t mean the code is infallible and you won’t have to fix bugs, but with every bug a new test increases the future resilience of the code.
And finally, tests enable new contributors to work confidently from day one.
I agree that tests are a liability, but the amount of testing we do in software is still laughable in comparison to any other engineering discipline.
> And finally, tests enable new contributors to work confidently from day one.
Exactly this.
If I run into an issue when using OSS, I tend to try and look at contributing a fix back.
The projects where this is most successful are those with a good range of tests - I don't have the time to sit and learn the workings of a project inside out for the sake of a single fix, a good test suite helps reassure me (and them) that I've not inadvertently broken something.
That same benefit exists for new starters working on closed source codebases - they can hit the ground running much faster, confident that tests will help make sure they don't accidentally blow things up.
But, the OP is also right that tests need to be written in the correct way - built based upon the intent, rather than the code that was actually written (where I can, I tend to try and write a test first - even if I might later need to go back and tweak it)
Yeah, but it’s the type of tests that are often the problem. You need tests that cover the functional (and as much as possible, non-functional), requirements of each “component”.
In practice, we tend to have a pile of tests that helped the developer write the code, but now are simply redundant. Then you have all the tests that assert on internal details (strings in error messages anyone?).
My favorite guide to testing well is in “Large-Scale C++ Software Design” by John Lakos. The problem at its heart is one of engineering practices. Structuring code bases physically into definable components/modules, interacting only via defined api’s/interfaces and providing test suits that both document and verify the contracts. Done right, refactoring/rewriting involves no, or extremely minimal, changes to the tests.
It can be done right, but it requires significant discipline.
1) Give you confidence that if you refactor something, when you do something silly it'll break the tests and show you where you went wrong quickly
2) Allow you as a developer to write new things in isolation, so you can run the code without needing to jump through any hoops to see that it's correct
I tend to write tests when the thing I'm developing isn't simple/cheap to access. Writing tests should help you do things faster, as well as stopping future you from doing daft things. Writing tests dogmatically however, is a waste, and if tests are hard or onerous to write, it shows that the application is probably poorly designed and architected.
well said! doing large scale refactorings or invasive performance optimizations without a sound testing infrastructure is a huge pain. the more often you run into this, the more you'll appreciate tests going forward. but as always, one must not forget the pareto principle and related mantras - don't aim for 100% test coverage, roughly 80% coverage in 20% of the time will often give the biggest ROI.
Books can be written on testing itself. One article does not cut it.
You cannot provide easy and straight answer "how to write a good test", just as you cannot provide easy simple answer to "how to create good code".
It requires knowledge, practice and experience. It also requires case-by-case study. Not every problem is a nail.
- having said that I have written some pure unit tests, but more often I wrote functional tests, even for "units"
- shifting towards integration tests, may make your tests grow bigger in scope, in initialization, in result checking. There are always trade-offs
- to have seat belts in a car you also need to design them, and produce them. It is really beneficial to have them though, and you should not take into account scenarios without them
- more often I tests API
- they produce sense of security, but they have never ever produced certainty. No one said to me "everything works because we have tests", therefore this argument seems to by more hypothetical then real
- without tests I would have troubles with writing stable software, and I think I am not the only one
Then I suppose we can agree
- writing some tests is a required
- writing tests for critical code parts is required
This is really interesting. I am someone who does not see the value in over 60% of tests. Even less so if they are written by the person who wrote the code. Huge fragile technical debt. Many times you see developers mocking or stubbing functionality in such a way that the code behaves identically to the production code only through a different mechanism. So you have at least duplicated work. The best kind of test is your error logs and your user feedback.
I am also someone who, at interview, is a proponent of TDD - but in reality, and after landing the job - has never used it nor worked anywhere that has used it where they say they do.
With that in mind. How the fuck do you manage change requests with TDD.
Riddle me this… given:
We change the code behavior, and now our test fails! But the failure isn't because we introduced a bug, it's because we changed our mind. The new message is now the correct behavior of greet(), and so the test is now incorrect and needs to be updated. Time spent "fixing" the test is pure overhead.
So you have two test specs, since if I am new to a TDD codebase I won’t know which tests, if any, I should change for a given behaviour change. The old one that returns “Hello Brandon” and now a new one written by me, “Hello Brandon welcome to blah”. Now I do the TDD loops. Code fails, I write my code, new test passes but now old test fails? What do I do? I should never change my tests to satisfy the needs of my code.
Do a whole back and forth with whomever comes up with the specs?
Or
Edit my code so on the second invocation it presents the correct message, assuming tests are run in a consistent and deterministic manner this would work.
In this instance it would be pretty clear there is a wrong specification so I could go back and ask someone for clarity.
But what if it is not clear. What if it is some intricate or subtle behaviour change where you can’t use your intuition to figure out what is a correct test or not.
How do you even know if the tests that you have written correctly describe the desired behaviour? Pass them by the decision makers who are most likely non technical?
The only thing that important is how your end users interact with your services and what is a dealbreaker to them. Everything else is a nice to have. These can not be tested in any way other than having your application in their hands. This mo has served me very well in my career so far.
The value of testing comes from not when its created, but months/years down the line where a new feature/bugfix changes something completely unrelated to the initial feature.
Sometimes almost or even completely useless tests are written. Its a cost I can live with.
Tests are not about bugs. Tests are about code correctness. If applying tests, which I personally do while I write my production code, is catching bugs then that is of course a positive side effect. I write classic unit tests for the code where it's possible but do I have external dependencies I either mock them or stub them.
While writing tests alongside writing production code you might become aware of code that does not conform to your own qualitative measurements. SRP, DRY etc. That process, writing tests while you write porduction code, imo, has a tendency to better your design. But I never look at design as being done, I look at it as being good enough for the time being.
If you write test after you write you write your production code, bettering your design becomes more difficult, since the code is already written and you might need to make larger changes that will adhere to how you really wanted the code to look like.
Tests should give you confidence. Without them you are clueless and have no confidence when you start making changes. You cannot juggle with, on a cognitive level, that if you apply a change to your code, that whings will just work. Tests gives you that confidence because it gives you fast feedback.
You can not argue that removing dependencies and therefor mocking is a possibility. Unless of course you have no dependencies, like the calculatur or greetings funtion in the post.
You can move your dependencies around and perhaps invert them, but you cannot remove them. A good example is that instead of injecting external dependencies to your own classes, inject objects which represent a dependency in isolation.
Then you can control the isolated object which is a representation of an external dependency but do it without mocking.
I think the author makes some strong points. It's definitely easy to write bad tests.
I think the best defense of unit testing I could make in response to the article, is admitting that TDD makes it painful to write tightly coupled code, and that's the point.
If you want to write tightly coupled code (maybe a quick demo of a feature?), or keep the tightly coupled code you already have, don't test it!
The author gets into this a bit, but I'd say the reason they correctly identify testing a UI as painful, is that a lot of UI code is actually data, not behavior. One could argue that enforcing a clear distinction between the two is a benefit of TDD.
> TDD makes it painful to write tightly coupled code, and that's the point.
That's also the problem with TDD. Another word for "tightly coupled code" is "cohesive code". There's a tradeoff between coupling and cohesion. The problem with TDD is that it can make people decouple cohesive code just to make it testable. I have seen this many times where the only use of some abstraction/indirection was the test. These abstractions/decouplings then mount up to make the codebase much more difficult to understand and maintain than a simple cohesive codebase (with decouplings only where necessary).
Testing should also be the author's declaration to the world about what they believe should be happen and when. Good tests give you insight into the mind of the author at the time.
* When I add-to-cart, it should update the cart total
* When I add-to-cart, it should not decrease the available stock
* When I purchase, it should decrease the available stock
While these might be tests, they are also a form of live-code documentation about what the author intends. If you find later code that violates these expectations, you know that the old behaviors were intentional and you are updating expected behaviors that were not a side effect.
Code examples included into the article clearly show that author does not know much about how to write unit tests. Not surprised he hate doing it.
I've had such colleague during my career, who declared "who to write tests if I can write new features?". That will be a valid question if he could write code without tons of bugs. When he left the company velocity of the team increased.
That's really not what technical debt means, but yeah, you have to maintain tests. There's a cost.
> So what is the optimal number of tests?
NaN. There is no right number of tests. And more test cases does not mean "more coverage."
> while tests cover individual inputs
Depends on the kind of tests you write. When I'm doing property-based tests I like to randomly sample inside equivalence classes to test that I'm setting boundaries correctly. Picked up the habit when I saw that catching bugs when we moved from 32 bit to 64 bit code.
> The fibonacci sequence is well-defined, it has a clear definition, it will never change
Then you don't need automated tests, do you? Manually show that it works once then have automation that diffs that code to make sure the promised "never changing" really happens! Not what I would do, but different strokes for different folks . . .
And I wouldn't do it because the implementation of something non-trivial and not made up as a straw man might have a valid implementation change. Which is a great reason to write behavioral tests for the function.
> calculation()
The proposed tests don't test the interesting stuff. What type can n be? Can the function overflow? Does it slow down when one type is cast to another?
Also, why do you have a function that just returns n * 2 + 6? That's gotta upset those anti-clean-code folks you work with! Ugly high-maintenance tests can point to dev code smells.
> isEligible()
This function does something so trivial that if it were actually used, I bet it's not used everywhere for the check. This is the kind of thing where people use isEligible() in places and just hard-code checks against the age in others. And whenever the age changes, something will break.
Again, the tests seem dumb, but I bet there's a better way to write the production code.
> Is the test simpler than the implementation?
Seriously? If so, that might be a sign that the implementation is too complicated. Functional tests will typically have some kind of setup (possibly complicated) then the simple thing they're checking, in your ideally easy-to-understand function.
> Mocking adds a ton of complexity and reduces the amount of functionality actually being tested. It makes code both harder to test and less valuable to test.
Without knowing more about how you've done mocking and why you say that, I can only sound like one of those zealot snobs when I look down my nose and say "you're doing it wrong." I don't want to do that. So without a solid example here, I can't really respond.
FWIW, I mock to see how the results of different wire calls will affect the code. A small, well-defined set of results. I'd rather do this in unit testing with mocks than stand up the service and surround it with fakes because that's a lot slower and more brittle and I have to think more. And I'd rather stand it up surrounded with fakes than test in a real, full environment because that is slower still, even more brittle, and requires some kind of magical powers to debug.
> Integration tests
> almost by definition you aren't really mocking systems, you're testing the interactions between them that are otherwise hard to test
I bet you have never heard of contract tests. They're a faster, lighter, less brittle, less brain-hurty way of testing interactions between services. I swear I don't get a commission from Pact.io (a SmartBear acquisition), but you should check them out.
> UI tests
> Often change output/behavior intentionally (leading to flaky tests)
That's not what a flaky test is. If you change some code's behavior and don't change the tests and the tests break, they're good tests - fix them!
When a test passes . . . except sometimes it doesn't, so everyone says "run it again," that is flaky. And flakiness sucks. Often caused by not waiting for (or being triggered by, depending whether you're doing polling or push/promises/etc.) the right condition before taking the next action.
> have trivial logic that's easy to verify by looking at it
So, again, I assume you have tests that diff that section of code to verify the tacit promise that the code won't change.
> I don't hate tests (I promise, Avid!), I just try to question them on a case by case basis, and I haven't met a lot of people who do.
I'd suggest trying to understand the tests instead of questioning to prove they're not useful.
> They have a very real organizational cost
Yes. And so does bug fixing. The biggest cost of bug fixing, IMO, is finding the places where there were contradictory expectations. If only someone had coded those expectations as tests . . .
> makes that coverage number go up, so it feels good to add them.
That's a process smell. And I don't just mean the gamification. Code coverage is really important but not for rising numbers. What matters is what's not covered - there be bugs!
However tests are often used or at least attempted to be used as a crutch for poor language/tooling/architecture decisions.
The path to code that breaks less is in good decision making (ideally early in the process) to ensure as much as possible the code you write is "obviously correct". Part of this is languages that protect you from silly mistakes, i.e statically compiled, strongly typed, etc and tools/libraries that help you write good code and make use of constructs that are easier to make "obviously correct". i.e sequential/blocking code styles vs async/callback styles. (obviously there is no such thing as "obviously correct" in an absolute sense, hence the air-quotes but some approximation thereof)
Similarly architecture plays a big role. If you concoct a pile of separate micro-services that then all need to be integrated tested together because you didn't settle on an IDL and contract between them etc then you are in for a world of hurt.
Some of it just comes down to principals also. Prefer idempotency. Try design around streaming systems instead of job queues if you can so you get replayability out of the box. Focus on data first, a good schema and using your DB correctly can save you thousands of lines of codes and all the tests that go with them by preventing bad states from ever possibly existing, etc.
If you don't give yourself a good foundation and try to "fix" it with tests and "process" you are doomed to a bad codebase that will just get worse with a test suite that seems to grow forever that either doesn't meaningfully reduce defect rate or if it does then it drastically reduces velocity.
The golden happy path has tests and process but only just enough of each to keep things in check, the good decisions are meant to carry most of the weight.