My rule of thumb: tests cast away fear. Whenever I get that sinking feeling that I'll break things when I change the code, I write tests until my doubts disappear. It works every time.
In response to the article: it's true that "you should very rarely have to change tests when you refactor code", however, most of the time, most coders are changing the expected behavior of the code due to changed or added requirements, which is not refactoring. Tests should change when the requirements change, of course. I am not contradicting the article, only clarifying that the quoted statement does not apply outside refactoring. (Refactoring is improving the design or performance of code without changing its required behavior.)
In my experience, a lot of unit tests are written with mocks, expectations, and way too much knowledge of the implementation details, which leads to broken tests simply by refactoring, even if the behavior does not change at all.
If you test units in isolation, while adhering to SRP which results in many smaller units depending on eachother to do a task, then simply refactoring without changing behavior screws up a considerable portion of your tests.
As for "tests cast away fear", that is definitely true. Whether or not the lack of fear is warranted is something else, and depends heavily on the quality of the unit tests. I've seen plenty of devs confident of their change because it didn't break any unit tests, only to discover that it broke something they forgot to test.
Most 100% code coverage unit tests break liskov substitution / interface boundaries.
But I have seen how really deep unit tests enable rapid deployment of code to prod since it produces high likelihood of correctness.
However, refactoring and the LoC bloat is also gigantic.
And we still needed integration tests, and forget about unit tests if there is a lot of network boundaries and concurrency. It might help, but it starts to fail quickly. The facades/mocks just assume too much.
If you are in java, for god's sake use Spock/groovy even if your mainline code is java.
In my experience, if you are baking in implementation details into your unit tests, they are necessarily going to be fragile. It is best to focus on input and expected output rather than the inner workings of the function under test.
I find it helps when I consider the tests to be part of the code. If I need to change existing functionality, of course I'll need to change the tests so they test the new requirement(s). If I'm adding new functionality I add new tests to test the new requirement, but I shouldn't break any of the old tests when I do that. If I'm changing code with no requirements changes (as in, pure refactoring, not "tidying up as part of new feature development"), all the existing tests need to pass unchanged...
I always see unit tests as a snapshot of dynamic behavior by 'recording' the tested logic. I make sure I'm aware of any logic changes since that change could eventually break the desired result.
Whenever I'm vetting in-/output values to a given set of parameters, I do move from white box (unit tests) to grey box testing.
When I'm done with grey/white box tests, I do make sure integration works as expected.
Why all the hustle of moving through 'the onion'? I wanna make sure to detect misbehaving/unexpected logic as quickly as possible. Searching for malfunction detected while running integration tests takes way more time than already catching them at the onion's most inner layer (unit tests).
When the result of the function under test is nontrivial, but JSONable, I make it literally a recording: set your favorite JSON tool to pretty print and compare the output to the expectation in a string literal or file. Any change will be easy to inspect, diff viewers are great. The expected counterpart may or may not start out to be manually authored. After changes, it's usually much easier to inspect the diff and copy the approved version or edit a copy of partially correct output.
Yes, I do exactly the same to make sure the output stays the same. Snapshotting is awesome (for go, cupaloy works quite well [1] - jest for react is great) but does not give me the chance to 'record' the underlying logic dynamics. That's why I think unit tests are still an important tool at hand.
I prefer to speak about "confidence" but I agree with this point quite a bit, if I am making changes and am not certain if my changes will cause breakages I'll manually test things and codify those manual tests are integration/unit tests so that I never need to write them again. Then in the future I can modify code in the same neighborhood with confidence that any breakages I'd cause would be caught by my tests - add in a willingness to liberally add regression tests for any errors that do make it through and I think this approach can really decrease the labour required to make changes, but it does front-load more cost.
I agree static types can help, but there are many kinds of errors not caught by static typing. Did you misspell a key while encoding to JSON? Did you add two values where you should have subtracted? Did you forget to log an error (and did you log everything you should)? These are the kinds of errors I face every day and it would be burdensome to prevent them with static typing alone.
These days I can't imagine writing any kind of secure software without writing tests, regardless of static typing. Static typing does not increase my confidence in code very much.
Programming language tools help get rid of classes of bugs. GC languages help get rid of or reduce significantly classic security bugs such as buffer overflows, use after free and so on.
Static types get rid of other kinds of bugs and unit tests you don't have to write as a result. It, of course, does not solve everything or remove the need for tests.
Rust's borrowing system help get rid of multithreaded data race bugs, something traditionally hard to write any sort of fast unit tests for.
Other static analysis tools also help you not write tests by testing certain things on everything. Even linting is another kind of automated meta-testing tool.
Also for your json 'string typing' example, that is a hint maybe you should use statically defined models instead of relying on unit tests to catch typo bugs?
Not saying you are wrong, but I think your examples are all a bit interesting (in a good way):
Misspelling a key while encoding to JSON: In a system with JSON in it, the JSON part of the system I would consider to be a part with dynamic types. So, that errors can be introduced in a part of the system that is using something resembling dynamic types can be seen as the result of moving away from static types.
Adding two values where you should have subtracted: In general, this can't be detected by static typing, but there are also examples where it can (for example - try adding two pointers in C - this is a type error because only subtraction is supported)
Forgetting to log an error: Personally, I don't see this as practical at the moment, but if the popularity of rust means that linear typing because a bit more "normal" to people, then having errors where it is a type error to not use them is a possibility.
Personally, I'm a believer in using languages with simple static type systems, but pretending they have fancier type systems by the way of comments and asserts. Sure, a compiler can't check the invariant you mentioned in your comment, but with the language tech we have at the moment (and I would guess for the foreseeable future) a typical human can't understand a moderately complicated invariant that a compiler can check.
If you are parsing known JSON schemas, defining them as types instead of generic dictionary lookups makes this a type-error, not a implementation error.
I don’t think that’s painful at all. Yes you define it as a root class, with dependent classes as needed. This is the contract. It needs to be defined somehow, and IMO this is as good a way as any.
From there on you just use JSON.NET to convert your JSON-string to an instance of the root type, literally one line of code.
And after that you get code-completion and type-inference and compile-time checking for all data-object access.
Static types do complement testing quite nicely; and more involved sorts of verification, for those prepared to do it.
> I'd argue this is more cost effective than tests
It depends on when the types are introduced.
Choosing to write a project in a strong statically typed language might be quite low cost. This often depends on library availability, e.g. pick a language like Python and there might be off-the-shelf libraries to do most of the heavy lifting; pick, say, StandardML and this will probably need to be written in-house.
Trying to add static types to an existing project might be quite difficult, and may give us hardly any confidence in the code. For example, if we're adding optional/gradual types to our code, but that information keeps getting lost when passing data through an untyped third-party library.
It seems like a lot of the logic behind "tests shouldn't break due to a refactor" presumes that tests only test the publicly facing endpoints into the code. The tests that do the best job of reassuring me are tests against code in utility functions and the like. It's hardly a refactor if none of that changes.
However if you test too much of your "internal" code, you just end up cementing the currently implemented logic.
I often see tests that are basically checking if the code didn't change. I.e. checking if function calls (on injected (mock) objects) have been done in the exact specific order, instead of checking for some correct end result.
If you have a user facing component that uses utility functions, let's say a component that shows a table with a sum (that uses a utility function). Then refactoring your sum function should not break your tests of the table component.
I'm not saying you shouldn't test your sum function using unit tests, but ultimately users don't care so much about the sum function, they care that they have a table, and it should show a sum, and so that functionality should be tested.
The level of experience of the people writing (and maintaining) the code is also a factor I think. As other commenters have said it's all about risk reduction. I definitely agree that you can get a lot of value writing integration tests. At the same time I'm a slightly concerned that if I'd read this article when I was first starting out as a developer I'd have thought unit tests were a side-note or a chore, rather than the building blocks for an application.
Doesn't a lot of refactoring involve change in responsibility between classes? At least I find that a lot in my code. In those cases, the interface of those classes might change (as the responsibility shifts elsewhere), which will of course cause test changes.
My understanding is that refactoring may require changes to unit tests, but not integration tests. On the other hand, there have been long, unfruitful Internet discussions about the precise definition of refactoring, and I find it's better to just find an agreement among your immediate peers. :-)
There are some who treat the phrase "public interface" as meaning the parts of a class which use the `public` keyword.
There are other people who treat the phrase "public interface" as meaning whichever way that users interact with the system, e.g. commandline arguments, HTTP request parameters, exposed modules of a library, etc.
Sounds like you're using the first terminology and others are using the second.
Why are you testing how the responsibility is divided between classes?
Yes, some times you have to make an unclean cut into your code to test something. But this is not the default situation. That's why the article has that "mostly integration" part on its title.
I think his argument was that unit tests don't hold up to "real-world" refactorings, because most of the time you change the interface of your classes. Unit tests work nicely if you have one fixed interface and you just refactor some specific implementation of that.
In response to the article: it's true that "you should very rarely have to change tests when you refactor code", however, most of the time, most coders are changing the expected behavior of the code due to changed or added requirements, which is not refactoring. Tests should change when the requirements change, of course. I am not contradicting the article, only clarifying that the quoted statement does not apply outside refactoring. (Refactoring is improving the design or performance of code without changing its required behavior.)