I've gone back and forth on this. Yes - sometimes your worry is completely unfounded, but this article lost me here:
>Untangling messy parts of the codebase, identifying and removing dead code, cleaning up build, deploy and monitoring systems - these tasks often feel overwhelming because they can overwhelm you, sucking up days for no real gain.
Those aren't the risks I'm thinking about when I'm worried about problems like this. Difficulty/time is not the concern. Unknown-unknowns are the concern.
Upgrading our NodeJS version takes about 10 seconds with Elastic Beanstalk on AWS, but we're not on the latest version, even though it seems to run without issue and our tests pass. Why? Because maybe it'll expose some race case that would only appear in production. And worse even, maybe that race case will affect a lot of people sometimes in an extremely hard to reproduce way. Maybe it'll break something that isn't apparent for weeks.
We had a bug where some SMS notifications weren't being sent in production, and it was hurting our retention metrics for a couple months before one of our engineers stumbled across it. It was caused by a maintenance upgrade of one of our very few dependencies. In a small company or on a new product, you don't have the kind of bulletproof reporting to know when something like that is broken - your metrics are moving around quite a bit by default.
Code that works and has been working for ages without issue is code that I am not interested in changing unless I absolutely have to. Leaving/avoiding code that hasn't been changed in forever is sometimes the best option to ensure stability.
This all being said, I agree that this worry can be carried too far. Like I said - I've gone back and forth on this. Fundamentally it's a tricky line to walk.
It sounds like the grungy work you'd need to look into is a canary deploy with a production traffic duplicator. Spin up a version N+1 in AWS, copy all the traffic N is getting, only hook N+1 to mocks and observed behavior.
If your monitoring and alerting and quotas are setup right you'll know if the version update is ok.
To me that is very scary work though. Very difficult and risky and this article re-applies (at least to me).
You can also collect code coverage statistics from this canary deploy. I don't know if this is possible with EBS but I've done it with JVM apps where you can connect a debugger remotely. Keep running the canary until 100% of code paths are hit. If a code path isn't hit, find out why and repeat.
It's also a great way to empirically find dead code.
Trying to build something like this with Intel PT, and it's great. I used to do it with a patched libgcov, but now it's even better. Getting counters for every code branch, notifications after a new path has been taken once N% coverage is reached, liveness info about periodic tasks, I/O threads, execution times too.
I agree with the points made. I would add one caveat: The culture of your work environment has a huge bearing on these decisions.
Reminds me of something I read, like an illegal life tip; Don't fix an unknown problem because if it goes wrong people often won't understand or care about your good intentions and you will end up with blame (blame culture).
Whereas if a problem where to occur on production, and you were to fix it, well then you would be "hero".
Not trying to advocate for any particular way but just trying to reframe the question in more of a political/human way as opposed to an engineering one.
I find that I'm often advocating for incremental improvements when doing code-reviews. Knowing that a wider view will be taken anyway by me will change the attitude of my peers when approaching problems.
Another benefit of code-reviews is that they share the risk. So making a bigger change will feel safer to the individual.
> The question is if being humble will ultimately negatively impact your performance.
Interesting. When I'm humble, this means I don't think of myself as a rockstar ninja developer. I'm fallible just like the next guy. Right?
But my own aspirations to do great work are still there! So because I'm not that great, I work hard to get better.
Reflecting on the above, it seems right now I'm not humble enough, I'm too satisfied with myself. Thank you for giving me the chance to reflect and improve.
You did a great job at expanding on the question, but didn't answer it :)
I think the main problem with your argument is that the reason we do work is not always so clear. Sometimes our work is just pointless in the bigger scheme of things, but we still have to do it. It can be hard to find motivation from within yourself in this case, and it's easier to play the rockstar developer role.
Yup, this is the psychology I’ve seen at work, and it’s weird because my own psychology is the opposite. When I’m coding by myself or on a team with code ownership, I naturally seek 100% understanding of the code I’m working on and worry about anything less. I am drawn to the thorny parts of a codebase (that nobody understands because ten different people have been playing bug whack-a-mole) so I can clean them up. Just as if one corner of a room had mold or bugs in it I would want to clean that up upon moving into a group house.
Anxiety Reduction: I embed realtime monitoring in my code (high performance, specialized HTTP servers) using Statsd and Graphite (Graphana). I have to monitor 132 servers around the world all running the same software. Realtime stats and graphs completely removes a whole level of anxiety about "what is going on with my code". Actively measuring things like.
1) Transactions per second
2) Total time to process HTTP request and return response
3) Timing of specific sections of code
4) Exception counts
5) Specific function counts (get as detailed as you need here)
6) Response time as viewed from an external requester (with percentile breakdown)
7) Version numbers
8) Anything you feel is important to know about your software
I always deploy canaries and compare the stats of the canary box to the ones running version canary-1. Any difference in stats must be investigated and explained as valid before a new version can be deployed beyond canary.
The greatest tool I have ever created for regression testing, I call "A/B test". I record incoming network traffic, right off the wire using "ngrep". I have a tool that plays back this traffic to multiple different destinations and compares the response from each. Any difference in response between old version and new version must be explained and not caused by bugs. This tool also reads log files and other forms of output and does the same type of comparison. Obviously some things are always different like random numbers in responses, unique cookies, timestamps can differ by 1 second sometimes etc. Those things are removed before comparing.
This tool almost 100% guarantees no unintended change of behavior slips through a new release. The only way something could slip through here is if some odd request that wasn't tested causes it, or something in the data that we don't compare (due to it always being different like random number) causes something. I run 10 million requests through this A/B test and a "no unexplained difference" result is required before fully deploying.
This also assumes that "nightmarish tasks" don't involve things that could lose you your job. I've worked to dodge plenty of tasks, both personally and on a team, where I felt that taking on that task would set me or my team up for failure in ways that could cause me issues with my career.
There's plenty of "have this baby in a month" tasks that I don't want to become entangled in.
Why would they? What is a shitstorm anyway? Most people don't really care about the company... They care about feeding their family and shitstorms are only shit if you care right?
I'd rather be paid money to write code/do technical work instead of getting into bikeshedding fights with opinionated members of other teams.
There's plenty of tickets I put off doing because I know it will be hard to feel like I'm making progress, or frustrating to navigate the social component of.
Sure, maybe if I approached my work totally dead inside and with no expectation of enjoying it or caring about anyone else I worked with, I could probably just mechanically go down the list of tasks, but I suspect that would also cause me a lot of other mental problems on the way.
I've been doing this for a while now and I just can't get anything done if such anxiety takes over, although I'm a junior developer I'm often found in situations to make decisions like a senior dev.
And I don't know what to do about it, which makes it worse.
Honestly, this may not not be a failure on your part but perhaps on your manager. As an employer, I found myself guilty of this until I noticed that the reason junior devs on my team were underperforming was because of me.
There is a misconception particularly about startups (and I'll be the first to say I was living under the same rock) where they're seen as a magical box where you put a junior developer in on one end and they automatically turn into a seasoned vet on the other end. Yes, this may happen and you definitely will ramp up in terms of skills (both technical and people skills besides learning how to operate as a creative professional, etc) but it requires effort on your employers/managers part too—training, one on ones, feature planning, talking about how you would build before you actually do, what you're building and why, and how/why architectural design decisions were made, etc.
If you find your self out of your depth and in positions where you have to make senior dev decisions with no context or idea how to make them and why you choose the outcome, your management has failed you and perhaps themselves as the outcome will likely harm the team/company more than it will harm you.
>Untangling messy parts of the codebase, identifying and removing dead code, cleaning up build, deploy and monitoring systems - these tasks often feel overwhelming because they can overwhelm you, sucking up days for no real gain.
Those aren't the risks I'm thinking about when I'm worried about problems like this. Difficulty/time is not the concern. Unknown-unknowns are the concern.
Upgrading our NodeJS version takes about 10 seconds with Elastic Beanstalk on AWS, but we're not on the latest version, even though it seems to run without issue and our tests pass. Why? Because maybe it'll expose some race case that would only appear in production. And worse even, maybe that race case will affect a lot of people sometimes in an extremely hard to reproduce way. Maybe it'll break something that isn't apparent for weeks.
We had a bug where some SMS notifications weren't being sent in production, and it was hurting our retention metrics for a couple months before one of our engineers stumbled across it. It was caused by a maintenance upgrade of one of our very few dependencies. In a small company or on a new product, you don't have the kind of bulletproof reporting to know when something like that is broken - your metrics are moving around quite a bit by default.
Code that works and has been working for ages without issue is code that I am not interested in changing unless I absolutely have to. Leaving/avoiding code that hasn't been changed in forever is sometimes the best option to ensure stability.
This all being said, I agree that this worry can be carried too far. Like I said - I've gone back and forth on this. Fundamentally it's a tricky line to walk.