Hi HackerNews,
After creating a Slack integration for Loqbooq, our decision log, we decided to make good use of all the code we wrote and implement a chat roulette app.
While there are already some Slack apps for this purpose, I think we hit a good spot between customization and usability and also a good price point.
There is a 30 day trial so any feedback is welcome!!
While the article delivers nice explanations for what kind of technical dept exist, it stays very generic and hand-wavy for the solutions. Bringing "microservices" into the room is definitely not helpful. Ownership and setting a quality bar..., sure but how does that look like? As I developer I maybe have an idea but I also need to sell that to management. As manager I have an idea too but I need to convince my developers to strive for quality and sustainability.
Sometimes I wish these articles would just state: Stop sprinting
Also: document your decisions! If you have to write down and justify your (prudent) technical dept, you maybe catch some stuff before it's written in code. It also makes sure more people are aware of it. Additionally, it sometimes prevents CV driven development, which imho is a big problem in tech. Overall, documentation is a very underused tool.
Finally: teach everyone (especially marketing) that software development is expensive, so everybody thinks twice before wishing for nice to have features done quick.
Could not agree more. The solution to tech debt is the one nobody wants to hear: Slow down!
Steve Jobs' presentation of Snow Leopard should be mandatory viewing for every manager. "No new features" to a standing ovation. Customers actually like stable, responsive, and efficient tools, who would have thought?
EDIT:
> Bringing "microservices" into the room is definitely not helpful
It's a pretty good way to increase technical debt. The microservices themselves might become small enough that it is easy to refactor some at a time, but then you have the actual deployment and communication infrastructure to contend with. And if you get that architecture wrong, oh boy you'll be begging to go back to a monolith in a day.
> It's a pretty good way to increase technical debt. The microservices themselves might become small enough that it is easy to refactor some at a time, but then you have the actual deployment and communication infrastructure to contend with. And if you get that architecture wrong, oh boy you'll be begging to go back to a monolith in a day.
I remember at a previous job, the monolith was drowning in technical debt. Someone decided the solution was Go microservices.
Fast-forward 18 months and 95% of the functionality is still in the monolith, but there are now 25 microservices, and no environment (except production) where you can test everything together.
> I remember at a previous job, the monolith was drowning in technical debt. Someone decided the solution was Go microservices. Fast-forward 18 months and 95% of the functionality is still in the monolith, but there are now 25 microservices, and no environment (except production) where you can test everything together.
I have the opposite anecdote. Currently working on a monolith that's been around for 5-7 years, huge enterprise Java mess, like many others in the industry. The clients decided that they'd like to upgrade to JDK 11, newer frameworks, all of that shiny stuff.
So for the past 4 months i've essentially been pulling my hair out and trying to rewrite significant parts of it all. When you have a monolith, you cannot upgrade the entire system if some parts of it break - even if i have, say, 200 dependencies but 10 break, i cannot move forwards with the updates and as a consequence am stuck with running on JDK 8 or even outdated frameworks.
And, of course, you cannot extract those parts of the system out either because you'll immediately be hounded by developers who aren't welcoming of change and will find nitpicky stuff to tear both your arguments and efforts apart, actively sabotaging any potential successful outcome (potentially exaggerating here, but many do not enjoy change).
Alas, there is probably some sweet spot to work towards from day 1. Not going crazy with microservices, especially due to how people interpret the "micro" part (e.g. service per person vs service per team, what the total count should be, domain modeling etc.), but not sticking to a single large monolith either.
I think that sooner or later the industry will try grouping code into services based on the "type" of functionality - the weird PDF export and reporting logic will live in service A, other attachment upload/download logic in service B, the web API in service C, and the old legacy server side rendered UI in service D. That way, at least your efforts to update the web framework and JDK for it wouldn't be usurped by the PDF library not liking it.
Then again, i've seen front end applications baked into back end applications instead of separate back end/front end deployments far too often, so i'm not hopeful about anyone genuinely exploring that approach anytime soon.
I don't disagree - this mononlith was running on three to four year old dependencies, and on the rare occasion that someone did make any big updates it was a case of hoping that the tests passed and then hoping that nothing broke in production.
But it wasn't broken up because it was too hard to update dependencies, it was broken up because it was a complicated mess. They just traded it off for another complicated mess.
Snow Leopard is exactly what I had in mind when writing the comment. I remember watching the keynote and my own excitement about introducing no new features.
Additionally to managers watching that presentation, software architects should mandatory have to work as normal developers in every project they designed for a a few months. Best, if it would be around 2 years after they came up with the architecture. If you don't taste what you cook, how can you learn?
This happens constantly, the entire tech industry runs in whatever direction Google in particular is going. First it was Hadoop and all the pain that came with that, and now it's microservices and kubernetes and all the shit that brings into our tech stacks.
Stop sprinting is good but also : stop following Google. You are not Google. You do not have the needs of Google. Just stop it.
Pick the simplest tech stack you can find. This gives 100x more time to your developers to work on company solutions and not fighting technical complexity.
It never ceases to amaze me how much effort we now spend on nothing but infrastructure and architecture. It's like the Cloud Way of "cattle not pets" and "infinite horizontal scalability" may no longer be questioned.
Meanwhile you could build almost anything in this space with little more than a rack of real servers, some respectable application code written in a fast language, and a bit of scripting and OSS for the glue.
Obviously that sort of environment wouldn't be sufficient for every modern, online-first application but how many years would it take to outgrow it? Many applications never reach that scale. The lucky few would be in a great position to expand from if their starting point was simple and transparent with minimal dependencies.
The problem is you eventually need the things k8s offers and you end up implementing part of k8s, badly. I've seen bigish places that state "we aren't Google" as an argument to not use k8s, containers, etc but don't consider that their tech stacks are a Hodge Podge of bespoke glue, a mass of technical debt and a bus factor of generally 1 or 2.
The question they should be asking isn't "is this tech stack perfect or a mess" it's "is this actively preventing, or about to prevent, us from meeting our uptime and cost and development goals"?
The idea that a shop that built a messy system will suddenly build a non-messy system just if they pick up a new set of tools - that they don't have practice with anyway! - is exactly the silver-bullet fallacy. The system turned out messy because it's complicated, switching to a very complicated tool doesn't take away the complexity, it just changes the way you manage it.
The last couple Kubernetes shops I worked at were that exact sort of hodge podge of custom glue with high tech debt and low bus factor. Just even doing something "simple" like "ssh into the box and poke around" when the two wizards are out suddenly has a huge learning curve. ;) So then you get into discussions of "oh you should've been using managed kube" or "oh, don't use TF -> helm -> kube, use [other way] of managing it" or blah blah.
After you fuck it up a few time you might have enough useful experience so that you can build it well next time... just pray that by then we're still using the same tools. :D
There is a huge learning curve with k8s and there really needs to be a "k8s the good parts" for tooling to use. The problem with k8s vs $custom_system is k8s is somewhat well thought out and $custom_system is generally not.
As someone who works on a hodge podge of bespoke glue, a mass of technical debt, a bus factor of 1, and a docker swarm setup, I don't really see how k8s would help ;)
> Customers actually like stable, responsive, and efficient tools, who would have thought?
_Existing_ customers like these things (and they aren't wrong for liking these things).
New business comes from the potential customers who weren't enticed by the existing feature set, robust or otherwise. You bring them in by adding new features. The retain/new business priority is often heavily weighted in favor of the latter.
Retaining existing customers stuck with a crappy product ain't that hard between sunk cost/lock-in effects and the evergreen insulation of people with purchasing authority from the day-to-day pain inflicted by their purchasing decisions. A couple fancy steak dinners for middle management effectively papers over the cost of driving a department of ops engineers to cirrhosis several times over.
Slowing down is really being able to say "no" to features for awhile while you re-architect the codebase - so you're delivering stuff at the same pace, but that stuff is not product-facing.
I've been through this slog - sometimes it requires fixing what's there and other times, a rewrite. People hate rewrites, but it really is the best way forward in a lot of cases if done correctly.
This is my experience with many different systems. I’ve found that mentoring and personal development identification techniques for what to learn, and when, can be helpful. Inviting a developer to be part of the decisions made regarding their future may seem obvious, but in practice it’s often not done…
And documentation… I’m known as “Just write it down,” but there is usually significant pushback… from all parties.
I never came across code older than 2-3 years that’s not a complete clusterfuck. So looking at it from this side, the fact that it still runs is more of a sign of quality, what do you think?
I have my own theory on this.
Green-field projects are usually whipped up and rolled out by the experienced devs in the shop.
Once the boat is in the water and floating,
the senior devs are posted off to next new-dev initiative,
and the following maintenance/deployment work is assigned to less and less senior devs, and/or student programmers.
Within 2-3 years, those inexperienced devs have mangled the original design into unrecognisability, and the application accumulates instabilities from their bolted-on "fixes".
A different perspective is that the original creators are never allowed back to fix their design mistakes.
I don't know if it's universal, but I have sure seen it in a lot of places, and I have myself been in most of those roles hinted at.
Disclaimer - my job is literally to take on the long term maintenance of those green field projects.
I wouldn't necessarily blame it on a lack of seniority or inexperience, it's a matter of not knowing the initial style, decisions, and limitations, as well as the vagaries of time. I find myself creating duplicate functions not because I wouldn't reuse the ones that exist, but because the original one was hidden away in a different place than I'd ever think to look for them. My approach is different and drifts even further over the years, I can't help it, so that the project starts to look stranger and stranger comparing the old code to the new. The original designers would find their style drifting too if they had to take care of a system for that long. They are just lucky enough that their style drift has project cut-offs, so each slice seems more cohesive.
I also don't know if this is a current maxim or not, but there should be a software principle that matches Peter's - every successful system will grow until it reaches a point where it's no longer successful. More and more requirements get added on, and unlike initial requirements where you could justify keeping scope tight due to staying lean, a system needs to grow beyond it's current limitations or it dies.
It's also that requirements often change in ways the original design hadn't foreseen. And usually there's no time for a decent redesign, so things get hacked together and slowly degrade hack by hack
An additional side effect is that management fails to understand what/why is going on and overvalues the skills of the "experienced devs" and undervalues the skills of the maintenance devs because they don't understand the order-of-magnitude speed advantage enjoyed by devs when writing greenfield code.
"Experienced Developer A wrote the whole damn system in three months! Now Maintenance Developer B wants three months just to add a few buttons?!?"
I've been both the experienced person and the junior maintenance person.
Exactly! As a maintenance dev, you tend not to get much praise, you need to continuously explain why things take so long. I'm currently adapting some code that is, in the eyes of management, adding an extra little wire, which should not be more than a day of work. In effect it will be more than 4 days plus some overtime...
Yeah. I've been in the industry for 20+ years and I have not figured out a way to "shine" when tasked with maintenance work, other than finding ways to avoid such roles entirely.
The root issue is that management never seems to be able to grasp things on a technical level. I've tried to communicate this over the years, and have been praised for my communication skills in general, but I've never managed to bridge this particular gap.
We don’t communicate via GitHub any more. See my other comments.
Also the assistant doesn’t help with the numerous third party components which were never ported to core, the entire MVC stack internal changes or the changes that took place in NHibernate. So even if it did work it’d break everything in the process.
But I should mention that nhibernate is not even in the dotnet foundation and not supported for .net core ( I visited their site and it mentions dotnet framework).
Not sure about your experience with Laravel, but I don’t see how the type system is bad for it. It works surprisingly well for me. With a bit of tooling I get even autocompletions for database fields when using Eloquent. PhpStorm with the (paid) Laravel plug-in are pretty awesome and give me more information during file-editing than for example Xcode.
Laravel has also a pretty active and productive community and so many things just work out of the box, I’m not sure if Go or Rust has anything comparable.
I have to mention: I didn’t touch Go for over a year now and never used Rust. I felt that Go is pretty easy to learn in its basics but quickly throwing things together for a prototype with DB access, I would go for PHP at any time.
It also could be that the learning curve is steep to get some things done and this was a side project that I wanted to get done. That could have contributed to my bad experience, also I'm not developing in php professionally anymore and sometimes when I want to get shit done, but this time it didn't worked for me.
I would not go with nodejs/deno because it doesn't feel a resilient foundation, let's say you put your php server out there, it wouldn't crash easily or if it "crash" would not kill the entire server, so there are benefits with php in that regard, it feels more easy to fire and forget.
Go should have my go to, but because never used didn't wanted to go that route, then Rust that is more painful than php in regard of the frameworks, there are not so mature ecosystem in the web category, but I can build a resilient system with rust, even I could write plain old sql that get's validated at compile time[1], that sounds amazing (well there are few drawbacks, but I like to be able to freely refactor the app in the future and the tooling tells me where I broke it, and with all other alternatives, except Go, there is no such library).
So this is more personal preference and php didn't meet my expectations, that doesn't mean that php is bad, but for me and my use case is, we cannot blame to php yet, but I think the core developers must push even harder the type system, offer better 1st party tooling and more guaranties in regard of the integrity of your program, this is one take of rust that if it compiles it works so when you do code review you only need to review the logic :)
I have some sort of a long-term to-do list, a list where I state my goals, intended projects, things to learn, etc.
Sometimes I decide that I don't want to do certain things anymore, maybe because I lost interest or something more worthwhile pops up, so I don't delete the entry from my long-term to-do list. Instead I strike it out and write a very short comment behind it. Doing it that way seems to keep my head clear and helps me later to stay focused.
tl;dr: I don't have a not-to-do list but I don't delete thing I decide not to do from my to-do list, instead I cross it out. Keeps my head clear, and helps me to document the reasoning. I would recommend it to others.
That sounds quite useful. Because the comment and striking out also shows the relationship -- to quite a big extent -- how you are changing as a person over time.
> This sounds similar to what is going on in Germany. We also have reports of systematic hospizalization where ambulant treatment would be possible and recommended. Intubations which aren't required and actually harmful and traumatizing. Same reason - financial incentives.
Never heard of that in any of a variety of news sources I consume, without a credible source I heavily doubt it. Instead it seems like doctors currently urge people to take precaution (like getting vaccinated) to keep them away from hospital because of the work load.
> Recently went to the doc for a normal cold. Was standing in a small unventilated corridor for half an hour with several coughing people.
A lot of doctors now offer visits via video chat, if you need a sick leave for work it's often enough just to call the doctor and they send you the papers.
True, there are problems with resistant bugs in hospitals and there are problems with testing facilities. The 'ashamed, stupefied and angry' feeling in Germany however can't be generalized like that: Some feel like they're living in a dictatorship because they have to wear a mask, some are angry because of the unclear and reactionary handling of the pandemic on a political level, some don't believe in science but follow alternative medicine or just think all of it is a big hoax, and there probably many more reasons people have.
Overall, Germany still has a pretty good record on keeping people alive while never using really hard measures. What we really could use though is a more civil communication.
> > We also have reports of systematic hospizalization where ambulant treatment would be possible and recommended. Intubations which aren't required and actually harmful
> Never heard of that in any of a variety of news sources I consume,
And I heard exactly the same about Italy from news sources that no one in their right mind would trust, which actually corroborates your point.
also check interviews of Prof. Klaus Stöhr - he mentions that regularly.
> A lot of doctors now offer visits via video chat, if you need a sick leave for work it's often enough just to call the doctor and they send you the papers.
they need my physical HI card every quarter ...
> Overall, Germany still has a pretty good record on keeping people alive while never using really hard measures. What we really could use though is a more civil communication.
Deaths per infections is relatively high compared to other EU countries. Again, check Prof. Stöhr interviews.
The WDR link discusses different opinions on intubation as well as the fincial incentives. It even mentions that guidelines got adjusted which led to less invasive care.
The second link is very 'Welt'-typical, at first using almost tabloid language only to then not delivering on the promise. The interviewee goes into the topic of hospitals having the wrong financial incentives for optimizing the duration of intubation but doesn't accuse a systemic wrong-doing or patient harming.
Fair enough, the system is not perfect but it's far from "systematic hospitalization where ambulant treatment would be possible and recommended".
Not sure about Prof. Stöhr, I read some of his criticism on how the pandemic is handled regarding schools, the idea of a mandatory vaccine and things like country wide measures vs. more local adjusted measures. I didn't hear him saying there is a _systemic_ problem of wrongly administered intensive care just to make money.
Speaking from my own experience as software engineer with burnout and doing sport: do it slowly, everything else is an additional source of stress.
My take on exercise is the following:
- a baseline of pull-ups and push-ups. Doing them multiple times a day, basically every time I take a break. I don't do them until failure but just enough for them being taxing. I started out with 6 push-ups and 2 pull-ups in a row and increased them every one to two weeks. It takes 5 minutes and also helps me with concentration.
- I go bouldering 1-2 times a week because it's fun. Sometimes I'm motivated, sometimes not but that's fine. The boulder-hall has a rowing ergometer which I often use to counter the sitting in front of a computer lifestyle.
- I take the bike everywhere I can.
That said, it's all about consistency. It's amazing how fast one gets used to sport and doesn't want to miss it afterwards.
For diet, I just put a lot more veggies in there than before and adapted to interval fasting (meaning: skipping the breakfast). After a week, I stopped being hungry in the mornings and I lost around 6.5kg in 3 month. Now my weight is stable and I'm happy with it.
I just played around with it and it's a really fun way to develop endpoints. I probably never had that fast of a feedback loop.
Working in the browser and using the browser's dev tools is also enjoyable. Maybe it's just me but if I have to juggle to many open windows I get confused and lost easily, so that's not an issue here.
Things like CD/CI or data processing pipelines are also annoying to develop for similar reasons. Lets ditch the YAML and have an environment we can attach debuggers and properly develop. So webcode is a building block to easier to develop future cloud services. Thanks to Observablehq its also very easy to document, fork, and code share. So this is just one aspect of a improved development workflow.