In a past life, we were reading from twitter firehose and the offshore implemented a solution that spawns a large spark cluster to read from twitter, dumps it to kinesis, then has a cloud function that writes it to dynamo, then merged it into redshift.
It worked poorly, took months to build, and lost data. I implemented a python script that read from twitter firehose and wrote directly to redshift. It was about 50 lines long and did the same thing, and I deployed it to two existing prod servers that were not network bound.
There was a few more cases like the on-shore database identity reconciliation being inefficient, so the company wanted to leverage cloud auto scaling. Well, if you have an exponential time algorithm and you put it into the cloud, it will still be exponential time and cloud auto-scaling will be extremely expensive, and also introduce latency between distributed system components, and indeed it was. The offshore teams initial logic was only that, cloud is auto scaling so can accommodate any performance the business logic needed.
Sometimes people just view tech/cloud as lego blocks to glue together, based on the product sheet marketing, and do not have enough basic understanding of computer science to understand performance, or understand what a simple solution would look like
50/50 It's what employers look for in Senior Engineers, but junior engineers are often hired on pure tech. Historically there was a high amount of BS where people would take credit for deliveries that they were only tangentially a part of, or get ridiculously lucky on their first big project.
It's next to impossible to hold an engineering team together to even KTLO if you don't have some project which grow's their careers.
The number one requirement for Senior Engineers is their extensive experience with the company's tech stack. At least that is what is written in job ads outside of FAANG. You're probably not passing the HR stage when you have experience in Java but the company is looking for a senior .NET developer.
You seem to be misunderstanding people’s motivations. Most developers aren’t trying to be “better” we are trying to exchange labor for money like everyone else.
“Grinding LeetCode” (tm r/cscareerquestions) doesn’t make you a better developer. But it does get you in the top paying companies. Well I didn’t. But that’s a different story.
On the other side of the fence in corp dev where pattern matching is the norm, employers aren’t going to want to hear that you used Sqllite.
Most developer's resumes would be vastly enhanced by delivering a lean, technically successful product to end users, or 5.
A lot of developers think playing with cool technology makes them better developers for some reason.
Which isn't true.
What makes them better developers is knowing what that technology does, and choosing the appropriate solution.
And building consensus, managing personalities and driving execution. By being a reliable, high quality human being.
Which is very different to knowing how to operate the API for that solution.