Is 50% that bad? If instead you hire engineer to maintain access to some kind of file storage on the internet, would it cost more or less?
Would be alarming if it is 500% the staff salary, but at 50% that just seems the cost of outsourcing to standard that likely won't be achieved in house.
Considering it was about 100 developers, it was horrible.
The two major problems were:
1. The volume of data itself was not that that big (I had a backup on my laptop for reproductions), but it was just too heavy for even the biggest things in AWS. Downtimes were very frequent. This is mostly due to decisions from 10 years ago.
2. Teams constantly busy putting out fires but still getting only 1-2% salary increases due to lack of new features.
EDIT: Since people like those war stories. The major cause for the performance issues was that each request from an internal user would sometimes trigger hundreds of queries to the database. Or worse: some GET requests would also perform gigantic writes to the Double-Entry Accounting system. It was very risky and very slow.
This was mostly due to over-reliance on abstractions that were too deep. Nobody knew which joins to make in the DB, or was too afraid, so they would instead call 5 or 6 classes and joining manually causing O(N^2) issues.
To give a dimension of how stupid it was: one specific optimization I worked on changed the rendering time of a certain table from 25 seconds to 2 miliseconds. It was nothing magic.
That does sound like an engineering problem more than anything.
On an off note, migrating to nosql might not have a lot of on paper benefit, but it does enforce developers to design their table and queries in a way that prevents this kind of query hell. Which might be worth it on its own.
How does NoSQL (and which flavor are you referring to?) enforce that? RDBMS enforces it in that if you don’t do it correctly, you get referential integrity violations and performance issues. You’d think that would be enough to motivate devs to learn it, but no, let’s use more JSON columns!
It's the human aspect of engineering, you can't join 15 different tables just by running an 200 line SQL command in nosql and this manual burden forces a re-thinking in what the acceptable design is.
relational DB is great, but just like java design pattern, it's being abused because it could be. People are happy doing stuff like that because it was low resistance and low effort, with consequences building up in the long term.
In my example the abuse was on the OOP part, not in the relational database part.
Database joins were fine, they just weren’t being made in the database itself, due to absurd amounts of abstraction.
I don’t disagree that rethinking the problem with NoSql would solve it (or maybe even would have prevented it), but on the other hand I bet having 5 layers of OOP could also mess up a perfect NoSql design.
My experience from offering devops services on retainer to a number of clients is that the ones that host in cloud environments spend more money on me for similar scale setups than the ones that host on managed setups.
And even if you don't want the hassle of storing the data yourself, there are many far cheaper outsourced options than S3.
There was a time when I felt my greatest programming ability is googling, I think that transferred well to LLMs since googling and asking question on stackoverflow required a very similar skill set.
Digging through source was also helpful to me (I still do it quite a bit) GitHub code search was for a time quite useful, especially for somewhat arcane configuration (over the years, stuff like Cloudformation properties that aren't frequently used, cmake stuff, webpack and others come to mind)
Often grepping through a codebase where I know another example resides of something similar with a vague idea of what i'm looking for helps.
Man.. there are times when trying to debug some error yields no search results and no leads to follow other than "dig through megabytes of the arcane source code with no context of how it's meant to fit together" and if it were possible to have an LLM be able to RAG an entire git repo online and answer questions about it, that would be pretty choice.
I've had basically zero luck with RAG myself so far, especially as I always try to go the local route for better infosec vs relying on cloud solutions like copilot. But trying to tap an LLM to help grok a public github repo wouldn't cover any sensitive data so I wouldn't mind using the cloud for a task like that.
Exactly, I never use LLMs for looking up the specific CFN properties for that exact reason. Populating a resource in a template sure, but the code-spelunking is always done to find the arcane property/value for said property.
I use LLM in my workflow a lot, however it never occurred to me there's any vibe until this online running joke.
And no, the vibe is largely the same as before, there was google, then stackoverflow, and then now here's an AI that would most likely take care of the specific question and generate boilerplate if you frame the question correctly.
Framing the question correctly isn't a vibe, it requires logical thinking that feels almost like coding, and results need to be checked. Even then, I usually don't go a day without spotting some sort of hallucination (correcting obvious wrongs during prompting doesn't count). And testing is essential because sometimes that's how you can catch this kind of hallucination.
I do find learning in terms of project and even coding itself improves a lot faster with a capable LLM. It introduces a lot of idiomatic ways to code, and provides a lot of practice to fix whatever didn't work or add things that are just easier to type then asking it to fix.
Depends on what you consider a "difficult challenge". If the challenge is to remember certain algorithm, it's not worth it - coming up with djikstra without hint was fun the first time, but needing to remember the detail for the nth time isn't.
And for good measure, those stop having value in the future - as they can be referenced from AI as easily as looking up how to convert a date format or center a div.
reply