Not that it killed us or anything, but we hired a Director of DevOps at my company who we tasked with the simple job of setting up a dev server for a Java REST server that would have like 6 concurrent users. It needed a cache, but no persistent database. A task beneath a director and one that the dev team would usually just do themselves, but he was here to show how to DevOps the right way and not be so ad hoc. He somehow managed to set this up to cost like $8000/mo after we have conservatively budgeted for $50. He was fired for myriad reasons and we spent like a week trying to figure out what he had done.
When there is a lot of money involved, people self-select into your company who view their jobs as basically to extract as much money as possible. This is especially true at the higher rungs. VP of marketing? Nope, professional money extractor. VP of engineering? Nope, professional money extractor too. You might think -- don't hire them. You can't! It doesn't matter how good the founders are, these people have spent their entire lifetimes perfecting their veneer. At that level they're the best in the world at it. Doesn't matter how good the founders are, they'll self select some of these people who will slip past their psychology. You might think -- fire them. Not so easy! They're good at embedding themselves into the org, they're good at slipping past the founders's radars, and they're high up so half their job is recruiting. They'll have dozens of cronies running around your company within a month or two.
That's not really it. Our company is small enough that I can talk one-on-one with the head of the tech department and I did give direct feedback about this person. That head of tech was responsible for the mishire, but also got rid of this person pretty quickly once all the feedback accumulated.
My company is service-based and just over 1000 people. Timesheets equal billable hours. It's occasionally very pressurized and we lose people pretty quickly when there's a lull in work, but it also means that useless people have absolutely nowhere to hide.
It sounds like your boss is making these decisions on his own without soliciting additional perspectives and feedback in advance as part of the hiring process. If so, that is a common pattern that, in my experience, leads exactly to these kinds of hires.
But with a fire-fast approach, it sounds like your company can move fast on hires and be ready to contain the damage.
My personal take on it is that a situation like that can be prevented from getting out of hand. But that requires a great deal of courage, often putting the entire business at risk. As a founder you will even come across as as a mean guy if you take on the task of enforcing integrity. Judging the integrity of people often means asking very hard probing, personal questions which I suspect is difficult for most founders.
If you hire people, you could ask or collect other kinds of feedback how your hire has performed (from someone else than themselves directly of course).
I counter this I've never had good feedback, because of people that wanted a solution, but not from me and sometimes I would bring a solution that will cost less overtime.
I have been bitten colleagues and it still hurts. Because they weren't that great with I.T.
I rather show it off what I can do and what I need to work on. Than relying on somebody else. (Again I have been bitten by that.)
Mishire. I don't want to doxx anyone, but the tech team realized pretty quickly that he was more of a technical manager and not a real engineer. He had a serious neckbeard mentality about being right about everything yet couldn't write a Hello, World on his own. He did little to win people over and got caught reusing work he'd taken from his team at his last job.
You know, it happens, to everyone, however good or experienced; what matters for a company's (and individual) sake is how we respond to mistakes.
You guys responded well, that was resilient. The next step would maybe be antifragility. Did something change afterwards, because of this bad experience?
We only identified two things that were unusual. For one, he used RHEL instances instead of Cent or Ubuntu and the other was he allocated a load of EBS capacity with provisioned iops. Idk if it's even possible to a complete history like if he had done other stuff that he had already undone before we looked.
AWS gives you the tools you need to answer this question. Cloudtrail logs every api action (there may be some esoteric corner cases, I think some aws services have launched features and then weeks later launched "oh those api calls are now recorded in cloudtrail", that kind of thing, but by and large it's good enough).
You should have a "global" cloudtrail turned on in all your aws accounts, with the integrity checksumming turned on, either feeding directly to an s3 bucket in yet another account that you don't give anybody access to or at least feeding to a bucket that has replication set up to a bucket in another locked-down account.
The cloudwatch events console can find some cloudtrail events for you, but you might have to set up Athena or something to dig through every event.
We didn't have enough expertise to do all that nor did we own the billing info. We also didn't spend too much time because it was moot. We shut down everything we could see and ate the bill.
