Hacker Newsnew | past | comments | ask | show | jobs | submitlogin
Fired employee deleted servers, causing it to lose S$918,000 (channelnewsasia.com)
56 points by bovem on June 12, 2024 | hide | past | favorite | 88 comments


> His contract with NCS was terminated in October 2022 due to poor work performance and his official last date of employment was Nov 16, 2022

This is why you don't force employees or contractors to work through their final two weeks. Too little benefit, too much risk.

> After Kandula's contract was terminated and he arrived back in India, he used his laptop to gain unauthorised access to the system using the administrator login credentials. He did so on six occasions between Jan 6 and Jan 17, 2023.

Oh nevermind, it's far worse than just that!


The real question is why he had active admin credentials months after termination.


my anecdotal guess: there was a single admin account rather than a group, and they didnt want to risk changing the password because of an unknown number of scripts/services using it.


Is this even remotely justifiable in a well run organization? Speaks more to the management than this fired employee.


I don't think anyone sets out to have this policy. What seems to happen, especially with organizations that existed pre-computing, is these things evolved gradually along with computing and best practices. You picked the person with the most computing prowess, sat them down and said "make this work". And they got it working. And then it was too important to stop working.

It takes an IT team with skill and a management team that trusts them and their decision making to turn that kind of thing around. It's a similar story of a company that fails to have a working backup or disaster recovery: "everything has been fine and we can't justify the expense", when in reality it's a time bomb.


> a well run organization

Those exist?


That's the point.


Broken off-boarding process.

HR forgot they have to create a ticket for when they ask a contractor to leave early. IT has the account expire on the date the contract expires, but somebody needs to tell them that someone left.


Yes, that's terrible mistake from the QA test lead


Only $678K worth of damage? Rookie numbers.

Worked with a number of folks that caused much more than that just by mere accident. Not disgruntled or anything. Just “fat fingered” a command or had a momentary brain fart (deleted prod db instead of backup!).

Guy truly was incompetent and deserves everything coming to him.


Incompetent is an understatement. According to the article (which by the wording is not written by someone technical, so take with a grain of salt I guess), the accused:

- accessed the servers tens of times post employment from India

- returned to Singapore

- lived with another active employee of the company he was planning to attack

- had a script kiddie Google history

- made no efforts to cover his access

It's hard to read in any way but him wanting to be caught, although I'd love a more detailed article to clear up some confusion. It was cold blooded months after, and he moved back to the country where he could be convicted. Wild.


Reads like arrogance and poor trade craft, not a desire to get caught.

He could have thought that the organization was too corrupt and incompetent to track the attack. He might have been unaware of law enforcement investigation procedures, or jurisdiction risks.

It would be common for script kids to lack sophistication in forensics, legal risk analysis, and not really understand the magnitude of their crime.

Assuming there is no evidence that he was getting paid by a third party to do this, I think his defense could have argued he was a naive and immature prankster that didn't think they were doing real damage, not a hardened criminal intentionally causing damage for profit.

He only got 2 years. Seems light, for nearly a million dollars in damages, right?

A 2-year vacation with free food and housing and criminal advocacy, and then try again maybe with more professional approaches? The birth of a real pro!

Doesn't sound so bad honestly. He'll be 41 when he gets out, and ready to strike again! Maybe he'll get a big brain, and go straight into security consultancy, a la Mitnick! Either way, the kid just made his boldest career move! Wishing him all the best of luck!


Actually I'm now 100 percent persuaded.

When the company let him leave without invalidating his credentials, he developed the idea that he could act with impunity because the company was just completely incompetent on the security front.

The problem was that he acted without knowledge of modern forensics.


> On Mar 18 and 19, he ran a programmed script to delete 180 virtual servers in the system.

> The system that Kandula’s former team was managing was used to test new software and programs before launch. In a statement to CNA on Wednesday, NCS said it was a "standalone test system".

> As a result of his actions, NCS suffered a loss of S$917,832.

Wondering if these are CI/CD pipelines, and how the loss amount was calculated since these can be spun up again.


I guess it is mostly the billable hours of testing and other teams waiting for few days without work (but paid) before new VMs come online


There's a reason why some companies are using measures that feel very inhuman when they fire someone: it's because of people like the one from TFA.


I told my colleagues to do me a favor: if I'm ever let go for any reason, please terminate all my access immediately and diligently. It's for my protection just as much as theirs. If I can't get in, they can't blame me if something goes wrong down the road.


That is very threatening, why would you share this? If you truly think this way, you may want to consider a different career or look internally to figure out what makes you think this way.

EDIT: I thought the "If I can't" portion read "If I can" in parent's comment. Disregard, sorry parent.


I was working at a medium sized accounting software firm over a decade ago when I was called into the CEOs office and sat down for an interrogation.

Apparently they had setup their Exchange server poorly and I'd had access to any number of mailboxes (including the CEOs) and other folders that I wasn't supposed to have access to. I was completely unaware of this access, but was being directly accused of having access confidential information.

I insisted, truly, that I wasn't aware of the access and hadn't looked at anything confidential, but they tried to tell me they had proof that I had accessed email I shouldn't have. At this I got upset and demanded they show me this proof, which they couldn't do because they didn't have any, but I'm guessing they thought I had seen some stuff and they could get me to cave it if they pretended they had proof.

Anyways, I didn't get in any real trouble because I wasn't intimidated, but it damaged the relationship and I ended up quitting a few months later. Now I feel the same way as OP, I don't want to be in a position where someone can make a plausible accusation of misconduct, you could be totally honest and still end up railroaded just for being in the wrong place at the wrong time. CYA.


What's threatening? I read that as just wanting their access removed as per policy/procedure, with no special treatment, because that both protects the employer from potential tampering and the employee from accusations of such tampering.

I don't read any implied threat that they actually intend to or think they might do something harmful, just wanting a CYA in the same way the company wants one.


That's exactly it. I'm not asking them to save me from temptation or anything like that, or to provide me an alibi ("ok, you're locked out now wink"). I want them to do the right thing for our mutual benefit.


Pretty sure they read

"If I can't get in, they can't blame me if something goes wrong down the road."

as

"If I can get in, they can't blame me if something goes wrong down the road."

That's the only interpretation that sounds threatening.


Ohhhhh, that makes much more sense now. If I read it that way, I'd also think I was a creep.

Yep, they confirmed it.


Just because someone wants plausible deniability doesn't mean they're considering doing something wrong.

They could just not want to be even considered a candidate for blame.

For example, if I was ever asked to babysit someone, I would want them to install a camera pointing at me at all times, for my sake and their peace of mind.


> That is very threatening

It's just creating "plausible deniability."

> or look internally to figure out what makes you think this way.

That's easy: Sarbanes-Oxley.


For me, HIPAA. I do not want access to a single resource or record that I don’t require to do my job.


If I work in a place where the data security practices are abysmal, where we have not been hacked and ransomed/extorted purely because nobody has bothered, I would definitely want to make sure that my access to any systems is revoked once I am no longer employed. I would not want to be anywhere close to a ticking bomb.


Emotions aside, isn’t it just sensible security policy to delete all permissions and invalidate all credentials of a terminated employee as soon as possible? Any other approach would be exceptional.


Both sensible, and required by many compliance frameworks.


It's called being a grownup. If I worked for a restaurant and was fired, I'd give them the keys immediately. If someone unlocked the door and stole stuff later, hey, look at someone else who still has the keys. I don't.

It's the same here. If I'm fired from a tech job, the company should do the right thing to protect themselves and lock me out right then and there. It's not because I -- the real me, not the hypothetical villainous me -- pose a risk to them. I wouldn't hurt an ex employer under any circumstances because 1) that's a bad thing to do, and 2) I don't want to be in jail. It's more that best practice dictates they do this. There's no benefit to them whatsoever in allowing me into systems I no longer have a legitimate need to access.

And again, this also protects me. If I'm long gone and something suspicious happens, talk to their current employees, not me. I don't want to access to their stuff, and I couldn't even if I wanted to.

Edit: Ah, honest mistake. I could see why you'd think that given the misreading.


Some people work in regulated environments with audit requirements.


Right? Just take the L and look for a new job


I don't see how that's incompatible with wanting your access revoked on termination to avoid accusations of wrongdoing.


who's threatening whom?


A more humane and complete way of handling this is to provide severance that is contingent on a smooth off boarding.


He didn’t do anything malicious until half a year after his termination.

This is a failure of the company.

An inhumane offboarding would only encourage this.


The reason of the measure you are describing is incompetence. But what happened here was a lack of: proper off-boarding, basic controls, and basic security hygiene.

Unless, of course, our interpretation of inhumane measures is very different.


Well, revoking credentials isn't inhumane. You can fire someone humanely and without compromising security.


Was gonna say, they can bid you farewell without giving you access to stuff.


Except, his damage was done way after he was fired.

Theyrevinhumane for more than just security


I hope your corporations all get taken down by persistent threats created by your own cruel policies, if you think any of that is justifiable.

Literally just revoke the credentials. If you want to be extra, don't be surprised if you find your company source repos on the dark web, for Justice and Profit.


The level of incompetence on NCS's part is criminal, they absolutely deserved what they got. It could have been much worst, as in the malicious actor finding a way to insert code that makes it into production and then exfiltrating sensitive data to be sold on the dark web. Luckily Kandula wasn't smart enough to think like one of us.

NCS sounds like a clown show based on this article. The administrator credentials should have been changed as soon as Kandula was let go. Ideally, these credentials shouldn't have ever been used and everyone should be acting as themselves with a elevated privilege step.

As for the $678k in damages, why didn't NCS have snapshots that they could have quickly restored? Sounds like their BCDR plans need to be reviewed and updated.

Moral of the story is don't do business with NCS.


Kandula's laptop was seized by the police and the script used to carry out the deletions was found on it.

full disk encryption is a thing. it's amazing how people who are otherwise technically competent leave such obvious incrementing evidence on computer


I assume he had to give up the decryption credentials when he handed the laptop for the investigation. Not complying with the investigation can make it worse for you in some places.


Cant you just forget decryption credentials during the investigation?


Sure. They may not believe you, though.

Related example: https://en.wikipedia.org/wiki/H._Beatty_Chadwick


I'd rather have the 14 years of my life than $2.5 million I can't even use. People are weird.


There are double encryption systems. You give your "password" and they access a volume, where another encrypted volume isn't even apparent.


You can, but depending on the justice system in your area, you might not want to.


That's why you have two decryption keys. One for the real stuff, one that decrypts to a decoy.

https://veracrypt.eu/en/VeraCrypt%20Hidden%20Operating%20Sys...


Yep. All very good advice on how to better conceal your crimes.


Or hide anything you wish to from those who want to use the wrench method. You're allowed to have things to hide without being a criminal.


Plausibly deniable encryption exists for a this very reason.


> incrementing evidence

It just keeps piling up!


>full disk encryption is a thing.

Just actually get rid of the evidence. Throw your laptop into a shredder and buy a new one. At least get a new hard drive.


I can't speak for Singapore's specifics, but outright destroying evidence is often its own crime.


Good luck proving it!


In a country where possession of chewing gum is illegal and 14 grams of heroin geets you a death sentence, I'm not sure I'd want to test the practical limitations of the burden of proof on that sort of thing.


But you already committed the crime, surely that alone will be much worse than a completely unprovable charge of destruction of evidence.


Some legal systems permit inference of guilt ("spoliation inference") based on attempts to destroy evidence; you might wind up with just extra convictions this way.


probably easiest to buy a second hand laptop, do the thing and then low level format the disk with a hammer.


For this individual, it's a little late to close the barn door.


Intent: thinking through this from a problem-solving perspective. Don't do this crap, kids, lest ye end up in prison or worse.

Or at least do a full wipe (including backups) and reinstall. "Here's my FDE key, but I erased everything after I left that place and I don't have their stuff anymore."


Use a throwaway disk or device, get rid of that and leave the official one alone.

I discourage this behavior but there could be many cases when this could be useful.


With SSD drive it’s enough to simply wipe the drive clean.


Remember: sometimes a simple 'rm' may not be good enough, that's why the gods of GNU gave us 'shred': https://linux.die.net/man/1/shred


Well, the file was found on his laptop and laptops are pretty much exclusively using SSDs. On SSDs a simple `rm` is enough. On an SSD:

    1. You run rm
    2. Your filesystem uses trim to mark the pages as invalid
    3. The drive's garbage collector finds blocks containing invalid pages and consolidates valid pages into new blocks and marks the old blocks as invalid.
    4. Then the drive resets the block to empty and marks it as available.
This improves write performance because SSDs can only write to empty pages (they cannot overwrite pages that have already been written, instead they'd have to first reset the page and then write a new page) so by proactively resetting pages, they have pages ready to be immediately written.

But this also means that the blocks containing your deleted file will be proactively reset/emptied which means it will uncharge the cells which is equivalent to all the bits being `1`, thereby destroying the file.

Source: https://kcall.co.uk/ssd/index.html


Cool, I learned something interesting today, thanks!


i dunno what privacy/civil laws are like there, but couldn't the police or courts compel Kandula to surrender the password?



I assume Kandula was a microsoft user and it's widely suspected there's a backdoor for law enforcement.


On built in disk encryption? I'd be surprised if security researchers hadn't found that target. Got a source? Who credible suspects this?


NSA and FBI both approached previous bitlocker devs to insert backdoors in the early days. It's no secret Microsoft cooperates with federal government branches to ensure the government keeps using their products. Further because bitlocker is closed source, there hasn't been any outside research done on the code.

https://boingboing.net/2013/09/11/how-the-feds-asked-microso...

Some good comments here:

https://old.reddit.com/r/sysadmin/comments/26vm25/why_is_the...

There's more resources on google and I remember attending a talk at either blackhat or defcon on why you shouldn't be using bitlocker.


Run a VM on a portable linux distro, delete the VM, easy days.


Easier to run tails or other live distros that don't store anything on disk and then run memtest afterwards.


He was googling for "how to delete VMs script" to do the initial attack, I think you overestimate his opsec capability.


He had an opsec capability? I assumed he'd never heard of opsec.


lol 34 people replied to this. I think this ranks in the top 10 of my most replied to posts ever. But no change in votes.


Probably thought he was safe in his country. You can be tech literate and international criminal law illiterate.


He moved back to Singapore, where he committed the crime.


Not just moved back there, but used the company Wi-Fi network!


My bad. Should have rtfa. Guess he's just an idiot.


I wonder how much he was fine (if he was - they also have caning as a penalty). Singapore is known to be incredibly strict with criminal punishments. There was a recent $8 billion money laundering case that garnered international headlines because SG is known to be corruption-free for the most part. I'm sure you can find the reasons for verdict (SG has no jury trials) and reasons for sentence. Generally, an incredibly well run state IMHO. (Yes, it has its downsides, criticisms and controversies). It'll be interesting to see how PM Wong will govern compared to LKY & LHL.


I’m sorry but based on this,

> NCS is a company that offers information communication and technology services.

And more importantly, this:

> After Kandula's contract was terminated and he arrived back in India, he used his laptop to gain unauthorised access to the system using the administrator login credentials. He did so on six occasions between Jan 6 and Jan 17, 2023.

The company is not just ignorant, but massively incompetent.

You don’t fire someone without totally withdrawing every last shred of access they have. The fact that he was able to use a common, generic administrative credential shows that NCS fails epically at even the simplest of security.


What about leaving an ssh key on there with a port open?


180vms and 678k loss. So 3.5k a pop.


[flagged]


>I would also imprison (or at least fine) the company's security chief.

This is just incompetency. It is ridiculous to think that this should rise to the level of criminal negligence.

Of course there are good reasons to fire him.


The occupation "engineer" does come with liability, and such incompetence can (and does) result in prison sentence. Of course not "CS engineers".

(Would you call it ridiculous also, if it was an article about architect, who designed a house that just razed itself to ground after a someone shut the door too hard?)




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: