I am not Indian but I work for a large Tata like IT firm. This hit way too close to home. There a lot of cultural issues here that comes down to management being rewarded if things are done cheaply and discouraging any agency or self-realization by the developers. If I saw this in the US, I’d walk out. They literally don’t have that option as there’s a 90 day salary clawback if they do. Some general thoughts:
- Most management has a non-tech background. So they get what they want to hear and don’t want to hear what’s wrong.
- Thinking this coming from the same team or from the same company is wrong. They silo developers like crazy. There likely was an API developer, an Office 365 developer, frontend developer (so specific it is down to the framework or stack!) and the developers themselves will not touch anything they aren’t “certified “ in.
- I have been in meetings on $100 million projects where they will seriously argue over the cost of sendgrid. Eventually this will come down to no one having “sendgrid experience”
And some developer saying they can do it in Office365.
- Security team will get the first cut in budget since it should “already be secure.”
- You are likely talking over the head of the nephew hired to do security for this. Will the government or anyone sue them? No, so why is this guy bugging us.
Developers aren’t encouraged to develop but get tickets out and not question them. The manga is “It wasn’t in the requirements” all the way down the chain.
I work with smart developers out of India but it is not a culture of innovation. This kind of work is treated like a call center. Don’t go off script, stick your little problem domain, if we aren’t failing we are winning.
> I have been in meetings on $100 million projects where they will seriously argue over the cost of sendgrid. Eventually this will come down to no one having “sendgrid experience” And some developer saying they can do it in Office365.
Reading this in your comment was physically painful.
No - employers are required to pay out paychecks as soon as employment is terminated and if they fail to honor that, they are heavy fines. Days worked is money earned.
I'm aware of employment contracts that require you to pay back the cost of training if you leave the job before some specified time period (maybe 1 or 2 years).
There was a car dealer (Honda affiliate) I had the unfortunate "pleasure" of dealing with back in the mid-late 2000s that stored finance applications by numeric incrementing ids. I never did report it, but I was able to pull up a bunch of sensitive info (SSN, DOB, names, addresses) on folks living in NJ. (I didn't report it because bug bounties weren't really a thing back then and the CFAA was).
I managed to get my application removed, but the vulnerability existed for several years until they updated to a new system. The new system also appeared to have some vulnerabilities, but I never invested time to figure it out. I just did not do business with that dealer ever again, and I'm super wary about car dealerships and finance applications these days...I usually get my financing from elsewhere even if it means a bit higher of a payment...thankfully my vehicle is paid off.
There is a huge missing niche for trusted intermediaries of identity information. We’ve been working on this at https://cerebrum.com in a different niche (background checks), but this comment just triggered a slew of ideas…
This isn’t a marketing push so much as an observation. Some company will fill this niche at some point. There is no reason to disclose your SSN to a car dealership if you can share a shielded, verifiable record of your credit history to them.
You can look through my comment history — I am not here to sell a product.
The security blunders are obviously horrible, but MAYBE explained by inexperienced developers tasked with something way beyond their understanding.
But how on earth did anyone approve storing confidential customer documents in an email account? This seems to indicate there's nobody in charge that understands anything about how to run this business. And if it's a subsidiary or outsourcing partner, it also shows that nobody has ever audited this business.
This is criminally negligent behavior from the company owners, and whoever is contracting them to do this work.
> But how on earth did anyone approve storing confidential customer documents in an email account?
Given the competence shown here, I doubt anyone approved anything. Most likely saving sent mail was a feature of whatever mail server they're using and it was a byproduct of the dumb decision to use an actual account for a "noreply" address.
I saw a fairly large estate agency system that bcc’d every outgoing email from their system to a shared account everybody then synced to Outlook. It was part audit log, part debugging tool, part database backup.
They changed when they realised employees were taking all their customers’ details to new jobs.
The most salient element of this story is that it is business trade secretes (such as customer lists) that motivate enterprises far more than customer privacy.
A friend who's taken Visa's data confidentiality training several times notes that customer data is secondary to Visa's own marketing campaign details.
Yikes! This an unusual exploit since it both has an absolutely massive impact (literally access to everything on SharePoint and Outlook??), with a relatively straightforward vector (just looking at client side JavaScript).
One nit: I'd rather see people redact sensitive data with solid blocks instead of blurs in screenshots. Can't be too careful!
I like to convert to transparent png, cut the area I need removed, use background color (eg black) to indicate that something was removed, and export to jpg.
> In January 2023, DPC was forced to increase the fine issued to Meta Platforms after a review by European Data Protection Board found that the initial fine was insufficient.[12] European Data Protection Board determined that DPC has failed to perform its enforcement responsibility with "due diligence". The critics have pointed out that 7 out of 8 decisions handed down by European Data Protection Board were against the Irish DPC, and that the DPC "always choose the most tortuous, lengthy and expensive legal route to a decision rather than a simple application of EU law".[13]
Not very surprising.
Ireland's entire economy is being based on being a cheap low tax mailbox provider for US Multinationals, trying to bend laws as much as possible in their favor just fits in there
That's why there is a big push on minimum tax rates in the European Union, nobody wants that race to the bottom
Yes and no. AFAIK it provides controls to ensure a certain level of privacy (with serious flaws IMO).
AFAIK it does not do much, if anything to punish breaches caused by incompetence. I have not heard of of any cases where companies were fined for breaches.
Not the whole of Europe. The EEA and the UK has legislation based on it what has not yet diverged significantly.
Not really. That links to a list of all enforcement actions.
If you search for "technical" you get "organisational and technical measures", and most are organisational rather than technical.
If you search by the word "hack" which seems to be the seems to be the usual terminology used there for vulnerabilities being exploited. There are 18 of these of 2182 entries. Not even one per EU country since 2018. Given how common data breaches are it is a tiny number.
Most of them do not give details, but those that do suggest the fines are levied only in extreme cases (for example allowing unauthenticated internet access to medical data: https://www.enforcementtracker.com/ETid-1015 ) or for certain types of failure (e.g. not having MFA). Most do not give details.
its better than I thought, but still far too little, and all the cases where any details are given it is for only a very narrow range of failures.
The search function isn't that good, "Insufficient technical and organisational measures to ensure information security" are basically all data leaks.
Here's a few famous ones, most of which are of course a few years old since government agencies tend to move slow but more recent ones will get what's coming for them.
A GDPR related 400 000€ fine because a company was storing confidential data without authentication using sequential IDs, _and_ they didn’t care when they were warned about the issue.
Let’s also appreciate that a monitoring email endpoint that was designed more or less as a communication worker/agent/runner has been abandoned and was basically matastasizing. That tells me that they aren’t monitoring email utilization or any other compensatory mechanism for identifying anamolous behavior - eg “hey why is email alias costing us [multiple of others]/month in storage”
“The noreply account could be the most important account in an organization because it could potentially have a record of everything they have ever sent to customers”
> Everything after October 18 is a back-and-forth between CERT-In and me trying to determine if there would be a bug bounty reward. TTIBI never responded to the question, so I decided to close the case on December 22 and CERT-In sent me a nice appreciation letter.
If a "leading Insurance Broker across India" can't afford to hire competent developers the least they can do is throw a couple bucks at someone who took the time to identify the multiple severe problems that jeopardized their customers and who notified them responsibly.
The fact that they didn't and still haven't reset the password of the compromised email account blows my mind. Why would I ever trust a company that acts like this to do anything right? It seems like Toyota Tsusho Insurance Broker India should be avoided like the plague.
I've seen similar levels of incompetence first hand. This isn't someone actively ignoring important security warnings. This is someone not understanding what you are talking about. This is someone who, at a fundamental level, has no grasp of the landscape they are operating in or the challenges they are up against. This is someone who wants you to go away because the jargon you're talking doesn't make any sense to them or their team.
"Please stop sending me these confusing emails. I have important work to do."
The only way to fix this is a "changing of the guard" at the organizational level. The IT boss, and everything he has ever touched, has to go.
> This is someone who, at a fundamental level, has no grasp of the landscape they are operating in or the challenges they are up against. This is someone who wants you to go away because the jargon you're talking doesn't make any sense to them or their team.
Sounds like exactly the kind of someone you wouldn't want to have to trust with your personal information let alone trust to manage your life/property/business/liability insurance.
Part of the problem is effectively using this inbox as a "free" SMTP account so they don't have to pay for outbound emails. There would not be as much sensitive information in this account's sent/inbox if they using something like SES, which is incredibly cheap ($0.10/1000 emails).
Indian developers don't care about security unless you explain how to build the software step by step. Many firms got burnt because of this. We stopped working with individuals and teams in 2018. They don't care about security.
I was trying to say this in polite terms, but failed to find the words. I am absolutely not knocking India at all. My all time best manager came from India and he taught me A LOT about software development in the earlier 2000s. He was incredibly smart, and there were 2 other folks at that company from India that also were very smart.
However, I've also had to deal with the reverse. Folks from India and elsewhere that just blindly churn out code according to literal instructions and don't give any thought as to how that code might not be safe/efficient/whatever.
That being said, I blame the company, not the people. You could easily end up in a similar mess here in the states if you don't take some time to vet.
(note: watching someone code on an interview aka pair coding isn't vetting, even take home assignments don't. If you do either of these, you aren't vetting, you are subconsciously looking at speed/accuracy/ability to think quickly, which may also mean you are discriminating based on age/disability -- i.e. people that code slower or think slower tend to be older or have a disability -- which is a violation of federal law here in the states.)
It is phrased as I believe it. I am not going to feel sympathetic towards Indian developers after they caused millions of dollars of damage to our businesses over the years. The great majority of them are very low-skilled and lie about their skills. When questioned they disappear. We even dealt with fake-named ones. Had some installing malicious scripts to random parts of the e-commerce systems.
I even had a conversation with one telling me "Americans can afford it, we don't feel
bad hacking them". This came from a developer who worked with us for over 2 years.
This is not a subject like genders and pronouns or whatever makes sensitive people feel angry. I advise anyone I meet the same thing. Avoid at all costs. Maybe 1 in 20 are okay but I don't recommend rolling the dice for that single gem.
This isn't about "sensitivity". This is about racism. Just because you had a bad experience with a few Indian developers doesn't mean that all of them are bad.
And where did that 5% chance come from? Any kind of data to back it up or is it just a number you pulled out from nowhere?
If your businesses don't know how to employ external contractors blame it on your managers. India is a massive country. Not every Indian developer is an incompetent buffoon, just like not every European/American is the next Fabric Bellard.
I am certain my observation is not different than other developers in the United States. I also work with Indian companies who happen to know other Indian developers and I hear the same quality issues from actual Indian people themselves. I am not talking about 1-2 developers out there. We tried over 50 over the years. Barely 1 or 2 worked out and eventually, they started working for other companies because loyalty doesn't exist.
It is not racism. Since we stopped working with them, we haven't had a single issue with our software development services. They simply do not understand business basics. They write code as if we ask ChatGPT. No second thought, no bug hunt. Just linear coding with no security in mind. They do not ask themselves "If I remove this code from here, would this other part break?". We worked more on fixing the issues they caused than we worked with them.
We won't make this mistake again. I am also very sure this is not only us. I heard it from many American developers. If you have some time, just read some horror stories on tech blogs and other forums. I understand things might be difficult in India that force people to lie about their skills to make money but now we rather pay double than deal with issues the cheap labor causes.
A simple APP_DEBUG=false in their production .env would’ve prevented leaking the Laravel stack trace and logs (Issue 3). I imagine they were debugging in production and forgot to switch it back
> Logging into the Microsoft account was surprisingly easy. There was no two-factor authentication set up or any other login verification prompts. If there was, it probably would not have been possible for me to successfully login.
This shouldn't have been possible in the first place for a few months now as Microsoft did a massive push to disable anything but OAuth logins for O365.
Wild-assed guess before I read this: in their greed for personal information, they took what should be a purely client-side scripted job into something that connects to the back end.
Edit: Yup! Instead of just doing calculations, it involved some e-mail workflow.
> The password could be used to log into the “noreplyeicher@ttibi.co.in” Microsoft email account.
I'm surprised this is literally true as described.
The actual browser itself makes the actual SMTP connection to the Microsoft e-mail host! The authentication name used is the above e-mail address. I typed out the Base64 to check:
There is a second IT silliness here, a minor one compared to the password gaffe. "noreply" addresses should not be real mail accounts or working aliases!
The noreply address is just a fake you put into the SMTP envelope and From: which will bounce due to not resolving if someone replies to it.
> The actual browser itself makes the actual SMTP connection to the Microsoft e-mail host!
This is not generally possible, browsers cannot make arbitrary socket connections in the way that would be required to reliably communicate with an SMTP server. The article makes clear that the frontend is calling a poorly-coded email-sending API implemented as an HTTP endpoint.
I see. That's what I would have thought so I was scratching my head; that lack of sandboxing would turn all browsers into horrible attack vehicles, rendering botnets obsolete.
This is a boggling level of disdain for customer security - even putting aside the insanely low levels of data security, it's mind boggling that the website remained up for months after the disclosure, and that even after being taken down the vulnerability remained open.
This is a boggling level of disdain for customer security
To be fair, this usually doesn't start as a boggling level of disdain. It usually starts out as 100% ignorance. It's how the people and the group respond to the negative feedback from experts and from reality, which brings in the disdain, even spiraling to boggling levels.
There are two deep lessons herein, rooted in game theory.
This is an "I have no idea what the hell I'm doing" level of incompetence.
Isn't it accepted security knowledge, that about 99% of everybody is at a "should not be doing it myself" level of security/crypto incompetence? I'm not saying that the example isn't particularly bad. It is.
Requiring competence would appear to be the wrong way to do it, here.
Given that the password hasn't changed, I'd assume that there are exactly 0 sysadmins or software engineers working at this insurance company. A web app was poorly hacked together a few years ago, and just ticks-over in the background. Nobody in the org knows about the exploit (and it's possible they don't have the capacity to understand the exploit).
There's a serious problem with human beings. A very loud, emotionally charged warning used to work perfectly for us. "SABERTOOTH TIGER!" is obvious and it's useful for the warning to be delivered with such emotional force.
However, there's a problem when the severe danger is disguised by layers of abstraction and complexity and obscured by time. Even emotionally neutral warnings will trigger our psychological attack defenses in these cases.
Note, I'm not saying op did anything wrong. What I am saying, is that delivering negative feedback about anything complex is itself a complex operation!
A security membrane which needs this kind of feedback to work correctly should be viewed as having a serious design flaw.
It's a bit hard to imagine this specific problem existing outside of the Microsoft ecosystem. I can very well imagine that there are loads of corporate resources provided through a valid O365 account that are useful for targeted hacks -- heck, the metadata in the corporate directory alone is going to be useful to a ne'er-do-well.
I really can't believe they haven't changed the password. I wonder what part of their workflow that breaks?
So crazy that things like this still happen in production. I mean, maybe I have survivorship bias (we never hear about the companies that don't have security flaws, or the hundreds of APIs that are completely secure), but it should be super easy to make a site that is secure. Even I know how to do it. It shouldn't be that hard to find people who know how to make secure sites.
You are either young or don't know any better. All major companies have bug bounties program and consistently, every few weeks, payout CRITICAL level bounties, as in attacker managed to get full server/access to any account etc. Security breaches are just a matter of time. Who is to blame is debatable, since being a criminal and breaking and stealing (into digital or physical business) is against the law.
The sad fact is that the law in most countries is so toothless (and the law enforcement agencies so far behind) that the legal penalties are mostly just academic.
Bug bounties (and proper education + screening processes for developers) are the most effective way for businesses to prevent security breaches - relying on legal recourse is more of a “shutting the stable door after the horse has bolted” sort of approach.
> Who is to blame is debatable, since being a criminal and breaking and stealing
Not debatable at all - if you get mugged, it’s the criminals fault.
But if you trust your money to a bank, they leave the safe unlocked, and your money is gone, it’s their fault. That literally the whole point of a bank.
Same with your data - when it stolen, it usually the company’s fault - after all if there is no security, sooner or later it will happen.
I am so with you. I should be the lowest common denominator when it comes to security- I am what in my head qualifies as a novice. But I notice basic flaws at almost every company I work for. Absolutely baffled how this keeps happening.
I've not very knowledgeable on the process of building a backend API but could someone explain how sending the email's password back in an error log could ever been a good idea?
Obviously, the answer is never (unless it's for _very_ specific testing in a dev only environment).
In this case, it's not that they were sending the password directly for any reason, but instead returning the raw SMTP log from sending the email; which as a byproduct had the password in it due to needing to authenticate with the SMTP server.
Passwords in error logs are only _ever_ good if doing very, very, low level debugging of why logins aren't working right. Even then it's usually enough to just log which auth backends are touched and their result state. However it MIGHT happen if an encoding issue is suspected. Ideally never on a production system.
>October 18, 2023: I noticed the vulnerability is now fixed – the email sending API now requires authentication. I ask CERT-In if TTIBI can offer a bug bounty reward.
>TTIBI never responded to the question, so I decided to close the case on December 22 and CERT-In sent me a nice appreciation letter.
The letter:
"Dear Eaton Zveare,
This email is written in appreciation and recognition of your efforts for bringing our attention to the "Cryptographic Failures" in one of the Indian websites on 08.08.2023.
The role of responsible security researchers is pivotal for creating a secure cyber ecosystem and CERT-In strongly believes in working actively with a researcher like you for the discovery of cyber security vulnerabilities and their subsequent remediation in a responsible manner.
We look forward to your valuable contribution in future as well.
- Most management has a non-tech background. So they get what they want to hear and don’t want to hear what’s wrong.
- Thinking this coming from the same team or from the same company is wrong. They silo developers like crazy. There likely was an API developer, an Office 365 developer, frontend developer (so specific it is down to the framework or stack!) and the developers themselves will not touch anything they aren’t “certified “ in.
- I have been in meetings on $100 million projects where they will seriously argue over the cost of sendgrid. Eventually this will come down to no one having “sendgrid experience” And some developer saying they can do it in Office365.
- Security team will get the first cut in budget since it should “already be secure.”
- You are likely talking over the head of the nephew hired to do security for this. Will the government or anyone sue them? No, so why is this guy bugging us.
Developers aren’t encouraged to develop but get tickets out and not question them. The manga is “It wasn’t in the requirements” all the way down the chain.
I work with smart developers out of India but it is not a culture of innovation. This kind of work is treated like a call center. Don’t go off script, stick your little problem domain, if we aren’t failing we are winning.