Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Chances if Microsoft or Crowdstrike will be held liable for financial losses caused by this outage?


Financial losses? The comment you're replying to is mentioning heart attack treatment here. We're talking about deaths. Most of us won't like to hear this but for all of us who work at SaaS that is deployed on servers around the worlds, our bugs cause people to die. It's a given that at least a dozen people will die directly (medical flights, hospitals both being hit) due to this broken update, let alone indirectly.


I don't think the parent comment was ignoring that. The penalty for a company who does this can't be to bring someone back from the dead, it's likely to be financial, which is the aspect they're talking about.


If this was a Japanese company, the entire c-suite would have committed seppuku by now.


[flagged]


As others have already stated, yes, that is how we should be interpreting comments, in good faith and in the most charitable way as the site guidelines suggests us to.


Good. That's the HN way.


That's how I read it, ie "will there be severe fines for the loss of life and other losses for this?".


You’re basically asking for a virtue signalling disclaimer. I think you’d prefer a different social network.


I've finally learned to spot and ignore all emotional arguments.

https://www.scribbr.com/fallacies/appeal-to-emotion


If companies want the nice parts of being "a person", they should also deal with the bad parts of being a person. Financial fines are not enough. Though I'm not sure how we'd build a jail cell for an entire company.


Fines are not enough because a large enough fine will kill a company, destroying lots of jobs and supply chains.

Why not dilute the shareholder pool by a serious amount? There's no need for a statization to formally happen, the government can sell the shares back over time without actually exercising control.

Also fire execs and ban them from holding office on publicly traded companies for the foreseeable future.

Seizing shares doesn't impact the cash flow of the company directly, thus shouldn't cause job losses, but shareholders (who should put pressure on executives and the board to act with prudence to avoid these kinds of disasters) are adequately punished.


> Fines are not enough because a large enough fine will kill a company, destroying lots of jobs and supply chains.

That could be amazing: "Ooopsie, in punishing Crowdstrike they've ended up folding and now there's a second global outage."


This actually sounds like a workable idea, but the implementation would be extremely thorny (impact on covenants, governance, voting rights, non-listed companies, etc) and take forever to get done. It would also punish everyone equally, even though they clearly do not share equal blame.

You probably want, in addition to your proposal, executive stock-based compensation to be awarded in a different share class, used to finance penalties in such cases where the impact is deemed to be the result of gross negligence at the management level.


> but shareholders (who should put pressure on executives and the board to act with prudence to avoid these kinds of disasters) are adequately punished.

So if I own some Vanguard mutual fund as part of a retirement account, it’s now on me to put pressure on 500+ corporations?

Perhaps it’s on Vanguard to do so…but Vanguard isn’t going to just eat the cost of increased due diligence requirements. My fees will increase.

How does that increased due diligence even work? It’s not like I or Vanguard can see internal processes to verify that a company has adequate testing or backups or training to prevent cases like today’s failure.

When, on average, X number of those 500 companies in my mutual fund face this share seizure penalty per year…am I just supposed to eat the loss when those shares disappear? Does Vanguard start insuring against such losses? Who pays for that insurance in the end?

This doesn’t even really hurt the shareholders who are best placed to possibly pressure a company. This doesn’t hurt “billionaire executive who owns 40% of the outstanding shares”. I mean, sure, it will hurt that little part of their brain that keeps track of their monetary worth and just wants to see “huge number get huger”…but it doesn’t actually hurt them. It just hurts regular folks, as usual.


If you own a mutual fund, then you do not own shares of the 500 companies, rather you own shares of the mutual fund itself.

Consequently you don't put pressure on the 500 companies, you put pressure on the mutual fund and the mutual fund in turn puts pressure on the companies it invests in and exercises additional discretion in which companies it invests in.

>Perhaps it’s on Vanguard to do so…but Vanguard isn’t going to just eat the cost of increased due diligence requirements.

Yes they do, because mutual funds do compete with one another and a mutual fund that does the due diligence to avoid investing in companies that are held liable for these kinds of incidents will outperform the mutual funds that don't do this kind of due diligence.

> It’s not like I or Vanguard can see internal processes to verify that a company has adequate testing or backups or training to prevent cases like today’s failure.

I don't know specifically about Vanguard, but mutual funds in general do employ the services of firms like PwC, Deloitte, and KPMG to perform technical due diligence that assesses the target company's technology, product quality, development processes, and compliance with industry standards. VC firms like Sequoia Capital and Andressen Horowitz do their own technical due diligence.


Just perhaps the idea of sticking everyone's retirement funds into massive passive vehicles was a bad one and has an unhealthy effect on the market, as you illustrate here. It is the way of things now so I see your point and it would be harmful to people, but getting in this situation has seemingly removed what could be a natural lever of consequence. We can't really hold companies accountable lest all the "regular folks" that can't actively supervise what they're investing in become collateral damage.


Other stocks will go up as a result. It's not like money is ever destroyed.


The death penalty could be an option? Dissolve the company, seize their assets, bar anyone involved from ever running or owning a company again.


Should be, but I don't know that that's appropriate for involuntary manslaughter.

Do it to Boeing, sure.


Hold the board of directors and the C-suite personally, corporally accountable -- immediate changes for the better will follow.


You'd seize the company from its current shareholders.

That gives shareholders of other companies good reason to care going forward.


> Though I'm not sure how we'd build a jail cell for an entire company.

Same thing with AI. You can't punish an AI, it has no body.


At least with AI you could do something like, destroy all copies including backups, destroy all training data and other code used to generate it. Which to me actually doesn't seem unreasonable punishment.


We must demand both financial and criminal liabilities against the perpetrators! Get the torches and pitchforks out! We need to teach them a lesson!


I did not mean to imply this, as there's a very long culpability chain. For this reason, I'm not sure if it makes any sense to imprison individuals for this. A lot of people playing a part in this causing such chaos.

But it is something to be very aware of for those of us who develop software run in e.g. hospitals and airlines, and should receive more attention, instead of only bringing up financial losses which is what usually happens. I noticed the same with the big ransomware attacks.


Indeed, pity that we need major failures like these, for goverments to finally start paying attention to give the same kind of laws as anything else, instead of careless EULAs and updates without field testing.


It's very bizarre to me how normalized we have made kernel-level software in critical systems. This software is inherently risky but companies throw it around like it's nothing. And cherry on top, we let it auto-update too. I'm surprised critical failures like this don't happen more often.


I can't tell if you're serious or sarcastic, but there is such a thing as criminal negligence.

CrowdStrike knows that their software runs on computers that are in fricken hospitals and airports, they know that a mistake can potentially cause a human death. They also know how to properly test software, and they know how to do staggered releases.

Given what we know now, it seems pretty likely that to any reasonable person, the amount of risk they took when deploying changes to clients was in no way reasonable. People absolutely should go to jail for this.


Also corporate manslaughter, in some countries: https://en.wikipedia.org/wiki/Corporate_manslaughter

This more or less originated with the unfortunately named MS Herald of Free Enterprise sinking (https://en.wikipedia.org/wiki/MS_Herald_of_Free_Enterprise) - after that incident, regulators decided that maybe they didn't want enterprise quite as free as all that, and cracked down significantly on shipping operators (though the attempt to prosecute its execs for corporate manslaughter did fail).


I made a separate (longer) comment about this..

Why don't orgs test their updates? Every decent IT management/governance under the sun demands that you test your updates. How the hell did so many orgs that are ISO 2700x, COBIT, PCI-DSS, NIST CSF, etc. certified failed so hard??

(ToS/contracts will probably get you out of any damages.)


Testing for most organizations is usually either really, incredibly expensive or an ineffective formality which leaves them at more risk than it saves. If you aren’t going to do a full run through all of your applications, it’s probably not doing much and very few places are going to invest the engineer time it takes to automate that.

What I take from this is that vendors need a LOT more investment in that work. They have both the money and are best positioned to do that testing since the incentives are aligned better for them than anyone else.

I’m also reminded of all of the nerd-rage over the years about Apple locking down kernel interfaces, or restricting FDE to their implementation, but it seems like anyone who wants to play at the system level needs a well-audited commitment to that level of rigorous testing. If the rumors of Crowdstrike blowing through their staging process are true, for example, that needs to be treated as seriously as browsers would treat a CA for failing to validate signing requests or storing the root keys on some developer’s workstation.


> Why don't orgs test their updates?

Because historically orgs have been really bad with applying updates: either no updates or delayed updates resulting in botnets taking over unpatched PC's. Microsoft's solution was to force the updates unconditionally upon everybody with very few opportunities to opt out (for large enterprise customers only).

Another complication comes from the fact that operating system updates are not essential for running a business and especially for small businesses – as long as the main business app runs, the business runs. And most businesses are too far removed from IT to even know what a update is and why it is important. Hence the dilemma of fully automated vs manually applied and tested updates.


> Microsoft's solution was to force the updates unconditionally upon everybody with very few opportunities to opt out (for large enterprise customers only).

Not a Microsoft's fan, but this is not true. Everyone who has Windows Server somewhere, with some spare disk space for the updates, has this ability. Just install and run WSUS (included in Windows Server) and you can accept/reject/hold indefinitely any update you want.


Not disagreeing, however:

1) the prevailing majority of laptop and desktop PC installations (home, business and enterprise) are not Windows Server;

2) kiosk style installs (POS terminals, airport check-in stands etc) are fully managed, unsupervised installations (the ones that ground to a complete halt today) and do not offer any sort of user interaction by design;

3) most Windows Server installations are also unsupervised.


> 1) the prevailing majority of laptop and desktop PC installations (home, business and enterprise) are not Windows Server;

They are not, but the point is elsewhere: that Windows Server is going to provide the WSUS service to your network, so your laptop and desktop installations (in business and enterprise) are going to be handled by this.

Homes, on the other hand, do not have any Windows Server on their network, that's true.

As a hack to disable Windows updates, it is possible to point it to a non-existing WSUS server (so that can be done at home too). The client will then never receive any approval to update. It won't receive any info wrt available updates either.

> 2) kiosk style installs (POS terminals, airport check-in stands etc) are fully managed, unsupervised installations (the ones that ground to a complete halt today) and do not offer any sort of user interaction by design;

That's fine; this is fully-configurable via GPO.

> 3) most Windows Server installations are also unsupervised.

See 2.


IMHO law should require such a firm, or any firm that may impact millions of other people, i.e. including all OS developers and many others, to maintain a certified Q/A process, maintain a 24/7 coverage and spend X% on Q/A. Such companies should never be allowed to deploy without going through a stringent CD procedure with tests and such, and they need to renew the certificate annually.

These are infra companies. Their incompetence can literally kill people.


My point/problem is that EVERY company (sorry for the caps) that is ISO, PCI, COBIT, NIST CSF, etc. compliant MUST be doing this!! (again sorry for the caps)

So they drop half the 'safety' procedures once the auditor goes away? WTF! (I am semi-angry because there are so many easy solutions and workarounds to not fall for this!! (inside screaming).

How irresponsible must someone be to roll out something to 1k-5k-10k machines without testing it first??

Hubris-Atis-Nemesis-Tisis!!!!

https://www.greecehighdefinition.com/blog/hubris-atis-nemesi...


I hope eventually law regards these companies as "infrastructure" companies, just like companies that build roads, bridges and such, that may and will kill people if not run professionally.

I'm not trying to enforce certifications because as a dev certifications always raise a bitter taste in my mouth. But those companies need certified processes that get re-certified every year. Sometimes even a cursory review from outsiders can find a lot of issues.


What you described is not a “CD” procedure. Lack of precision around such terms is part of the problem here.


I thought that is a deployment issue? Or maybe a QA one because looks like no QA has been performed...


Updates do get tested. Windows updates can be held and selectively rolled out when a company is ready. As far as I can tell though, CrowdStrike doesn't give companies the agency to decide if updates should be applied or not.


The updates should be rolled out incrementally rather than all at once


Since we live in a capitalism, financial losses are the only one anyone cares about at scale. What's a human life worth nowadays? About 10 million for a healthy prime age adult? Negative for elderly?


I think it depends what passport etc. you hold... One dystopian take is the trolley problem, where the self-driving car in question uses smartphones to determine the identity of the people involved, to work out who is cheaper to kill.


That reminds me of why McDonalds got such a high penalty in the court case everyone remembers as "person sues for spilling hot coffee on themselves".

The reason this reminds me of that, assuming that I remember right, is that I think they had even taken the decision that the cost of paying lawsuits for those injuries was lower than the increase in revenue for being able to say "we have the hottest coffee"… and that was why they were deemed so severely liable.

They were definitely shown to have known it was resulting in injuries from other settlements:

https://en.wikipedia.org/wiki/Liebeck_v._McDonald%27s_Restau...


Not true. Making C-level executives of software companies criminally liable with the chance to go to jail did change their behaviour in some recent lawmaking situation (forgot which, sorry).


None whatsoever, their contracts with customers will limit liability to the price paid for the software/subscription. If there was open-ended liability for software failures then very little software would get written.


Caveat to this: In the UK and many other countries, you cannot limit liabilities that cause death or personal injury arising from negligence.


Yeah but if it's a hospital, they should be able to operate without these IT systems. Nothing critical / life-or-death / personal injury should rely on Windows / IT systems.


> they should be able to operate without these IT systems.

Is that even possible any more? (That said, "operate" isn't a boolean, it's a continuum between perfect service and none, with various levels of degraded service between, even if you mean "operate" in the sense of "perform a surgical operation" rather than "any treatment or care of any kind").

All medical notes being printed in hard-copy could be done, that's the relatively easy part. But there's a lot of stuff which is inherently IT these days, gene sequencing, CT scans, etc., there's a lot that computers add which humans can't do ourselves — even video consultation (let alone remote surgery) with experts from a different hospital, which does involve a human, that human can't be everywhere at once: https://en.wikipedia.org/wiki/Telehealth

> Nothing critical / life-or-death / personal injury should rely on Windows / IT systems.

If you think that's bad, you may want to ensure you're seated before reading this about the UK nuclear deterrent: https://en.wikipedia.org/wiki/Submarine_Command_System


Also Silicon Valley: AI will replace doctors and nurses.


Why? Because you simply wish it to be so?


Because the suppliers of IT systems (eg Microsoft, Crowdstrike) do not agree that they can be used for life-critical purposes

If someone is injured or dies because the hospital has inadequate backup processes in the event of a Windows outage, some or maybe all liability for negligence falls on those who designed the hospital that way, not the IT supplier who didn't agree to it.


If your assumptions rest on corporate entities or actual decision makers being held legally liable, then you've got a lot of legwork ahead of you to demonstrate why that's a reasonable presupposition.


Because it's evidently a bad idea and there are reasonable alternatives.


That’s easy for you to say, with the benefit of recency bias, and with presumably zero experience in running a hospital.


That's not about experience, that's about following the regulated standards. This is well known ever since technology (not computers) got into hospitals.


None of the points you mention detracts from the correctness of his/her statement.


And? People and institutions constantly make bad decisions for which there are reasonable alternatives, and that's assuming that the incentives at play for decision makers are aligned with what we would want them to be, which is often not the case. Not that that ends up mattering much except as an explanatory device, because people and institutions constantly pursue bad ideas even seen in terms of their own interests.


It would be like orthopedic surgeons heading down to harbor freight to pick up their saws instead of using medical grade versions.

The tool isn't fit for purpose


Because you should always have a backup.


When has a software company successfully been sued (or settled) over this liability?


From windows tos:

Disclaimer. Neither Microsoft, nor the device manufacturer or installer, gives any other express warranties, guarantees, or conditions. Microsoft and the devicemanufacturer and installerexclude all implied warranties and conditions, including those of merchantability, fitness for a particular purpose, and non-infringement. If your local law does not allow the exclusion of implied warranties, then any implied warranties, guarantees, or conditions last only during the term of the limited warranty and are limited as much as your local law allows. If your local law requires a longer limited warranty term, despite this agreement, then that longer term will apply, but you can recover only the remedies this agreement allows.


"We give you no guarantees, unless the local law says we have to give them to you, in which case we do."

So they might get sued on a local level?


It doesn’t really matter what the contract says. Laws take precedence over contracts. For example, Boeing’s liability for 737 airliners that crash due to faulty software certainly isn’t limited to the price of the planes.


But only $243.6M for fraud, which caused death of 346 people.


Yes, software industry as we know would not exists if companies where held liable for all damages. But in the current state of affairs they have little incentive to improve software quality - when incident like this happens they can suffer an insignificant short term valuation loss but unless it happens too often they can continue businesses as usual.

Many companies paying lip service to quality/reliability but internal incentives almost always go against maintenance and quality of service work (and instead reward new projects, features e. t. c.).


> Yes, software industry as we know would not exists if companies where held liable for all damages.

Of course it would. Restaurants are held liable for food poisoning, but they still operate just fine. They just - y’know - take care that they don’t poison their customers.

If computer systems were held liable, software would be a lot more expensive. There would be less of it. And it would also be better.

I think I can get behind that future.


I like that future too, but to play devil's advocate:

Write me software that coordinates all flights to and from airports, capturing all edge-cases, that's bug free. Then tell me the number you estimate and the number of years to roll this out.


Sure, but ... thats not a spec. Specs have clear goals and limited scope. "All flights from all airports forever" is impossible to program, full stop.

The right way to write code like that is to start simple and small - we're going to service airports X, Y and Z. Those airports handle Q planes per day. The software will be used by (this user group) and have (some set of responsibilities). The software engineers will work with the teams on the ground during and after deployment to make sure the software is fit for purpose. Someone will sign off on using it and trusting its decisions. And lets also do a risk assessment where we lay out all the ways defects in the software could cost money and lives, so we can figure out how risk averse we need to be.

Give me scope like that, and sure - I'll put a team together to write that code. It'll be expensive, but not impossible. And once its working well, I'd happily roll it out to more airports in a controlled and predictable manner.


Crowdstrike's stock closed at $343 yesterday, I imagine that and MSFT are going to be cratering later this morning.


Pro tip: your stock can't go down if you crash the stock exchange


It honestly did not occur to me. In all seriousness, was stock exchange ever really hacked ( not just data exfiltration -- write access to everything )?


Trading has been halted on stock exchanges due to technical issues many times. But there's are also more than one stock exchange.


No, it can't, if there is no stock exchange online to process the prices.


"Tell me, Mr. Anderson, what good is a phone call when you are unable to speak?"


Pretty good time to buy MSFT I would imagine, given that this isn't really their fault.


So far MSFT is down by ~2%... Even Crowdstrike is only -20%. When they probably did more damage in a day their entire net worth.


I'm mystified it's not much lower. Perhaps the market hasn't really priced in the damage yet.


Yeah, if I had a spare million, I can imagine buying that dip.


I'd expect crowdstrike to take a big hit. Between this and the russian hack [edit: actually not, sorry, confused with SolarWinds], I am not sure they are not causing more problems than they solve.


Crowdstrike was hacked by Russians?


Sorry I confused them with SolarWinds. Strike that


it hovers around -20% in pre-market (at the moment)


MSFT will be fine. They are riding the AI waves, this is not meaningful, especially since they are not at fault.


The waves that are already looking like a storm in a teacup ?

There is no 'AI', that is always only hype. There is machine learning, which is a very powerful technology but I doubt MSFT will be leading that revolution. As for LLMs, MSFT might have some competitiveness there but I doubt it's going to be a very lucrative market. MSFT is highly overvalued.


<< There is no 'AI', that is always only hype. There is machine learning, which is a very powerful technology

I agree with you on the technical aspect, but the distinction makes regular people eyes glaze over within 5 seconds of that explanation. AI as a label for this is here to stay the same way cyber stopped meaning text sex of IRC. The people have spoken.

<< MSFT is highly overvalued.

Yes, but so is NVDA, the entire stock exchange and US real estate market. We are obviously due for a major correction and have been for a while. As in, I actually moved stuff around in my 401k to soften the blow in that event 2 years ago now. edit: yes, I am a little miffed I missed out on that ride.

So far, everything was done to prevent hard crash and in the election year, that is unlikely to change. Now after the election, that is another story altogether.

<< I doubt MSFT will be leading that revolution.

I think I agree. I remain mildly hopeful that the open model approach is the way.


https://www.aqr.com/-/media/AQR/Documents/Whitepapers/Unders...

You should stop trying to predict the next crash. According to the study, most people (including institutional investors) consistently believe there is a >10% chance the market will crash in the next 6 months when historically the probability is only 1%


<< You should stop trying to predict the next crash.

Hmm? No. I will attempt to secure my own financial interest.

<< According to the study, most people (including institutional investors) consistently believe there is a >10% chance the market will crash in the next 6 months when historically the probability is only 1%

Historically is doing a fair amount of work here. I would argue there is little historical value to the data we face. Over the past few decades we went through through several mini revolutions ( industrial, information and whatever they end up calling now ) in terms of how we work, eat, communicate and, well, live.

All of these have upended how humans interact with the world effectively changing the calculus on the data that preceding it if not nullifying it altogether in some ways.

Your argument is to stop worrying since you are likely wrong anyway, by a factor of 10. I am saying is 1935 people also thought they have time to ride the wave.

edit: ok, need coffee. too many edits


> Now after the election, that is another story altogether.

Agree. First half of 2025 could be pretty spectacular (if/when we get through 2024).

I suspect there might be some pretty radical plans for US debt monetisation being drawn up, to be implemented early in the new presidential term.


My brain goes there too, but the other part of my brain says "line always goes up." The richest among us are heavy owners of stocks, and this country does everything it can to keep those numbers up. Look at that insane COVID V-shaped recovery that happened. That's just not a real/natural market reaction in my book.


The worst part is that I get the need to do something to rein it in, but I get the feeling it will, as always, not be the actual rich ( owns color blue rich level ), who will suffer from those plans. There are less and less moves the government has as time progresses.


It may not be their fault directly but it is causing Windows systems to bluescreen, which IS their fault and their responsibility, ultimately.


How is it their fault and responsibility? Isn’t falcon sensor basically running like a kernel module? Does it mean that Windows is not engineered properly when it can be crashed by this?


Are you saying that they should prevent or limit the ability of their users from installing third party software? Or at the very least prevent it from running in kernel mode?


A more reasonable claim would be that microsoft should have a way to allow virus-scanners to run without needing to be able to crash the kernel.

That isn't an easy thing to do, but it should be possible.


I don't think that is possible. How can an anti-virus not in kernel mode defend against viruses running in kernel mode then?


Ebpf can, I believe, not crash the Kernel


Windows blue screen was never Microsoft's responsibility. /s


That’s what the license agreement says. Wait till every man and his dog sues them.


This is an insane take. Do you think other industries get away with limiting their liability to the product cost? No, because that doesn't provide adequate incentives for making a safe product. The amount of software that gets written depends mostly on the demand for that software. Even if Micrososft would not be willing to up their game to make the risk viable then someone else would.


The thing is we know how to make (eg) food that is safe or to a lesser extent bridges that don't fall down. If you sell food that makes people sick you should have known how to avoid that and so you can be held liable.

We don't have a good idea how to make software that is flawless, at least, not at scale for a cost that is acceptable. This is changing a little bit now with the drive by governments to use memory-safe languages, but that only covers a small part of the possible spectrum of bugs in software and hardware.


Nothing is without flaws, it's about limiting risk to an acceptable amount. Critical software should be held against higher standards.


What's "critical software"? Software controlling flight systems in planes is already held to very high standards, but is enormously expensive to write and modify.

In this case it seems most of the software which is failing is dull back office stuff running on Windows - billing systems, train signage, baggage handling - which no one thought was critical, and there's no way on earth we could afford to rewrite it in the same way as we do aircraft systems.


Something that has managed to ground a lot of planes and disable emergency calls today is in fact critical. The outcome of it failing proves it is critical. Whatever it is.

Now, that it was not known previously to be critical, that may be. Whether we should have realised its criticality or not, is debatable. But going forward we should learn something from this. So maybe think more about cascading failures and classify more things as critical.

I have to wonder how the failure of billing and baggage handling has resulted in 911 being inoperative. I think maybe there's more to it than you mention here.


Agreed, there is no such thing as perfect software.

In physical world, you can specify a tolerance of 0.0005 in but the part is going to cost $25k a piece. It is trivially easy to specify tolerance, very hard to engineer a whole system that doesn't blow the cost and impossible to fund.

Great software architectures are the ones that operate cheaply, but are bulletproof when software fails. https://en.wikipedia.org/wiki/Chaos_engineering


Given how widespread the issue is, it seems that proper testing on Crowdstrike's part could have revealed this issue before rolling out the change globally.

It's also common to rollout changes regionally to prevent global impact.

To me it seems Crowdstrike does not have a very good release process.


> but is enormously expensive to write and modify.

We're talking about critical software. If we can't afford to reach the level of safety needed because it's too expensive, well so be it.

Besides, the enormously expensive flight systems don't seem to make my plane ticket expensive at all...


There's only one piece of software which (with adaptations) runs every Airbus plane. The cost of developing and modifying that -- which is enormous -- is amortized over all the Airbus planes sold. (I can't speak about Boeing)

What failed today is a bunch of Windows stuff, of which there is a vast amount of software produced by huge numbers of companies, all of very variable quality and age.


I meant critical software a short-hand for something like: quality of software should be proportional to the amount of disruption caused by downtime.

Point of sale in a records store, less important. Point of sale in a pharmacy, could be problematic. Web shop customer call center, less important. Emergency services call center, could be problematic.


I, as a producer of software, have effectively no control over where it gets used. That's the point.

Outside of regulated industries it's the context in which software is used which determines how critical it is. (As you say.)

So what you seem to be suggesting (effectively) is that use of software be regulated to a greater/lesser extent for all industries... and that just seems completely unworkable.


What you're describing is a system where the degree of acceptable failure is determined after the software becomes a product because it is being determined by how important the buyer is. That is backwards and unworkable.


It isn't, though. "You may not sell into a situation that creates an unacceptable hazard" is essentially how hazardous chemical sale is regulated, and that's just the first example that I could find. It's not uncommon for a seller to have to qualify a buyer.


I think the system is rather a one where if you offer critical services then you're not allowed to use a software that hasn't been developed up to a particular high standard.

So if you develop your compression library it can't be used by anyone running critical infra unless you stamp it "critical certified", which in turn will make you liable for some quality issues with your software.


I assume you mean "if the buyer will use the software in critical systems."

That's very realistic and already happens by requiring certain standards from the resulting product. For example, there are security standards and auditing requirements for medical systems, payment systems, cars, planes, etc.


> Software controlling flight systems in planes is already held to very high standards, but is enormously expensive to write and modify.

Here's something I don't understand: those jobs pay chump change compared to places like FB and (afaik) social networks don't have the same life-or-death context


Hence, Windows should blue/green kernel modules and revert to a past known good version if things break


Would not shock me for AV companies to immediately work around that if it were to be implemented. “You want our protection all of the time, even if the attacker is corrupting your drivers!”


This seems like the kernel module was faulty for some time. The update only changed the input data for the module.


Crowdstrike should have higher testing standards, not every random back-office process.


> Software controlling flight systems in planes is already held to very high standards, but is enormously expensive to write and modify.

Boeing disagrees.


We don't know how to make general software safe, but we do know how to make any one piece of software safe. If you're software is going to be used as infrastructure then it should be held to the same standards. If you don't want it to be treated as infrastructure don't sell it to hospitals.


Mixing up the responsibility, in your world hospitals shouldn't purchase it.


Responsibility can be shared.


How about people in charge of choosing these clown solutions - both crowdstrike and windows?


Windows being clown solution? Out of touch with reality is huge here


I can't imagine starting a project from ~2010 and on while choosing Windows as the stack.


The production simplicity of having a standardized OS and being able to drop in a .exe and have it run everywhere without worrying about building for 1000 system combinations cannot be beat.


Enterprise Linux can fairly consistently be assumed to be RHEL, Ubuntu, or SuSE, with the first two being far more likely in the U.S. That’s not that much to ask for.


That's... not reality even on desktop PCs, and never was. If your business is more complex than selling hot dogs or ice cream (or even that on big enough scale), IT of such company will become a small monstrosity over time, and complexity of such deployments on Unix vs Windows is nothing compared to overall picture.


I see you somehow avoided learning what dll hell is, what various .net runtime incompatible versions are and what optional compatibility levels windows 10 offers.


Easily done if you target x86-64 statically


Clowns are taking over reality, on many levels. And they will tell you that you are the clown. Welcome to clown world.


Such a naivete


In your world i should switch my modest 1000 seats over to Linux desktops?

I'm not sure how i'm going to explain the productivity loss and retraining costs to the board if im honest.


Plus, CrowdStrike runs on Linux as well. _This time_ they only crashed Windows devices, but there's no guarantee that switching to Linux would prevent any of it.

You can switch away from CrowdStrike but I doubt you'll be able to convince whoever mandated CS to be installed to not install an alternative that carries exactly the same risks.


>CrowdStrike runs on Linux as well. _This time_ they only crashed Windows devices, but there's no guarantee that switching to Linux would prevent any of it.

In fact there was a recent CrowdStrike-related crash in RHEL:

https://old.reddit.com/r/crowdstrike/comments/1cluxzz/crowds...

https://access.redhat.com/solutions/7068083


At least on Linux it runs on eBPF sniffing so the chances of fudging something are lower. There are some supported Linux distributions where they also have a kernel module and there might a higher chance of that exploding.


No you should switch over to Chromeos, iPads, ... anything but Microsoft.

Crowdstrike only exists because Windows and other Microsoft products are so insecure their default configuration.


There's nothing special about Windows beyond the fact that you can run arbitrary executable files. The problem could just as easily have happened for Linux or iOS/Mac and in fact it has. ChromeOS kind of works if you want to run a web application that's hosted on some web server... but it's not appropriate for running programs where a dumb browser doesn't suffice.


What defaults would those be, and how would you change them?


I'm not in IT anymore and we run 100% macs, so serious question here: isn't nearly everything a webapp nowadays? Every "non dev" thing that I have to do for work happens in my browser or an electron app. I guess maybe MS Office apps may be the biggest hitch? We use Google Workspace and that's all in browser.


Legacy apps are quite common. I have recently been doing IT for State Farm Insurance.

Every State Farm insurance office in the country is still using a DOS App from the 1980's to run their office.


I'm interested what these DOS applications are running on. Is it virtualised or a real physical machine?


Not at all. "Industry", think: manufacturing is still big on desktop applications.


Shouldn't be too hard to bundle them together with qemu, or some other vm solution.


None of my enterprise ERP/PLM/CRM systems run on Mac Server OS


There are actually web versions of the office suite now.


It's horrible to use though. Google's suite is somewhat better than MSFT's web one, but it still is weak compared to any established desktop office suite, even libreoffice.


I've found it alright to be honest. I'd like to use libre office but the incompatibilities with .docx make it too annoying. Finally I can easily work with .docx on Linux, thanks to the web version :)


It's only good for viewing and simplistic editing. More complex stuff ends up being unavailable on the web version very often.


Your 1000 seats crashing won't prevent airplanes from landing.

These things should have gone from mainframes of yore to various unix systems, ideally a mix of different unix systems in hot failover.

Without running uncontrolled "agent" software of course.


Enjoy the circus then!


I dont think you can hold Microsoft liable for 3rd party software pushing its own update. Microsoft didn't make anyone install Crowdstrike or it's update files.


Some people in the comments claim CS was used for compliance reasons. Some others claim Windows & CS do not offer warranties. How can a product satisfy the compliance check-box, if it does not offer the warranty and not accept liability for the related features?


While software is often warranted, contracts won't often accept liability in terms of business damages etc, and that's not usually a requirement for compliance.

If it was, it would also make it impractical for a small business to contract with a large one because of risk.


Depends. I'm at an EMR maker; our Windows machines (as well of those of our clients - read: hospitals and doctors offices) are down. That is, of course, bad for the patients under their care.

Do these clients have SLAs? If so, they're definitely on the hook for something. You could probably get a few businesses together for a decent class-action against Crowdstrike. You're then expecting a lawyer to be able to convince a dozen semi-random people with varying degrees of computer knowledge that Crowdstrike's software was negligently designed, developed, and deployed in a way that caused financial or life losses for customers.

So, really, it's a coin flip.


What if your company mandated your customers run crowdstrike in order to run your software? What are the legal implications of that? Wouldn't that also put your contracts on the hook?


Prison time for the CEO and board of directors would be nice.

Enough of this limited liability nonsense, there need to be serious, severe, life-changing consequences.


Hypothetically even if they were liable they would bankrupt before even a few percent of damages is recovered. You cannot pluck a bald chicken.


>they would bankrupt before even a few percent of damages is recovered

Wouldn't that be the desirable outcome, though? Given the amount of damage they have caused, they should cease to exist.


Sort of. They need to be sued into bankruptcy. Current shareholders get completely zeroed out; the company still exists, but is sold to the highest bidder with the proceeds paid out to affected customers.

We need this so that every company board is always asking "are we investing enough to make sure this never happens to us?"


A local rooflayer is absolutely corrupt. He cheats every customer, produces leaky roofs, doesn't even pay taxes completely.

It takes 2 year for the legal system to catch up, at which point he starts a new company, bankrupts the old one, sells all his tools cheaply to the new company, and fires and rehires his workers. I've seen this game going on for 14 years now.

I think Crowdstrike would do the same: Start a new one, sell the software, fire and rehire the workers, then go on as if nothing happened


I'd call BS on this story, but I know a friend that bought a home a few years back from a homebuilder that did a similar thing, except at a whole home level. Absolute disaster. he's been chasing him for half a decade now via legal means to get things fixed.


Not really though. Whether they should continue to exist into the future should depend on if the expected positive value of their services in that future exceeds the expected damage from having a big meltdown every once in a while. That some of their devs made a fuckup doesn't mean the entire product line is now without merit.

Killing the company because they made a mistake doesn't just throw away a ton of learned lessons (because the devs will probably be scattered around the industry where their newly acquired domain knowledge will be less valuable) but also forces a lot of companies to spend resources changing their antivirus scanners. For all we know, Crowdstrike might never fuck up again after this and forcing that change would burn hundreds of millions for basically no reason.


"Whether they should continue to exist into the future should depend on if the expected positive value of their services in that future exceeds the expected damage from having a big meltdown every once in a while"

I don't think that's right, since it ignores externalities.

You want to create a system where every company is incentivized to make positive security decisions. If your response to a fuckup of unprecedented scale is just "they learned their lesson, they probably won't do that again", then the message these companies receive is that it is okay to neglect proper security procedures, because you get one global economic meltdown for free.


This is where public executions of executives help.


But the Ticketmaster software would buckle under the strain :-)


>You cannot pluck a bald chicken.

Haven't heard that one before but I love everything about that!


Financial liability often doesn't equate to actual recovery of damages


Yes, SLA. No one gets held liable if the legal is done correctly and there were no guarantees, but on cloud there is 100% SLA so they will pay out.


Do we not remember "Ma" Bell?This should perhaps be a wakeup call in regards to Microsoft and other large tech having concentrated fingers in too many pies. This appears to be an anti-trust issue at its core.

Was it really a botched update? Or was it a test run for holding the world hostage prior to a coup?


Negligence at Crowdstrike is not covered by any SLA. Even if insured, Crowdstrike could be fucked. Let alone, companies going to try and how much cost this has. Long term, their fucked.


There will be no Crowdstrike left after this. I am just upset I cant short it…


> Chances if Microsoft or Crowdstrike will be held liable for financial losses caused by this outage?

Zero. Exactly Zero.

Clearly you have never been involved in buying insurance or writing contracts for IT products/services.

Loss of contracts, profits, goodwill, economic loss, loss of data and all that jazz is excluded in whole or limited to a fixed monetary value.

It is known as indirect, consequential or special loss, damage or liability.

No lawyer worth their salt will let an IT product/service company draft a contract that does not have the above type of clause..

And good luck finding an insurance contract that will pay out for such losses, indeed most of them have conditions that state your contracts with customers must exclude or limit such losses.

Most software also has clauses excluding use in safety critical environments.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: