Boeing's software fix, announced today, is to compare readings from both angle-of-attack sensors and disable MCAS if they disagree significantly. The obvious question is why they didn't do this in the first place?
One possibility is incompetence. But Boeing engineers are smart people, so I'm not convinced by this. The elephant in the room is the requirement to maintain a common type rating with older 737 models.
Suppose they did originally do what the fixed software does now, and disable MCAS if the AoA sensors disagree. The problem Boeing face is that with MCAS disabled when this occurs, the plane no longer flies like an older 737. They'd need to announce to the pilots an AoA disagree, and announce that MCAS was disabled. Now what? A pilot certified and trained on the older 737 would not know how the Max now differs from what they trained on. If they'd done this, they'd have needed to provide additional training, and this must have concerned Boeing management that it might jeopardize the common type rating. Hence it seems likely they didn't add the AoA sensor comparison for this reason, reasoning that it was unlikely to be a problem anyway. We now know that reasoning was flawed.
What does this mean going forwards? Will EASA and other CAAs refuse to certify the modified 737 Max under the same type rating as the older 737? This certainly seems possible. If they did require a separate type rating, this would likely kill 737 sales, regardless of whether the plane is now safe.
> One possibility is incompetence. But Boeing engineers are smart people, so I'm not convinced by this.
That's still a possibility. Stupid decisions can emerge out of smart people.
Boeing is huge, and what they develop is incredibly complex. There are a lot of people with differing level of competence, ethics, and goals.
For example (I am not saying that happened), the engineers designing MCAS didn't expect incorrect AoA data, thinking the checks were done elsewhere. At the same time, the "sensors" team thought that raw, unchecked data was expected. The integration guy didn't read the specs correctly (sometimes, it comes down to a single word), didn't catch that, and checked the OK box. His manager, focused on a more pressing issue took that as granted and it went to production.
It is possible that the engineers did an excellent work, but didn't question the specs they had. The integration guy is normally super reliable but he just had a bad day. And his manager handled the other problem beautifully and overlooked the MCAS/AoA because, normally, the integration guy is reliable. A series of small mistakes that ended up in a catastrophe.
There are a lot of safeguards but the complexity is so high that sometimes, something goes through. Especially if the company is under pressure.
> For example (I am not saying that happened), the engineers designing MCAS didn't expect incorrect AoA data, thinking the checks were done elsewhere. At the same time, the "sensors" team thought that raw, unchecked data was expected. The integration guy didn't read the specs correctly (sometimes, it comes down to a single word), didn't catch that, and checked the OK box. His manager, focused on a more pressing issue took that as granted and it went to production.
> It is possible that the engineers did an excellent work, but didn't question the specs they had. The integration guy is normally super reliable but he just had a bad day. And his manager handled the other problem beautifully and overlooked the MCAS/AoA because, normally, the integration guy is reliable. A series of small mistakes that ended up in a catastrophe.
What you describe here would be a major failure of systems engineering on the project.
The systems engineers are responsible for flowing top level requirements down to the individual systems. They are responsible for ensuring the specs the engineering teams receive for their systems are correct, and for handling requests to change said specs. If the spec of the output of the AoA sensors does not match the spec flowed down to other teams on the input from those sensors the systems engineers responsible did not do their jobs.
Systems engineers exist to manage complexity like this, to ensure that the various engineering teams across the various disciplines are technically coordinated by providing clear, consistent specs and interfaces for them to work to. If that didn't happen, then I would not say those engineers are competent, let alone smart. It would be especially disappointing to me (as a former systems engineer) if this were the case, as systems engineering is the last place I would expect such incompetence from a company like Boeing. It's at the core of everything they do.
I want to address one specific point in a different context.
> the engineers designing MCAS didn't expect incorrect AoA data, thinking the checks were done elsewhere
You always expect out-of-spec conditions to be a possibility and have something in place to handle those conditions appropriately. To not do so is incompetence bordering on negligence.
I agree that someone with a title like Systems Engineer should be responsible for such issues, but (having worked at Boeing) not every group has systems engineers overseeing their work. Or they're working in a different city and you get a phone conference with them once a month.
This should be no surprise. Conway's Law in action. You can look at the issues that Boeing has and guess to a reasonable degree of accuracy what their organizational structures look like.
> You always expect out-of-spec conditions to be a possibility and have something in place to handle those conditions appropriately. To not do so is incompetence bordering on negligence.
I don't think it's possible to build an airplane if you have to expect this from every system. Any flying machine has many single points of failure. An F-15 once famously flew and landed after losing a wing. This is clearly out-of-spec but no aircraft considers this condition a realistic possibility or has contingency procedures for it.
Exercise for the reader (and maybe a good interview question for a manager): draw the organizational structure for a team to design and manufacture a new airliner. List the single points of failure on the aircraft, and point to who is responsible for them. Would you fly on this? What did you miss?
2 planes crashes in less than 5 months. So yes, there was either a system engineering failure, or a completely inept decision in choosing an oversized engine for an antic airframe. Your choice.
Or a training requirement failure. Or a UX failure. Or a documentation failure. Or an unrelated failure, given that the RCAs haven't been completed on the crashes.
It's amazing the hubris software engineers have in assuming everyone else in an idiot.
I am not an expert on aircraft-scale hardware/software codesign projects, aerospace engineering, high-reliability engineering (to aerospace standards), or complex systems analysis. I strongly suspect you, and most of HN, aren't either.
Boeing has people with all those skillsets. As does Airbus. As do NTSB and the variety of national certification agencies.
So before we toss rocks of certainty around in a comment thread, maybe wait and listen to the experts?
We can agree that not all skepticism is valuable skepticism though?
I came across this Rand report on the NTSB [1, 1999?]. The conclusions were not good about the funding & staffing levels vs modern accident load. Partly due to more incidents, but moreso due to increased systems complexity per incident.
So the NTSB, with a budget of around USD$100M, takes multiple years to deliver a report. And they're a professional worldwide standard on accident investigation. Suffice to say, we're not going to crack this case open on HN.
Which isn't to say it isn't productive to debate the relative merits of different regulatory approaches, takeaways for other disciplines, lessons learned, and all manner of things. But let's just have some humility in pretending we're all experts on everything [2].
How are those "rocks of certainty"? 2 planes crashed, resulting in deaths of hundreds of people. There is validity in questioning "rocks of certainty". These people were trusted in certifying the plan. These people were trusted in doing the correct choices after the first crash. Obviously, the "experts" did not do what they had to do.
There is nothing about a "Software Engineer" which precludes them from achieving a grasp of the finer points of physical engineering disciplines.
Finite element analysis, error propagation, systems analysis, statistical process control, mechanics of materials, and all the other equations we've come to rely upon are equally applicable by anyone with the wherewithal to learn to learn them, and are in fact, more likely to be picked up by someone who spends 80% of their time learning "the next tool".
In short, while I agree many software engineers may be in possession of an inflated sense of competence, there are many who are legitimate polymaths. This makes appeals of the sort you assert less than useful in the pursuit of substantive conversation.
Time is the fire in which we burn, understood. That's why we learn things. So we can predict things, and save time.
There's 75 + years in your lifetime. Even auto-didactically you should be able to sift through everything in
and use that as the starting point to build up some deep dives into authoritative literature.
When you can start to accurately predict outcomes, you're on the right track.
You'll know you're in the right neighborhood when you have an increasingly hard time tolerating poor models of reality, and start drifting off thinking about how easy it would be to make that thing if you only had the tools.
Happy adventures in the wonderful world of Math!
(Don't forget to read the Ethics handbook, and if you want to do it professionally, take the FE, PE, and never forget the iron ring!)
If you look at plane crash reports, the disaster is almost always the result of a chain of events. Taken individually, each event is almost insignificant. It's the combination that makes the disaster.
In the vase of the Boeing 737 Max the sequence would be something like:
- Boeing decided to fit an oversized engine on an older airframe, which can cause a dangerous stall in some unusual configurations. It is not that much of a problem, they just reduced the flight envelope to exclude these configurations. It is not an unusual thing at all.
- In order to make sure that dangerous situation is never encountered and to limit training expenses for the pilots, they used software. Then again, not unusual.
- In order to avoid writing entirely new software, they extended the functionality of existing software (MCAS). Again, fine.
- First problem: the MCAS, now a critical piece of software wasn't properly requalified.
- Second problem: The AoA sensor, now critical, isn't properly managed.
- Third problem: The pilots weren't properly trained on what to do when MCAS starts crapping out
- Fourth problem: The pilots didn't have enough skill to deal with the unexpected situation.
- Fifth problem: Maybe some pilots dealt with the problem correctly but the report wasn't properly made and accounted for.
Anyone of these steps done right would have averted a disaster. We can't really blame a single one.
Some investigative journalism indicates that this is not what happened.
The FAA has different levels of redundancy and uptime requirements based on the outcome of system failure. A system categorized as “catastrophic” failing would lose the plane and all passengers, while “hazardous” might hurt some and kill a few, and so on. The FAA requires that catastrophic systems must have a backup, while hazardous can go without one if it’s reliable enough.
Boeing classified the MCAS as “hazardous” only under certain flight characteristics, and they categorized it as a lower level than hazardous during level flight. This means that the crews designing the specification made the decision to depend upon only a single input, and it appears to have been built to spec[0].
It also appears that the FAA was experiencing internal pressure to delegate more certification authority to Boeing, with disastrous results.
> Boeing is huge, and what they develop is incredibly complex. There are a lot of people with differing level of competence, ethics, and goals.
That is true... however it's also true both of the big aerospace companies have an abundance of very smart engineers who by their very nature would have been responsible enough and inquisitive enough to notice this defect even if it was not their direct responsibility (good engineers will step out of their box).
Knowing how vital that sensor is (regardless of the stupid reason for it being vital) and not building in redundancy is such an obvious flaw to any engineer - The reason with almost certainty will have been because that multiple observations of this safety flaw would have been squashed by some big execs due to the business requirements and economics and a severe lack of morals.
Malignant cultures can do a lot to reduce this. The platonic ideal engineer will cross any number of organizational boundaries to deliver a needed piece of information, but there can be a lot of pressure to leave those boundaries un-crossed. Depending on how political the organization is, a strange engineer (or their manager) dropping in from nowhere to tell you a mistake was made may even be seen as a bad thing.
In addition aerospace, like most other traditional engineering fields, is heavily biased towards seniority. Years of experience actually matter[1], and more junior engineers often have an uphill battle trying to convince more senior engineers of errors. In aerospace in particular it's a side effect of having systems so complex that first impressions of a subsystem can give you intuitions that are wrong in the context of the larger system, even though they may seem right in isolation.
The Challenger disaster is a good example of an engineer who did warn the right people but they ignored him; I would bet bottom dollar something similar happened with these Boeing tragedies. The pressure to launch is great and who wants to be the person who stalls "progress".
Every single human on earth would agree that this is what should be done, but this is naive. Of course it should be done. How do we actually accomplish it?
good engineers will step out of their box
Good engineers are often prevented or discouraged from stepping out of their boxes by organizational structures.
Engineers also typically have intense workloads.
Getting a solid week's worth of engineering done is hard enough as it is.
Heroically crossing boundaries and leaping outside of organizational boxes to solve problems carries, at the very least, a high risk of falling behind on other work assigned to you.
In defense of organizations (and those who create them), coordinating hundreds or thousands of people is really hard. How do you create a strong enough structure to avoid chaos, while simultaneously allowing people to step outside/across that structure when the need arises? It ain't easy.
Also, and I don't know anything about how Boeing operates, but I suspect they already are successful at this to a large degree. Their record is not spotless, including some cover-ups they're guilty of, but it is very good. They would not be a world leader in aviation with highly competitive planes if they weren't pretty good at this stuff.
This problem does not take heroic leaps outside of boundaries and risk to careers... it's so obviously wrong that simply pointing it out should suffice, this isn't another challenger problem, this is staring anyone in the face blinding fucking obvious.
My point is that this _would_ have happened, people will have pointed it out, because it takes so little effort and because it's such an obviously wrong flaw. But some exec will have overruled them multiple times because money > safety.
That mistake of one team not talking to the other about AoA would have been caught at system level hazard analysis, which is a requirement for any half decent engineering organization churning out safety critical requirements. Heck it is part of required submission to FDA for medical devices which are considered even moderate safety risk, let alone an airplane.
>> It is possible that the engineers did an excellent work, but didn't question the specs they had.
The reason I find that scenario hard to believe is that MCAS very existence is to correct a design problem inherent in the plane. A safety problem - the plane is not safe without MCAS or at least without new training which they wanted to avoid.
Supposedly the history is that the single sensor input was decided when the MCAS could only exert a fairly small amount of change to the stabilizer. Later, the amount of control given to MCAS was drastically increased, apparently without considering the ramifications of a single sensor feed.
The explanation I’ve read (sorry, don’t recall the source) is that this system repurposed a purely advisory one that existed before, for informing the pilot about AoA. Being advisory in nature, it didn’t need duplicate sensors.
Next up: you need to implement MCAS and everything is conveniently there in existing code.
That, plus toxic culture for internal people doing FAA (yes, really), plus what you write, plus business pressure on certification...
What about integration/ subsystem level testing? It seems MCAS wasn't thoroughly tested with all boundary values. Whats the use of systems like IAHM [1] if they can't predict what faults can occur how safety is ensured/ guaranteed in various scenarios.
I agree with you that this was ALL about keeping type rating. I wish the government would offer a whistleblower award to anyone inside Boeing who could prove that this was indeed true especially since it seem that that is how the software originally operated. Companies will do whatever it takes to drive sales and revenue and stock price. Employees don't want to raise their hand and get fired as they have families to support. A true whistleblower program with WITSEC level provisions for protection and monetary support would help cut this down. Once it happens once or twice companies are very disincentivized to continue down this road.
The current generation of executives seem to be willing to gamble in this way. They know that as long as they have money, they won't see much (if any) prison and the chances of even getting to a trial will be minimal.
I don't think that good whistleblower protection would help any. The justice system needs an overhaul to be blind to someone's wealth and color.
I bet this would be virtually impossible to prove, and career suicide for anyone who tried it.
These aren't mustache-twirling villains who distribute memos that say "Let's ignore safety issues to get this approved faster". They really believe they're doing what's best for all involved. We'll save everyone time and money, and make it easier for pilots. How is that not a good thing? We have no reason to believe safety will be compromised.
Predicting the safety implications of design decisions, years in the future, is not an easy task. If the AOA sensors (I think?) were a tiny bit more reliable, we'd never have seen a problem, and the MAX program would be considered a great success in efficiency.
I'm sure we've all worked for managers who made decisions we disagreed with, but couldn't prove they were making the wrong one.
However, they did actively choose to not put in redundancy and the status systems in the cockpits which should have been done out of the box following any common sense and failure mode analysis procedures. This decision itself is enough to bring this to court and as a result the internal communications on these decisions will be explored.
I am not in aviation, but if they added redundant sensors and a new indicator, with an accompanying change to the flight manual for how to react to said indicator, wouldn't that have run contrary to the goal of "no new training/certification required"?
There are likely document retention policies that limit how long the relevant emails and text messages may be retained. There could be Word files or other documentation that captures discussions still floating around though.
Hopefully whoever is investigating this is acting fast to acquire the emails before they automatically get wiped.
From what I've read, the assessed severity of the failure of aviation systems is rated. The rating for this system was not severe enough to require redundancy based on the assigned rating. I'll update my comment if I can find a reference.
>These aren't mustache-twirling villains who distribute memos that say "Let's ignore safety issues to get this approved faster". They really believe they're doing what's best for all involved. We'll save everyone time and money, and make it easier for pilots. How is that not a good thing? We have no reason to believe safety will be compromised.
That is an incredibly naive reading of the situation. For starters, there is no such rationale as 'reason to believe,' this is a highly regulated process for good reason, which requires testing and verification.
Sure it could be true, but the far more likely motivation is along the lines of Dieselgate. And yes, it can be proven that managers make bad decisions that incur legal liability.
Naive? I worked at Boeing for a couple years, and was on a software team where I was regularly asked to do things which flew in the face of the known best practices of the industry. (My team is not to blame for this. It wasn't for the 737MAX, it wasn't avionics, and the project was cancelled long before it was at all usable.)
It's not as "highly regulated" as you might want to believe. They talk a good game about CMMI but if you try to improve something they remind you that CMMI is only about process, not quality. Hurry up and ship something (deadlines!), and if it's not perfect we'll find it in test.
Given that the company has this culture, I find it much more plausible that this is to blame for their product issues. I don't need to hypothesize a big evil conspiracy to explain bad software.
In fact, that's true of almost every software organization. The James Bond joke [1] fell flat precisely because you don't need a James Bond villain to get buggy software. It's what you get by default.
I think that's what the parent poster meant by "a true whistleblower program with WITSEC level provisions for protection and monetary support would help cut this down."
A lot of people in a lot of industries might come forward if they didn't have to commit career suicide to do it.
Sure, but you can't guess who might be able to prove it. Are you going to offer WITSEC to every engineer who happens to disagree with their manager? Half the programmers I've ever worked with were annoyed by management and thought they were being asked to implement terrible decisions.
I agree with you that this was ALL about keeping type rating. I wish the government would offer a whistleblower award to anyone inside Boeing who could prove that this was indeed true especially since it seem that that is how the software originally operated.
Why would a whistleblower award be necessary? Boeing has been very open about this being the reason for the MCAS system; it's been discussed in a number of articles from NYT and WaPo.
The thing they lied about was that they plane flew the same as the old 737s. So maybe I need to be more specific and say that someone inside Boeing that could prove they knew it flew differently and lied about it to keep the type rating. Keeping the type rating is fine as a goal as long as it is true. Two crashes and hundreds dead prove it is absolutely false.
As I understand a simple software fix is not possible according to regulation.
The problem is as follows, as you described it partly: 2 sensors are not enough. If the MCAS is an important part for the flight safety, a simple redundant safety system is not enough. Because an airplane is not about functional safety but mission critical safety. In functional safety, if there is an error the safety function is triggered and the system is transferred into a safe state. But there is no safe state here. If the system is mission critical, then it is not safe to assume to switch it off in case of an error. That means for mission critical system we need at least 3 readings and with a vote can decide on which reading is most likely the correct reading.
If the MCAS would not be part of the mission critical path, then we could ask why is there in the first place? There must be reason why it was introduced.
I assume, it is not done by a simple software update, if there are only 2 sensors. It will be partly redesigned to fit the requirements and regulations. But of course, this will not be publicly announced. Think about the share price. They will maintain a communication that assumes that this is an easy (and cheap) fix, a software update.
This question -- if it is a safety feature, why is it acceptable that it can it be disabled -- comes up a lot, so I think we should recognize that, in general, there is an answer, though whether it was the right answer for MCAS goes a step beyond.
Whenever you have something that would usually improve safety, but which presents a risk if it fails, then the rational response is to ask whether it demonstrably improves safety overall. This calculus depends on how much it improves safety when working, how much harm it does when failing, and how likely it is to fail. These considerations can be modified by limitations on operations both when things are working and when they fail, as is the case of twin-engined aircraft use on trans-oceanic flights.
If, in addition, this thing can be disabled, the principle is the same, but there are more cases: what are the chances it be disabled even though working, or (as in this case) not be disabled when failing.
Disabling MCAS does not itself put an airplane in a dangerous situation, it merely increases the risk somewhat. The part of the analysis that seems to have been flawed is that covering the risk introduced when it fails but is left engaged.
MCAS isn’t needed to fly the 737 MAX, you can perform a perfectly fine flight without it — the autopilot directly trims, pilots could fly via trimming themselves using the control on the stick, and there’s a manual wheel for emergencies when the jackscrew motor fails (or is disabled to override MCAS).
Rather, MCAS is needed to fly the 737 MAX like other 737 type aircraft.
MCAS has the capacity to override pilot control via continued trimming; MCAS is not a safety requirement, but can trigger a critical failure; MCAS was not built to critical failure specs; pilots were not trained on MCAS; MCAS is required to maintain the type rating, because otherwise the plane would handle differently during a stall. (This is the only purpose — tipping the nose to fake handling characteristics during a stall.)
It sounds like a reckless design, made out of political rather than safety considerations.
That's a bit of a misunderstanding. MCAS is needed to satisfy the certification requirements for an aircraft under part 25. It only makes the 737 Max more like the 737NG in that it also makes it more like any other certified aircraft.
To put it another way, the alternative to MCAS was not more training for the pilots. It was the 737 Max not passing certification.
My information suggests that while all modern aircrafts require trim control for stabilization, particularly near stall, the way that MCAS operates isn’t required. Rather, pilots would simply have to fly slightly differently to control the 737 MAX compared to the 737 NG. However, that difference in stall prevention would require the 737 MAX to be certified fresh, as opposed to the same type as the 737 NG.
In short, that MCAS trimming is only required if you’re not going to train pilots to trim correctly on the new airframe, because you’re trying to assert type compatibility.
FAR 25 has a number of specific requirements around pitch stability, specifically, FAR 25.255(b)(1) and FAR 25.203(a), and some others. FAR 25.203(a) says "No abnormal nose-up pitching may occur. The longitudinal control force must be positive up to and throughout the stall"
My understanding is that on the MAX, without MCAS, once you've pitched up beyond some AoA beyond 12 degrees or so, you can let go of the yoke, and the plane will continue to pitch up further until it stalls. That does not comply with the regs, and so you have MCAS which dials in some nose down trim in this situation to counteract the aircraft's natural tendency to pitch up further.
If the chart presented in the article below [1] is fairly accurate and not merely representational, I think that the 737 MAX, without MCAS, does not actually become statically unstable in pitch at least until after the stall, but the response was still unacceptable, and the reason MCAS was needed for certification.
There is a complication here in that the stick forces are generated by an elevator Feel and Centering Unit, which is fed by a dedicated pitot tube and the stabilizer position.
Pitch instability after the stall was part of the rear-engined jet deep stall problem, and the reason for stick pushers [2].
I admit it's not as authoritative as I'd like but this is the best source I've found:
> MCAS is a longitudinal stability enhancement. It is not for stall prevention or to make the MAX handle like the NG; it was introduced to counteract the non-linear lift of the LEAP-1B engine nacelles and give a steady increase in stick force as AoA increases. The LEAP engines are both larger and relocated slightly up and forward from the previous NG CFM56-7 engines to accommodate their larger fan diameter. This new location and size of the nacelle cause the vortex flow off the nacelle body to produce lift at high AoA; as the nacelle is ahead of the CofG this lift causes a slight pitch-up effect (ie a reducing stick force) which could lead the pilot to further increase the back pressure on the yoke and send the aircraft closer towards the stall. This non-linear/reducing stick force is not allowable under FAR §25.173 "Static longitudinal stability". MCAS was therefore introduced to give an automatic nose down stabilizer input during steep turns with elevated load factors (high AoA) and during flaps up flight at airspeeds approaching stall.
However, on Boeing's own website they give the 'makes it fly just like the NG' explanation. On balance of probabilities I think that's unlikely to be the engineering justification.
My reasoning being I don't believe that there is any formal requirement for an aircraft to exhibit the same handling behaviours to be counted on the same type rating. For example the 757 and the 767 shared a common type rating. The 757 was a pilots favourite precisely because it was sporty in comparison to the 767.
This is still a bit unclear to me. It's possible that it was originally intended to provide equivalent handling characteristics to the NG, but after flight testing, the magnitude of the MCAS correction increased by over 4X. It's possible that based on the predicted performance, the MAX would have met FAR 25 but not had equivalent handling to the NG, but after flight testing, MCAS took on a more important role in making the MAX certifiable at all.
The answer is FAR 25.672 (c)(2). "It must be shown that after any single failure of the stability augmentation system or any other automatic or power-operated system - ... The controllability and maneuverability requirements of this part are met within a practical operational flight envelope (for example, speed, altitude, normal acceleration, and airplane configurations) which is described in the Airplane Flight Manual"
MCAS is only required in high AoA, flaps up, manual flight. To get high AoA, you generally need some combination of slow speed, a steeply banked turn, and abrupt maneuvering. If you tell the pilots to keep the speed up, keep the bank angles down, and fly gently in the event of an MCAS failure, it meets the requirements.
From a practical standpoint, the aircraft should never see an AoA that would trip MCAS in normal flight. The only time a jetliner would typically see an AoA over ten degrees is on takeoff and landing, when the flaps are down (and MCAS is inhibited), or momentarily in turbulence.
From a risk control standpoint, you're multiplying probabilities together (of course assuming uncorrelated failures). Spitballing the failure probability chain, flights that would trigger MCAS are probably about 10^-4 to 10^-5 probability, and the lack of MCAS might cause an unrecoverable stall in perhaps 10^-1 to 10^-2 flights, so figure 10^-6 probability per flight of the airframe's poor stability leading to an accident. This is too high to be certified. If you have MCAS, but it's disabled when an AoA vane fails, figure the probability of an AoA vane failure is 10^-4, you get an accident probability of 10^-10, which is acceptable.
Boeing has delivered 376 737MAXes since May 2017. About as many were delivered in 2019 as 2017, so figure their median age is 9 months. Figure a month from delivery to going into full service. That makes for 90,578 flight days of service. Figure 6 flights per day, you get about half a million flights. We know that the AoA vane had a failure in at least 3 flights, and we should probably assume that there were another three failures on the alternate side. That gives a failure rate of 6/500,000 = 1.2*10^-5. That's about 8x better than my spitball 10^-4.
The reason two planes have gone down is that the MCAS cure is worse than the instability disease. MCAS incorrectly activates at 10^-5, and empirically this failure is 66% fatal. If you can get MCAS to only activate when it's needed, the fact that it won't always be there to protect you is not so crucial.
Think about it like this: Would it bother you if the airbag in your car only went off 99.9% of the time when you got in an accident? It shouldn't, and it would hardly make any difference in auto fatalities. On the other hand, if the airbag in your car randomly went off on 0.1% of drives you took, you'd have that happen to you once a year or so, and there's a good chance that it would eventually cause you lose control of your car and crash.
> If the MCAS would not be part of the mission critical path, then we could ask why is there in the first place? There must be reason why it was introduced.
It was added to meet certification requirements for handling at high angles of attack. Specifically, to counter an increased pitch up tendency due to the larger, more forward engine nacelles.
As to how mission-critical it is I don't know, but as I understand it the angles of attack where it would become active are not encountered in normal commercial flight. It's more of a system that is there to meet requirements for handling near the "edge of the envelope."
I read this thread earlier and became frustrated by the number of times people repeated the same old misunderstandings. Thanks for taking the time to try to correct them.
Having a critical system, whether mechanical, electric or hydraulic is an unavoidable fact of aviation. For example on the airbus if there is a complete failure of the fly by wire system the plan is to fly the aircraft using only the trim for pitch control and the rudder for lateral control. I'm sure pilots routinely give this a go in the simulator, but I'd be very surprised if the expected outcome was much above a controlled crash. It's a case of managing the risk of a problem against the hazards.
An AoA disagree warning will probably be written up as a fault requiring written authorization to depart from a maintenance base. The pilot would be expected to be extra vigilant in avoiding flight close to the stall until the error was corrected. Perhaps settling for easier approaches, longer runways etc.
Yes! But even if you have 2 out of 3 vote you can still have two sensors that can fail, say due to a bird strike or something else and MCAS would have to be disabled which means pilots need to learn to fly the plane w/o MCAS.
The one part of the narrative that Boeing has tried to Jedi Mind Trick away is that even if the pilots knew to disable MCAS by pulling the fuse, they're still left with having to fly a plane which no longer flies as the previous generation that they were trained on -- which is what MCAS was designed to do. So you still have a safety issue.
In normal flight, MCAS is not active. The plane handles pretty much like any other 737. MCAS becomes active at high angles of attack, near the stall, to meet handling requirements for certification.
Doesn't matter. The flight characteristics are still drastically different toward the extremes, and you never assume away the extremes. You now have to guarantee that every crew when faced with an MCAS disable situation that required piloting in a way that the MCAS would kick in can successfully mimic the action of the MCAS with only the manual trim wheels.
I'm not saying that's impossible, but you'd want your aircrew to have sufficient training to be a stand in for the automation.
Fully agree that Boeing should have disclosed MCAS, why it's there, and details of its operation to the folks flying the planes. If I were a pilot I would certainly want to know about such a system, even if its behavior is not normally something I have to deal with. Sort of like a stick shaker. It's there for safety near the extremes, but it's not something a pilot ever encounters during line flying.
I think "drastically different" is probably overstating the truth. It certainly doesn't jibe with anything I've read.
It's worth bearing in mind that, from what we know, neither of the two crashes had anything to do with handling difficulties in the new aircraft near the stall.
I say drastically different because no one pays the complexity tax of increased automation to deal with something that isn't important. In this case, to the certifiability of the airframe.
I understand your caution against overstating things, but I'm not trying to be hyperbolic. If you think about the consequences of doing basic maneuvers in adverse conditions (weather/load/engine power conditions), you rapidly begin to get into areas where what seems minor would get into uncharted territory without a working MCAS.
It is a different plane in terms of manual flying. The automation non-withstanding.
I get where you are coming from, and it's true that "the FARs are written in blood" but I'm not convinced that a non-linear stick force would have caused any real world issues. For a start, all of the critical phases of flight (takeoff, landing and go-around) usually involve some amount of flaps, for which MCAS is apparently unnecessary.
This is an interview with D.P. Davies, a test pilot of the Aviation Review Board of the UK. He faced many of the same arguments back in the 1950's with regards to the certifiability of the 707-300 I believe, but nevertheless rejected it for non compliance.
In that case, even if the plane stalled itself, it would immediately pitch back down, making it a "benign" aberrant flight characteristic. However, as he successfully argued, certifying that plane as is would set a precedent
Which would deteriorate the overall airworthyness of airframes over time as the goal posts were allowed to slip further and further away from what was required by law.
We call this Normalization of Deviance, and it's an Engineering firms favorite way to get people killed in high profile ways.
I am far from the best example of a person who doesn't exceed the boundaries of "rules" when they keep me from getting something done, but once you have the thing done in a non-compliant way, it is absolutely imperative to bring it back into compliance somehow or ensure every stakeholder (regulator/customer/user alike) is aware of the exceptionality of the non-compliant piece, and owning it in whatever capacity is required to get the job done safely.
Creators, designers, manufacturers, etc are given a great deal of leeway when it comes to doing things; part of that is the expectation that you know the rules you are breaking, and you are making the best effort to either communicate or remedy the non-compliance.
Don't know as it will change your mind, but I hope it makes where I'm coming from clearer.
> The problem Boeing face is that with MCAS disabled when this occurs, the plane no longer flies like an older 737.
The other problem Boeing faces is that with MCAS enabled the plane no longer necessarily flies like an older 737 - it can try to force its nose down unexpectedly.
The older 737 could do that too - it's known as a stabilizer runaway, and could happen if (for example) the stabilizer trim switch failed closed in the nose-down-trim position. It's something pilots train for, and there are trim-disable switches to handle this. Presumably Boeing believed that this was sufficient for pilots to cope with MCAS too.
The problem is that MCAS doesn't run continuously - instead it runs in 10 second bursts, so it looks different from a runaway trim event that pilots had trained for. And MCAS doesn't just result in runaway trim - the failed AoA sensor is also used to correct airspeed, so this gave the pilots an unreliable airspeed indication, and it also triggered the stick shaker on the captain's side, indicating an aerodynamic stall. The combination of all these things together seems to have confused the pilots, and resulted in them not disabling electrical trim.
> The problem is that MCAS doesn't run continuously - instead it runs in 10 second bursts, so it looks different from a runaway trim event that pilots had trained for
What's even worse is that a runway trim could be disabled by pulling hard on the yoke which would disable the autopilot driving the trim wheel motors. Then the wheels could manually be brought into position.
MCAS is not disabled by pulling on the yoke. Even if you set the trim wheels to the correct angle manually, MCAS would continue to move them, in increments that do not resemble a runway trim where you need to hold fast on the wheel.
So a sequence of actions commonly used by pilots to fix the runway trim is not disabling MCAS, making the pilots very confused about what's actually pointing the aircraft nose down.
What's even worse is that a runway trim could be disabled by pulling hard on the yoke which would disable the autopilot driving the trim wheel motors. Then the wheels could manually be brought into position.
Not if the runaway is caused by a stuck trim switch --- which is why there are the cutout switches and the procedure for runaway trim involves using them. Finally, if that doesn't work, manually grasp and hold the trim wheels:
(...and if that doesn't work, there's a big mechanical problem which is unlikely to be recoverable, since the trim wheels are directly linked to the stabilizer.)
> Finally, if that doesn't work, manually grasp and hold the trim wheels
You might think of grabbing and holding the wheels is there is a continuous movement there looking like an automated system going haywire. But with MCAS, it's not. It appears that the trim wheels move in short bursts in random movements, which they ALWAYS do. There is simply no reason to suspect that the wheels move in a single direction (pointing the nose down), unless you can visually see them. Which you can't because they're beneath and behind the pilot seat.
How come the pilots weren't aggressively keeping up with/against the MCAS? They simply gave up? Does that require some very serious effort (pushing many buttons, fiddling with the control), or do they have to simply keep "the joystick" in one direction?
Originally the captain was flying, and he was keeping up with MCAS using manual electrical trim. But near the end, he handed control to the first officer, so that he could read the quick-reference handbook and troubleshoot. And the first officer, not knowing how much trim the captain had been using, failed to keep up with MCAS. Very sad - if the captain had kept flying, they may have survived.
There may not be enough elevator authority in a severely out of trim configuration. However the trim switches can be used to correct that, or failing that the electric trim system can be switched off and the trim wheels can be turned manually to bring the aircraft back into trim.
From what I read it's a bit different. At high AoA the airspeed sensor doesn't work very good, therefore at high AoA the computer indicates that the airspeed is unreliable.
The failed AoA sensor indicated high AoA, so the computer indicated unreliable airspeed.
> this gave the pilots an unreliable airspeed indication
EDIT to clarify: Apparently this was misread as "unreliable (airspeed indication)" = "displaying an airspeed which was unreliable", rather than "(unreliable airspeed) indication" = "signal that airspeed should not be relied on".
Take a look at the graphs I linked to - they show this quite visibly. Both airspeed and altitude are taken from the pitot-static tubes, but these misread at extreme angles of attack, as the airflow isn't directly in the direction of the pitot tube. Thus the air data computer uses AoA information to correct the raw altitude and airspeed readings.
It doesn't really matter if the airspeed is slightly incorrect. What matters is that the aircraft is telling you the airspeed is unreliable, because that distracts from the actual problem you face.
We can see from the data that the left stick shaker is running almost continuously, while the right one is not. That is nominally saying that the airspeed is dangerously low. I assume that asymmetry is also a consequence of the faulty sensor, and I wonder if pilots are taught that this asymmetric stick-shaker response is a sign of equipment failure (and would it be clear to the pilots that this asymmetry is happening?)
Of course, if you don't know about MCAS, you won't know that it could also be messing with the trim. While trim runaway, at least when considered in isolation, seems straightforward to detect and stop, the pilots were trying to make sense of a set of not-clearly-related symptoms. I am curious as to how the pilot training and certification requirements are presented - for example, are they required to demonstrate handling a trim runaway, or to recognize and handle the various consequences of a given system component failure? It might be argued that there are vastly too many variations of the latter to be covered, but if so, that might be indicative of a broader issue, as real problems don't always present themselves like an exercise in responding to one specific symptom.
The stick shaker is driven from the angle-of-attack system, not airspeed, because that's the least difference between physical world (the wing stalls at an angle of attack, not an airspeed). In this case, the differential stick shaker may have provided an additional clue to the crew. While I'm generally critical of the crew training and performance, particularly the Ethiopian Air crew, even I can't reasonably hang missing the differential shaker indication on the crew.
Your last paragraph raises an excellent point, and one which points to the need to put fully trained, but equally importantly fully seasoned, crews into large transport aircraft in passenger service.
(some) People in aviation cried about the implementation of the 1500 hour requirement to get an ATP (Airline Transport Pilot) certificate after the Colgan Air crash in Buffalo. To me, these two accidents [and others outside of US/Western European carriers] show the wisdom in that decision, especially the most recent one. I have right on either side of 1500 hours (all but 20 in piston airplanes, about 4 hours in 737 level D sim) over a 21 year span. If I went to and passed a type rating school now, I'd hold myself barely qualified to do the walkaround when it's raining, sit right seat, run the radios, carry the captain's bag, and eat whichever meal she didn't want.
350 hours is not an appropriate time (IMO) to have a front-seat position in a transport jet flying passengers.
This can actually vary depending on where you are in the world. There are places, like Canada, where the term engineer has a specific definition under the law, and it includes things such as professional ethics and liability.
It's not so much that it's gatekeeping, but there's a big difference between writing bloated Electron applications with few, if any, constraints vs. writing Ada in an extremely constrained environment where there's memory constraints and other overhead that doesn't exist in the web development realm. If the former fails, a user gets slightly annoyed. If the latter fails, people die. One takes way more skill/training than the other. To treat the two on the same level is insincere at best.
It doesn't take substantially different training beyond recognizing that one is applying a different tool in a different environment.
It's all programming. Different patterns, stacks, and pressure to test? Yes. Fundamentally the same work loop, however. I'd advise against raising one level of endeavor over another. It seldom produces substantive or useful conversation.
Does anyone know what the typical salary of this kind of software developer is? I hope they are paid several times more than I make as a web developer.
I don't work in that area but could image it's much more layered. In consumer/webapp type software a lot of developers are more or less "full stack." In aerospace I could see it being much more rigorous and more like waterfall so by the time the specification gets to the programmer it's already been designed, reviewed, and signed off by several other engineers and the actual programming is a rather rote task.
Certain delegated systems engineers are 'responsible' for the system and they would work for Boeing or the FAA.
In a field like Aerospace, you don't have 'software developers' per se. You have systems specialists who write software. You don't move fast and break things, you run simulations of the software over and over again while varying inputs and fixing edge cases, and then you do real world testing.
Not sure how this relates to the issue of pay, but to answer your question: it's complicated, figuratively and literally.
There is a significant amount of technical coordination required to produce a system as complex as a commercial airliner. Systems engineers, at multiple levels, are responsible for wrangling this complexity. The top-level "spec"[1] is codified and broken down by subsystem to create a set of subsystem specs. Systems and lead engineers for each subsystem take those requirements and evaluate the feasibility. If they see a problem with the requirements, they push back to the higher level with requested changes and justifications. The systems and lead engineers at that higher level then evaluate the request in the context of the other subsystems, because a spec change to Subsystem A may require changes to Subsystem B as well, so they need to communicate with the team on Subsystem B (who may have made their own change requests as well) to determine if they can deal with the necessary changes. Subsystems that are themselves composed of subsystems repeat this process for their own subsystems, creating the possibility that a request could ripple up multiple levels and down a different branch in the system architecture. At some point changes may flow back to the top level and require negotiation on the broad top-level requirements, pulling them away from "wish list" territory and towards concrete, deliverable requirements.
This entire process is known as requirements gathering and flow-down. It is this process that defines the behavior of the system, through defining the behavior of each of its subsystems in turn down to the basic modules that are just collections of off-the-shelf or custom-built parts. Any of those basic modules that are entirely or primarily software should have a software engineer as the lead, helping to define what its requirements are and defining its interface with the rest of the system. Systems engineers tasked with managing those interfaces should also have significant familiarity with software development and architecture, perhaps even having started out as software engineers[2]. Ultimately, though, the lead and systems engineers[3] are responsible for making sure this process works effectively, and they have to do so as a team because a system like this is far too complicated for one or even a few engineers to fully understand.
[1] I use this term loosely, because it's more like a wish list of parameters and features than a real spec.
[2] "Systems engineer" is not a title you can drop in to straight out of school. It requires practical, cross-discipline knowledge of the design and architecture of the various subsystems that need to be combined to form the larger system. This is essential to be able to both understand how various subsystems affect each other and to effectively communicate with the engineers working on those subsystems.
[3] At the higher levels, where you run into a "subsystem of subsystems", the lead may be a systems engineer.
They tend to be in the range of aerospace engineer salary. High end of engineer salary but not approaching silicon valley pay. Check out ONET for government data on salaries
> Suppose they did originally do what the fixed software does now, and disable MCAS if the AoA sensors disagree. The problem Boeing face is that with MCAS disabled when this occurs, the plane no longer flies like an older 737....
But it’s been reported that this was an option you could buy when you bought the planes. And the crashed planes didn’t have this option.
So if that’s correct, then any plane shipped with this optional package would require the recertification. But it appears they don’t either.
If they did it would show up as very suspicious and I’m surprised nobody has reported on it:
Here buy this plane without this optional package and you don’t need new training.
Or buy it with the optional package and you need to learn about these new components we’ve added that may be disabled and undergo new training.
An AoA disagree indicator was an option. But it was never an option for this to disable MCAS, as this would have resulted in a change of the flight characteristics that would have required training.
All 737s have two AoA sensors. The left one feeds the left air data computer and the right one feeds the right air data computer. The AoA disagree indicator displays when the two disagree, but is optional and was not fitted on the planes that crashed. The old version of MCAS only uses one AoA sensor as input, even if the AoA disagree indicator was fitted. It does, however, alternate which of the two it uses with each flight.
I fail to understand how is the disagree indicator useful, considering the MCAS system will use the same AoA input regardless of disagreement. Is it useful for some other function?
If that's true (that it alternates between the sensors each flight) then it's extremely obvious that a business decision was made to only use one sensor at a time.
There’s no indication that before these accidents an airline buying the optional AoA disagree package got extra training to tell them about handling MCAS. And therefore it’s not clear that if these planes had this package they would’ve been saved. The pilots would’ve had a new warning light but not known that it means they need to disable MCAS because there was no training to that effect. If anything I assume the pilots being unaware of MCAS would only think an AoA disagree light impacted autopilot, which is separate from MCAS and disabled at these times. They would still have needed to identify the trim and MCAS problem another way.
(Now it is a different matter since everyone knows it’s there.)
Yes it all comes back to the requirement from on high to not require any retraining or recertification even though they were delivering essentially a different airplane. Trying to simulate the feel of a different plane via software is adding a huge new layer of complexity and failure risk.
> If they'd done this, they'd have needed to provide additional training, and this must have concerned Boeing management that it might jeopardize the common type rating.
Yes. There were no simulators to train pilots (only 4 delivered up to now, vs. 376 planes delivered! -- by the way, the value of all MAX orders, including these still not delivered, is around 600 billion with a b dollars!) and if I'd guess the simulators can't simulate the plane behavior when MCAS is off. Because the selling point is "MAX behaves the same as the old one." Which is just not true.
> The problem Boeing face is that with MCAS disabled when this occurs, the plane no longer flies like an older 737.
The bigger problem is the MCAS was only added to fix a major design fault, where by the aircraft would automatically pitch up when accelerating.
So with the MCAS disabled, the aircraft then runs the risk of stalling when accelerating.
I don't understand how design engineers would ever think a software workaround would be a suitable fix for what appears to be a major aerodynamic design flaw.
All aircraft with underslung engines want to pitch up when accelerating. That's just physics. MCAS is there to counter pitch up at high angles of attack, caused by the aerodynamic characteristics of the larger, more forward engine nacelles.
It's not the pitch that is the problem, but instead the degree of pitch.
This particular aircraft had a tendency to pitch so badly they had to install a software workaround to automatically catered for the behavior.
And presumably it was so extreme they thought an automatic option was needed, out of fear the pilots on their own would not be able to handle the situation.
According to Blancolirio on YT (a wholehearted thumbs up for his journalism, e.g. the video on atlas prime air is worth a watch, he currently flies as FO on the 777 I believe), there exists an angle of attack disagree light already in the 737max options sheet. There's also an option to purchase an AoA indicator dial, and he said one of the major us carriers did buy that option on their aircraft.
Yes, that's correct. But the implications of AoA disagree are different for the older 737 and the Max with the new software installed. On the Max, this means that the plane will now fly differently at high AoAs than it did if the AoA disgree light was not lit. This is something pilots will need to be trained for.
There is a huge difference of displaying angle of attack as input to educated pilots to feeding it into a system which has the authority to override the pilot with little warning.
The problem with the 737MAX airframe is that the pitch up seems to be so strong that nothing else than immediate MCAS feedback can reign it in.
Agreed. I think what this may mean going forward is that the CAAs are going to have to consider demanding that the training specifications be designed around a scenario where some (as-yet-to-be-defined) subset of the smart systems are disabled, and if the airframe behaves differently in that configuration, it demands re-training.
I'm somewhat surprised acceptance criteria weren't already there. You don't plan for the common case when lives are on the line.
> If they did require a separate type rating, this would likely kill 737 sales
Would it, though? I'm genuinely asking because I don't know how much all this costs. Certainly certifying pilots for a new aircraft isn't free, and probably isn't cheap, but the MAX line promises significant savings in fuel cost. In the long run, would the latter outweigh the former?
It likely would; if you have a fleet full of 737s and the 737 MAX turns into a new type rating, there's a much weaker argument as why they must get the 737 MAX rather than an A320neo family aircraft, and that gives Airbus a big in-road with potential customers.
That's not to say that Airbus would get all the sales (they won't!), but they then have a much easier time to selling to former 737 customers.
So Boeing may have to reduce the price a little. C'est la vie.
However ti's unlikely to cause major problems, presumably the tooling and spares for the non-max 737 are pretty similar to the 737. If you own a fleet of Boeing mid-range jets, why would you suddenly start to buy airbus?
Airlines will all have their own long list of criteria to help decide which aircraft to buy.
Removing common type will mean a few less points in a given airliners scoring system - it might tip the score to favour the Airbus for some, and for other airlines it might not even be a consideration.
Why would they? It's not like Airbus planes haven't had their own fatal glitches. Heck, it hasn't even been all that long since they had their own version of an MCAS style failure which was only recoverable because it happened a lot farther from the ground.
And notably, all the build slots for the A320neo family are sold for the next few years; I believe an order today won't be delivered till 2022 (and if you have other aircraft going off-lease, especially if some other airline is taking them on, you might not be able to wait). And you won't order today, because sales take time.
If you have a delivery date prior to that for the 737 MAX, you may well just want to wait and hope everything gets rectified before your delivery.
Boeing also have an incentive to shift their product, and with their negotiating position weakened will likely offer to sell their product for a lower price than they currently would to customers who already have 737s.
I expect it makes switching to an Airbus model more compelling - if you're going to have to retrain your crew, you might as well get the plane you want, not just the one you think will be easiest to train for.
I have no idea if the equivalent Airbus (a320neo maybe?) is any better or worse than the 737Max, but losing the common type rating would be one less vote for the 737 max.
The wings of the A320 have more ground clearing than the 737 NG. Airbus could easily fit the larger and more efficient engines. Boeing had to move the engines slightly before the wings (or face an expensive complete redesign). The resulting different flight behavior would have required a new type rating (which is expensive). Instead software (cheap!) was used to correct for the different behavior.
This is not an indication on other aspects of the A320 vs. the 737 MAX.
And remember that on the Boeing 737 Classic and NG that the engine nacelle has been non-circular and been running with smaller fans than other aircraft with similar engines: engine size isn't a new problem with the MAX, it's been a problem since the move to high-bypass engines.
The problem was not that the redisign was expensive - we are talking about trillion dollar market here but that it was slow - clean sheet = 10 years. Making an octopus by nailing four more legs to the old 737 5-6 years.
I am fairly certain that if boing could have solved the timing issue by throwing money at it they woyld have done so.
I guess for an airline that already has a mixed Boeing/Airbus fleet, or one that is only 737 and wants to completely switch to 737 MAX (where A320neo might be a decent consideration instead), this might make sense. But surely pilot and crew training is only one part of the equation. You also need new maintenance training, and you probably already have a logistics pipeline for sourcing Boeing-compatible parts that would need to be significantly changed, etc.
I would suspect that training a 737 pilot on differences in the new MAX would still be far less involved than training them to understand an Airbus, which is a totally different philosophy in control systems and piloting.
Disclaimer: I'm not a pilot, I am just plane enthusiast. Type training for 737 starts at roughly $13k, 767 rating starts at $18k so I guess 737 MAX having more systems to go trough than 737 should cost somewhere in between if it would be separate type. Currently 737 MAX need just type differences training if pilot have already 737 rating, so it cost somewhere in the ball park of $2k to do that. All of those are prices available to pilots, so if pilot want to get the rating and pay it for himself. I assume that airlines would have some kind of discount.
>Certainly certifying pilots for a new aircraft isn't free, and probably isn't cheap, but the MAX line promises significant savings in fuel cost. In the long run, would the latter outweigh the former?
The savings in fuel cost isn't everything, because you're missing the fact that Boeing was competing directly against the Airbus 320neo, which likely offered the same fuel economy but without the extra training costs. Why bother buying a Boeing when you can get the Airbus for less?
It seems pretty obvious to me that Boeing has been milking the 737 airframe for far too long, instead of proactively developing a replacement when they had time. Instead, they just stuck with what they had, and then when Airbus came up with their plane, Boeing tried to come up with something competitive on short notice using old junk they already had. They should have developed a successor to the 737 many years ago and moved to that, spreading the development and certification costs out, but they were short-sighted and cheap, and now it's going to cost them very dearly.
Switching from Boeing to Airbus would be more than just pilot and crew retraining, though, right? Maintenance and part sourcing would be significantly different as well. Obviously there are differences between 737 and 737 MAX there as well, but I can't imagine they'd be as significant.
In addition, since the 737 and 737 MAX are very similar from a piloting standpoint, I would expect that getting type certified on a MAX when you already are certified on a vanilla 737 would be much easier and cheaper than type certifying on a completely different plane from another manufacturer.
I read recently the the 737max was specifically designed (and rushed out) as a response to American Airlines ordering Airbus. Boeing made the max a bit more efficient, and kept the type rating the same as selling points.
I don't think it's just the personnel retraining. My understanding is that this decision helped keep the aircraft from being a "major" configuration change that would necessitate a more lengthy FAA certification process. Supposedly, a major air carrier gave Boeing a timeline deadline which may have influenced design decisions to avoid that longer certification.
Of course the FAA may be partly culpable for delegating some of their oversight decisions to Boeing
I would guess at the end of the day it wouldn't end up being a totally separate type rating. It would be additional training on the differences. But I think Boeing wanted to avoid the requirement even for that.
This is pretty much it in a nutshell as far as I can tell. If the sensors don't agree, and MCAS switches off, then the pilots have to be ready to deal with the plane trying to pitch up and stall on their own.
When would that happen? Take off and go-arounds.
Pilot is coming in for a landing, something goes wrong (too much cross wind, plane on the taxiway, Etc.) what they do is they pull back on the stick and push the throttles up to max to get into a climb. If MCAS is disabled and the pilot hasn't trained to fly the plane without it, there is a risk it will pitch up and stall onto its tail. Not a good place to be.
"Boeing's software fix, announced today, is to compare readings from both angle-of-attack sensors and disable MCAS if they disagree significantly. The obvious question is why they didn't do this in the first place?"
Because you had to pay for the second sensor and the disagree light.
If it's indeed solvable by software then it'll still sell fine, but this is one of the articles I have read where the claim is made that it is a fundamental design issue and cannot safely be fixed. I know nothing about aviation but I do spend a lot of time in planes and I sure as hell won't be boarding a MAX until the fix has been flying around without accident for a few years (I try to fly Airbus only anyway but that's not for safety reasons).
Perhaps it'll sell, but were I a buyer I'd be asking what else that Boeing might have skimped on and FAA rubber stamped.
We know the plane was built in a rush, and the single-sensor MCAS is an odd choice. It's unlikely MCAS is the only problem, although hopefully it's the only dangerous one.
Don't forget the perception of your ticket-buying customers.
Yes but how many years will the revetting take? And will such audit be considered. We are probably talking 2-3 years time frame with no 737 max if they do full audit.
A year from now, only a small fraction of the flying public will remember the 737MAX problem, and even fewer will consider it when they are booking flights.
IIRC the two planes that crashed only had a single AOA sensor (the 2nd redundant one being only present in a premium add-on that those airlines didn't purchase), so this software fix would have not changed anything.
From what I've gathered, all MAX planes (including the ones that crashed) have two AoA sensors, but MCAS just uses one. As far as I know there was never an option to have a redundant configuration. The premium add-on was the indication that they disagree.
No, all 737s have two AoA sensors. The one on the captain's side feeds the flight control computer on that side, and the one on the first officer's side feeds the flight control computer on that side. At any one time, one computer is in charge. They can compare data, but Boeing decided not to for MCAS.
All 737 MAX have two sensors. The Lion Air flight had sensors disagreeing by 20° right from the start. This made the stick shaker go off on one side right after departure (throughout the entire flight) and led to a multitude of other alarms (altitude disagree, unreliable airspeed).
No, it was never redundant, in the "premium variant" only the "they don't match" signal would be displayed / sounded, but only one used still, and the pilots would have to turn MCAS off and fly with the plane which behaves differently than the one for which they are trained.
I was under the impression the "base model" only came with a single AoA sensor. Adding a second sensor and the warning light if they disagreed was an expensive upgrade that neither of the planes that crashed were equipped with.
This reminds me of The Slow Winter by James Mickens [0]
> "John was terrified by the collapse of the parallelism bubble, and he quickly discarded his plans for a 743-core processor that was dubbed The Hydra of Destiny and whose abstract Platonic ideal was briefly the third-best chess player in Gary, Indiana. Clutching a bottle of whiskey in one hand and a shotgun in the other, John scoured the research literature for ideas that might save his dreams of infinite scaling. He discovered several papers that described software-assisted hardware recovery. The basic idea was simple: if hardware suffers more transient failures as it gets smaller, why not allow software to detect erroneous computations and re-execute them? This idea seemed promising until John realized THAT IT WAS THE WORST IDEA EVER. Modern software barely works when the hardware is correct, so relying on software to correct hardware errors is like asking Godzilla to prevent Mega-Godzilla from terrorizing Japan. THIS DOES NOT LEAD TO RISING PROPERTY VALUES IN TOKYO. It’s better to stop scaling your transistors and avoid playing with monsters in the first place, instead of devising an elaborate series of monster checks-and-balances and then hoping that the monsters don’t do what monsters are always going to do because if they didn’t do those things, they’d be called dandelions or puppy hugs."
Software ECC is already a thing. There's a lot of similar theoretical work on equivalent ideas for computation. Just remember that the idea is never about getting to zero. It's more like getting from one failure in 1e12 hours to one failure in 1e18 hours.
There is a debate to be had, but this is a naked propeganda piece. The crux of the article is based on:
“Among Boeing’s critics is Gregory Travis, a veteran software engineer and experienced, instrument-rated pilot who has flown aircraft simulators as large as the Boeing 757.”
... someone who uses flight simulators. This is not credible journalism.
Exactly... it's not clear if the author of the article is even an engineer, and he's speaking authoritatively on something. good engineer doesn't even speak authoritatively outside their specific area of expertise.
And the author's source is someone who is not an aeronautical engineer and has never flown any airliners apparently.
As a software engineer, I would never consider myself qualified to declare an airframe good or bad.
> ... someone who uses flight simulators. This is not credible journalism.
When talking about "instrument-rated", this most likely means a real rating certification. As for the flight simulators, best class "Full Flight Simulators" actually allow for zero "real" flight time for type rating transfers, as well as being actually used for the required regular training of airline pilots (per https://en.wikipedia.org/wiki/Full_flight_simulator)
When talking about "instrument-rated", this most likely means a real rating certification.
It is a real thing, but he's not type rated on a 737 and it shows. He and eetimes get a number of things plainly wrong. The article was mostly content-free clickbait and I'd encourage you to simply flag it.
An article written by a non-pilot about an article by a GA pilot with no experience as an ATP.
The elephant in the room is not the type-rating issue so much as the speculative cause of the crashes: if your aircraft is out-of-trim and control pressure cannot restore it, you have a runaway trim condition and you need to disable the electric trim system immediately.
If, in these cases, it turns out that the AOA sensor was faulty, that is only one of many possible causes for a runaway trim condition.
The core problem is not the specific cause, but the failure of pilots to respond appropriately to a common and easily-remedied situation.
We are armchair engineering the issue on hn, can there be “credible” journalism on this?
I feel for the engineers involved, I assume they all had good intentions and did their best. This seems like one of those things we’d read about in 2030 where some clever engineering and software allowed Boeing to fend off the rivals for a fraction of the cost, or likewise it took them to the brink of survival. As startup and entrepreneur people, we live this stuff daily with smaller stakes, I can’t help but sort of admire their attempt.
The amount of armchair engineering going on here on HN about this particular issue has been disappointing, but educational. I've seen a couple of very good technical write-ups about what's really going on with MCAS, and by my guesstimate probably 90% or more of what is said on HN about this issue is factually incorrect right from the start.
I was thinking 'faulty airframe' was a bit over the top. OK the stall characteristics seem not ideal but I'm sure a lot of planes have aerodynamics that are not ideal in places. I'd be happy enough flying in one if they turn off the MCAS gizmo.
The article reeked of anything but credible journalism as soon as it opened with "The saga of Boeing’s 737 MAX serves as a case study in engineering incompetence, and in engineering ethics – or the lack thereof."
By this point's it's obvious to everyone that the engineering of the plane is pretty far down the line of causes which lead to this.
There was a Twitter thread[1] a few weeks ago which explained it very clearly:
Some people are calling the 737MAX tragedies a #software failure. Here's my response: It's not a software problem. It was an
* Economic problem that the 737 engines used too much fuel, so they decided to install more efficient engines with bigger fans and make the 737MAX.
This led to an
* Airframe problem. They wanted to use the 737 airframe for economic reasons, but needed more ground clearance with bigger engines.The 737 design can't be practically modified to have taller main landing gear. The solution was to mount them higher & more forward.
This led to an
* Aerodynamic problem. The airframe with the engines mounted differently did not have adequately stable handling at high AoA to be certifiable. Boeing decided to create the MCAS system to electronically correct for the aircraft's handling deficiencies.
During the course of developing the MCAS, there was a
* Systems engineering problem. Boeing wanted the simplest possible fix that fit their existing systems architecture, so that it required minimal engineering rework, and minimal new training for pilots and maintenance crews.
The easiest way to do this was to add some features to the existing Elevator Feel Shift system. Like the #EFS system, the #MCAS relies on non-redundant sensors to decide how much trim to add. Unlike the EFS system, MCAS can make huge nose down trim changes.
On both ill-fated flights, there was a:
* Sensor problem. The AoA vane on the 737MAX appears to not be very reliable and gave wildly wrong readings. On #LionAir, this was compounded by a
* Maintenance practices problem. The previous crew had experienced the same problem and didn't record the problem in the maintenance logbook. This was compounded by a:
* Pilot training problem. On LionAir, pilots were never even told about the MCAS, and by the time of the Ethiopian flight, there was an emergency AD issued, but no one had done sim training on this failure. This was compounded by an:
* Economic problem. Boeing sells an option package that includes an extra AoA vane, and an AoA disagree light, which lets pilots know that this problem was happening. Both 737MAXes that crashed were delivered without this option. No 737MAX with this option has ever crashed.
All of this was compounded by a:
* Pilot expertise problem. If the pilots had correctly and quickly identified the problem and run the stab trim runaway checklist, they would not have crashed.
Nowhere in here is there a software problem. The computers & software performed their jobs according to spec without error. The specification was just shitty. Now the quickest way for Boeing to solve this mess is to call up the software guys to come up with another band-aid.
I'm a software engineer, and we're sometimes called on to fix the deficiencies of mechanical or aero or electrical engineering, because the metal has already been cut or the molds have already been made or the chip has already been fabed, and so that problem can't be solved.
But the software can always be pushed to the update server or reflashed. When the software band-aid comes off in a 500mph wind, it's tempting to just blame the band-aid.
I am not sure it even uses the term airframe correctly. And it is not faulty. Probably case of we have not had a boeing bashing hot take in 48h find someone that speaks vaguely pilot lingo and interview him.
I've read a variety of articles on this and they often said somewhat different things. What I've been able to gather about the timeline of events is:
1. The new engines on the MAX shifted the center of gravity forward (and I assume center of lift stayed the same).
2. Boeing was worried that #1 would cause the plane to nose up during high angles of attack (so, take off and landing?), and added software, MCAS, to pitch up to counteract this.
3. There's some confusion over when this software kicks in and how to cancel it (something about the trim controls not cancelling MCAS?)
4. Regardless of #3, this software seems to have confused pilots and the current belief is that MCAS was active when pilots didn't want it active.
5. ????
6. Planes crash.
Also, I've read about some concerns about the fact that the handling behavior changed so much but the plane wasn't reclassified as a different type. I'm still unclear about how classifications plays into this story.
My core point of confusion is, if MCAS is the culprit why isn't the solution to remove MCAS? Is tendency to pitch during high angles of attack unusual, and something pilots cannot be expected to counteract manually? I've only played sims like DCS and X-Plane (and not very much at that) but "nose goes up when I don't want it to, so I push stick forward" doesn't seem too complicated to me. Of course, I'm no pilot so I'm probably drastically oversimplifying the situation.
Your point #1 is incorrect. The problem with the larger engines is that their larger nacelles placed further forward produce extra lift at high angles of attack. This lift is further forward than the centre of mass. The certification requirement is that to produce steadily increasing angles of attack, you need to steadily increase back pressure on the yoke. The problem with the Max is that this is no longer true. Past a certain angle of attack, the back pressure needed to further increase angle of attack reduces. The plane is not actually unstable, but it's closer to being so than the certification requirements allow. And it's certainly behaviour that Boeing couldn't claim was similar enough to older 737s to allow a common type rating. Hence MCAS, which was supposed to detect this condition and make the aircraft fly like an older 737. This allowed a common type rating, and allowed the aircraft to be certified. But fundamentally, the airframe has an undesirable property, and you'd never have designed it this way unless the desire for a common type rating dominated other design decisions.
It is incredible: the airframe has been designed to reuse the certification of 737. A flaw that could be fixed in a proper way, instead has been worked around using unreliable subterfuges.
The aim of the certification process is to ensure the safety (and reliability) of aircraft. The required mindset shall be that the certification process helps to highlight defects in order to build a better aircraft. Here, the mindset was that the certification process is a burden with arbitrary constraints that have to be fulfilled even if this means a worse aircraft.
IMHO, the people (managers) with the wrong mindset shall be replaced and the faulty airframe of 737 shall be killed.
This is unfair. If done appropriately, making a plane behave the way pilots expect should also make it safer. A "worse aircraft" with a better UI can actually be a safer aircraft.
Only so far as we can assume the UI never breaks and the sensors are always correct. The issue is pilots need to be trained to understand where the UI hides the underlying performance of the plane because when it breaks it can go wrong incredibly quickly. MCAS altered the way to aircraft reacted and also would restart it's changes unless completely cut out. Pilots didn't receive enough training to recognize the issue as being the MCAS system adjusting the trim fast enough to prevent the crashes during takeoff and ultimately the whole point of the MCAS was to avoid having to retrain and maintain the common type rating.
Thank you; I've been following this since the second crash and this is the first time I've seen an explanation for why moving the engines forward made the plane unstable at high angles of attack.
So, I must be missing something but all that re-engineering was to put a larger engine on the airframe but still have ground clearance, while on the ground, correct? What was stopping them from increasing clearance by increasing landing gear height? (And thus not impacting flight characteristics)
They did increase the length of the nose landing gear, which helped a bit. But increasing the length of the main landing gear would have required moving the gear further outboard, as there was no space to extend the gear in an inboard direction when it's stowed. You can't move the gear a lot further out, or it ends up behind the engines. See this picture someone else on this thread posted: https://i.stack.imgur.com/GFzcj.jpg And you really don't want to move the engines further out, because that's going to affect the engine-out behaviour, and require a larger vertical stabilizer to handle asymmetric thrust.
> Boeing was worried that #1 would cause the plane to nose up during high angles of attack
Your #2 is flawed. Shifting the center of gravity forward would cause the plane to nose down relative to the older model. The problem is that the engines are more powerful, mounted higher and further forward shifting the point of thrust to a location where the plane would rotate further (nose 'up'), that is why MCAS attempts to push the nose down by trimming the tail plane if it senses the plane to be in a too high angle of attack.
From what I read purpose of MCAS is to make stick give same feedback as on 737 NG, i.e. certain linear force increase would correspond to deflection amount on the elevator. With MAX it was no longer true due to air frame geometry and engine changes, and at certain high angles same amount of force applied to stick would result in more deflection. All this had to be maintained in order to not have re-certify all the pilots from NG but be allowed to only give minimal training I think.
> From what I read purpose of MCAS is to make stick give same feedback as on 737 NG
No, the MCAS system is not in any way responsible for feedback to the control stick. It engages the stabilizer (tail plane) trim motors based on AOA sensor readings, the state of the autopilot and whether the flaps are engaged or not. It has a limit of 2.5 degrees change but this limit is reset every time the system is reset in effect allowing it to fully deflect the tailplane until the limit stops kick in.
Same thing, surely? Altering the trim means that pulling a certain amount on the stick doesn't make the plane pitch up as much as it would otherwise, hence you need to pull more on the stick (than a plane without MCAS). So the (intended) end effect is the same, right?
The bit you put between brackets '(intended)' is the key here. What they intended was one thing, what they got was an entirely different thing. The stabilizer being as large as it is and the degree of authority afforded the MCAS system could cause the tail plane to deflect to a degree that the stick could not overcome it at all, the elevators are simply not large enough to counteract a tailplane that is trimmed purposefully the wrong way to the maximum deflection, in part because trimming the tailplane wrong also limits elevator effectiveness.
No, because what changes is how far you move the yoke not necessarily how hard you have to pull (on the penultimate Lion Air flight the elevator feel system was non-op which meant that the yokes were probably extremely light for the whole flight). There will also come a point where you cannot overcome the stabilizer with the elevator.
We don't even know the real problem, how can we make speculation like this. Every article seems to cite "faulty sensor data", but is this really true? Would making the sensor more reliable or having an additional backup sensor have prevented the issue? Is it possible for the software to even detect when the sensor is faulty? Maybe the sensor worked fine, and the MCAS worked fine, but the plane itself has faulty aerodynamics.
At this point, it seems clear that MCAS had something to do with the crashes, but it's far too early to point to the root cause.
> Would making the sensor more reliable or having an additional backup sensor have prevented the issue? Is it possible for the software to even detect when the sensor is faulty? Maybe the sensor worked fine, and the MCAS worked fine, but the plane itself has faulty aerodynamics.
The sensor on the Lion Air flight was 20 degrees off. While it wouldn't have "prevented the issue", a simple disagree light in the cockpit would have told the pilots that the AOA sensor was "wrong" while they were still on the runway. Simply put, the plane was unfit to fly and had this light been installed on that plane, the flight would have never left the ground and the crash wouldn't have happened.
Boeing charged extra for this light.
Since this has come to light, Boeing have announced that the disagree light will now come as standard on all 737MAX's at no additional cost.
There is no AOA on the ground. it requires airflow. You would not have seen an AOA disagree until the aircraft was in the air or at least moving at a pretty good speed down the runway.
I read that in another heavily upvoted comment on here in an earlier discussion a week or so ago. I really shouldn't assume that randoms on the internet know what they're talking about.
Yes both points 1 and 2 seem the wrong way round. Presumably the centre of lift has moved relatively further forward than the centre of gravity to create a greater pitch up moment (which apparently exists on older models).
MCAS wasn’t introduced to simplify pilots’ life. It was to make the plane way cheaper overall by avoiding the plane being a different one according to regulations. Regulations that were there to make planes safer to begin with by making sure differently enough planes warrant training, processes, reviews, etc.
Sadly, MCAS introduced unintended side-effects and is not as transparent as it was supposed to be, from what looks like.
Therefore, if you remove MCAS, the plane should be easier to predict; but who knows what that implies now that the plane has been sold and flying for a lot of time (financially, legally...).
> if you remove MCAS, the plane should be easier to predict
And the prediction is: the plane would then not pass the "is it safe?" tests which don't exist "just so" but to make sure that the plane can avoid to crash as soon as something small and insignificant happens.
Without the MCAS the input from the pilot would lead to plane failing easier in higher angles of attack. The MCAS was added to avoid that. So MCAS should work and should not have a single point of failure, as long as 737 MAX flies. MAX is indeed less stable in higher angles of attack.
All Boeing claims "they can simply turn it off" are misleading. The plane is actually more dangerous without properly functioning MCAS. But to properly function the sensors have to be redundant and reliable, and the control that activates it too. And the control software which deactivates it should also be smart enough that "activate/deactivate" cycle isn't itself the crash cause, which is what it looks like was what directly caused the last two crashes:
- the sensor used was only one, i.e. was not redundant.
- the control software blindly trusted it.
- the software, apart from blindly trusting the input of the sensor, was not even properly tested in the condition of the faulty sensor, specifically, the impact on the pilots.
- the whole feature of MCAS was intentionally downplayed, and even hidden initially, to hide the fact that the pilots would have to train for the conditions in which MCAS fails (it can happen, as it at the end depends on mechanical and computer devices) -- the pilots have to be trained to "save" the plane without MCAS functioning, even if the plane is less stable. Note: MCAS should properly function as much as possible to minimize the chance for any small error causing big crash, but in the case when MCAS is not functioning the pilot still has the chance to save the plane only if he is trained exactly in the simulator which simulates the plane with no MCAS (when the controls don't behave "normally", which is what MCAS provides when working). That kind of training was not made at all because the selling point of MAX was "it's the same old, no need to pay for pilot's training." (Or maybe, then the pilots would know that the chance to save the plane was so reduced that they would complain more about the overall plane design?)
- the decision to "just let pilot turn it off" was based on the false assumption that the faulty MCAS behavior is "obvious" to the pilots so that they turn it off easily.
On further investigation, it seems #3 is more: MCAS had much more control authority (i.e. was allowed to change control inputs from the pilots by more) than the FAA had approved.
#5 is: a fault in a single, non-redundant sensor could cause the MCAS system to falsely believe the plane was stalling, and push the nose down as hard as it could. Which (see above) was harder than a pilot could override by just pulling in the other direction, as is apparently intuitive for them.
I've always been confused here on this, does that mean the 737 MAX has a high chance to have faulty reading from their AoA sensors? Or is that pretty much the industry standard right now? This has always seem like the actual culprit.
Sensors break. They get gunk on them. They ice over.
Generally a sensor with such a critical failure mode would be triple-redundant - if one fails, the discrepancy between sensors is flagged and the aircraft runs on the other two until the broken one is fixed. In this case, the aircraft had two sensors of the type (Angle-of-Attack), but MCAS was only listening to one of them.
Why isn't this compounded with something that works by estimating the AoA from other factors? A few simple gravity based sensors would be able to tell the vector of the plane, and simply assuming that wind (airflow) is parallel to the ground would go a long way. Or is the vertical component of the local airflow so variable?
When in flight, accelerometers only tell you what forces are being exerted by the wings and the engines, not which way is down. Consider, for example, the way the drinks in a cup don't spill when a plane turns.
I know that an airplane can make a roll that keeps the fluid in the cup, but when usual commercial planes fly, they don't do stunts, so usually there's no centrifugal force to mimic gravity. When a big jet points its nose down or up people and fluids feel it pretty much as if they were on the ground on a slanted flat surface as one of its edge is being raised or lowered.
Sure, this would probably worth next to nothing in turbulence, but in a simple take off and landing (where MCAS is already active and depends on AoA) it might help.
And of course I might be completely ignorant of most of the relevant problems with using any kind of gyroscopic or acceleration based sensor.
No, but cost probably does. There have been a number of satellites flown (GOCE, GRACE, SLATS) with the sort of equipment you'd need. With that equipment, simply measure the strength of gravity in multiple parts of the plane. This gets you altitude, just as GPS or air pressure would, and then you can determine the angle of the plane.
An edit as response to the followup mentioning Einstein, due to HN throttle:
Yes, yes... and it doesn't matter for this purpose, because you can measure gravity at multiple points within the aircraft and because gravity falls off with distance.
"Einstein’s ground-breaking realization (which he called “the happiest thought of my life”) was that gravity is in reality not a force at all, but is indistinguishable from, and in fact the same thing as, acceleration, an idea he called the “principle of equivalence”.
Thanks for the elaboration. Could you help me further understand one more thing? When you say MCAS only listens to one, does that mean during the time when one AoA sensor fails? Or it always listens to one during normal operation?
Also, it does seem like Boeing dropped the ball here to not build further redundancy here.
MCAS alternates between the left and right sensor each time the plane lands. With the Lion Air flight I think cycling the electrical power for diagnostic work caused MCAS to pick up the faulty sensor for two flights in a row.
Pilots and first officers tend to switch who has flight control on each leg of their flights (which is the Pilot Flying vs Pilot Assisting), and the MCAS system uses the AoA vane associated with the side of the cockpit that currently has flight control.
There is no good reason for only listening to one sensor.
There is a sort of good reason for having a split between pilot/copilot side: the instruments are redundant (both physically and electrically), so in the event of malfunction you can failover to the other side.
That actually sounds awful, sorry for my naivety if this is just industry standard. But for such a mission critical piece to have no redundancy build over it is just poor. Especially that it's prone to failure since it's situated on the outside of the plane.
It just seems to be that this is some terrible engineering done on Boeing's end of not fully understanding the critical situation here.
Generally two failures:
1. a lack of redundancy in a mission critical sensor
2. a blind trust on MCAS's priority over pilots
There is redundancy in the sensors, but the sensors are not being used in a redundant manner. There are whispers that the 767 fuel tanker (KC-46/KC-767) has a system similar to MCAS that will look at both alpha vanes for disagreement, which is a bit damning to say the least.
a blind trust on MCAS's priority over pilots
The entire purpose of MCAS is to engage only when the pilot is flying to prevent the pilot from doing something dangerous. Previous generations of 737 had the same problem but the MAX is more delicate and compounds it with nacelles that generate lift.
Part of the problem was that MCAS was originally designed with very little control authority, and so wasn't considered safety-critical. However, during testing they realized they needed to up the gain, and made pretty major retuning without reexamining their safety assumptions.
The lack of redundancy by default coupled with extreme profit seeking by Boeing for the 'upgrade' is inexcusable. TL;DR They shit out a faulty product hundreds died as a result
Even redundancy doesn't really solve the issue. The sensors are out in the same environmental conditions, so will likely all fail at once, for example a bad pattern of water followed by cold causing icing.
Instead, the sensors should detect failure, for example by using a motor to detect if the vane is stuck and cannot turn freely.
The flight controls should also be able to fly even if all sensors of a certain type have failled. Angle of attack for example can be approximated with an accelerometer and gyro well enough to keep the plane in the air.
You're right - you need to combine with altimeter or GPS, and for more accuracy you can also combine with wind direction forecasts or airspeed measurements.
The point is, there are lots of data sources, and with even a subset of them, it's possible to fly the plane to a safe landing.
It's tangential but when I read those comments saying that a system could effectively override a pilot, by putting a feedback beyind human strength on a critical control, I always think how enormous kudos is owed to the experienced badass that flew the plane before, during critical situation figured out he knew better and tirned it off.
The thing about the 737 is that you have massively powerful engines essentially hanging from the underside of the wings, and the wings are in a low-wing configuration (i.e. they sprout from the bottom of the fuselage and not the top like a Cessna).
Imagine you're on final into a busy airport and you have to abort your descent because $reasons. If you slam the throttles forward, the thrust is all below the aerodynamic centre of the aircraft, so you have a net nose-up pitching moment created. In other words, when you slam the throttles forward the aircraft tends to nose up. A 737 pilot told me recently that in a take-off/go-around situation you actually need to push the yoke forward to restrain the aircraft's nose-up tendency until you can re-trim for climb.
MCAS was introduced because the new engines on the MAX, essentially, make the airframe more dangerous at high Angle of Attack than the previous generations of the 737, and so MCAS was intended to automagically keep pilots from entering the dangerous regime.
MCAS was required in order to keep to the "zero training delta" commitment. The solution might be removing MCAS, but that will/would require more pilot training which would essentially render the MAX a completely new aircraft as far as regulation is concerned.
> the thrust is all below the aerodynamic centre of the aircraft
Just like 95% of airliners. In fact the Max engine thrust line is higher and closer to the datum than most.
The Max's problem is not thrust line, it is additional lift generated by the forward-mounted engine nacelles at high AoA which reduces stick-load and makes pitch-up to a stall likely. Nothing to do with thrust.
The Max's problem is not thrust line, it is additional lift generated by the forward-mounted engine nacelles at high AoA which reduces stick-load and makes pitch-up to a stall likely. Nothing to do with thrust.
Pitching up on thrust was a problem with the 737 Classic, why wouldn't it be a problem on the MAX?
>Just like 95% of airliners. In fact the Max engine thrust line is higher and closer to the datum than most.
Correct.
>The Max's problem is not thrust line, it is additional lift generated by the forward-mounted engine nacelles at high AoA which reduces stick-load and makes pitch-up to a stall likely. Nothing to do with thrust.
Nothing you've written here contradicts what I wrote previously, so I'm not sure if you're just aggressively agreeing with me, but I'll assume that this is the case. Yes, the aircraft is dangerous at high AoA, as I wrote in my previous comment.
MCAS operates on the horizontal stabilizer (and is not a stick pusher like the article claims) the yoke operates on the elevators. The elevators are much smaller and lack the pitch authority of the stabilizer. The stabilizer moves much slower and by the time you've stalled under the conditions MCAS is designed to prevent, you may not have time for the stabilizer to move.
So the issue is that MCAS was controlling the big control surfaces, the pilots' attempts to correct with yoke input operated on the small control surfaces? The former overpowered the latter.
Interesting that MCAS wasn't like cruise control where any significant input deactivates the automated controls. But then again, I know next to nothing about how big airliner control systems work.
> Interesting that MCAS wasn't like cruise control where any significant input deactivates the automated controls.
It would disengage... but then would immediately re-engage if the bad sensor readings were still there. And would, in the meanwhile, have reset its control limiter so it would push even harder than the first time.
Also thanks to "zero training delta" commitment pilots probably didn't even know it exists or in a high-stress situation like imminent stall probably just forgot about it all.
It's worse: the pilots are supposed to "jump in" instead of the software when the plane is typically already in the critical state and then to control the plane which behaves exactly differently from the way they are trained. Only when the MCAS operates "properly" the plane behaves as they were trained!
That "omission" was intentional as the Boeing selling point was "it is the same old, no new pilot training needed." That's how they have "sold" the worth of 600 billion dollars of the MAX planes!
The probable strategy for Boeing was "we will blame the pilots" and had the two crashes not happened so fast one after another they would have probably got away with it!
They're supposed to jump in and resolve the problem within forty seconds apparently. Otherwise they're screwed[1].
Those involved in the testing hadn’t fully understood just how powerful the system was until they flew the plane on a 737 Max simulator, according to the two people.
The 40 seconds is misleading. The MCAS system could in the worst case run the trim full nose down in 40 seconds (it's active for 10 seconds with a 5 second delay between cycles) if the pilot does nothing to counter it.
It's like if you are driving and the road starts to curve. If you do nothing, yes you will get to a point where a crash is unavoidable. But you know that when your car is not following the road, you turn the steering wheel. Pilots know that when they have to pull the yoke to keep the plane level, they need to trim.
The problem is that if you do the intuitive thing (which is what pilots are trained to do for most trim overruns - pull on the stick to counter the trim), the MCAS goes through the increasing cycle to full nose-down. It's 40 seconds to realize that your usual strategy isn't working, figure out what to actually do, and then turn off the exact right system.
From what I have understood they used MCAS so that they didn't need to reclassify the plane model and that pilots would not need to retrain for it as with the MCAS the handling would be similar to the older planes.
The actual problem might not be with the design but using workarounds to get through red tape of reclassifying and retraining pilots for a new plane
Wasn't the idea that they wanted to avoid pilot re-training? So if pilots would be expected to counteract manually, they would have to train to understand when to do that.
Pilots are already trained to handle runaway trim, the idea was that an MCAS failure would behave similarly and the same checklist they already trained on could be used. Due to issues in how MCAS was implemented however, the behavior was different from runaway trim and more confusing.
> with Boeing aircraft when the autopilot is off you're fully manual
The 777 and 787 are full-time FBW, there is no manual reversion. There is no way to move the control surfaces directly. The only difference with the AP engaged is the origin of the control demands.
You can have cables, push rods, hydraulics, power assisted hydraulics, full hydraulic, and electrical actuators. As long as their control inputs are only from the pilot they are identical in every way but their failure modes.
There is big big difference between FBW and the aircraft generating control inputs on it's own. Boeing aircraft had two modes, 'manual' and 'auto pilot' The 737 MAX adds a third case, 'sometimes'
Boeing never told the pilots about the 'sometimes' part. And never told them that 'sometimes' has ultimately more control authority than they do.
I think you misunderstood my post. There is literally NO manual command over control surfaces in any current Boeing airliner other than the 737 variants. Neither the 777 nor 787 have any pushrods or cables.
The Max does indeed have a hybrid approach ( manual actuation, FBW spoilers, mystery MCAS ) but that's not Boeing's current philosophy.
1. earlier 737 models didn't pitch up and Boeing wanted the plane to "feel" the same and not have to re-train the pilots
2. there is some certification requirement that the pitch has to be constant during climb (or something like this); if the plane doesn't have this, it is not certified to fly. It wouldn't pass FAA certification without MCAS.
If MCAS is required for the plane to be airworthy, then it is not airworthy when the pilot disables it via stabilizer trim cutoff. And if the behavior of the plane is not subtly different in approach to stall, stall, or stall recovery, then the pilot might also not be type certified for the airplane with MCAS disabled.
I'm really suspicious, in a variety of ways, of cutoff switches effectively decertifying a plane in flight. And then how it's OK for pilots to not at least be made aware of that potential situation in advance?
An airplane suddenly rendered not airworthy, and pilot suddenly rendered without a proper type rating. It's absurd. I don't know how a software update gets them out of this predicament, if it's true.
Airbus fly-by-wire aircraft have numerous layers of safeguards (laws) in place. Each can be removed or degraded depending on the circumstances. But pilots are expected to know all of them, and know the consequences of each safeguard being removed, including the ensuing natural flight behavior of the airplane.
> there is some certification requirement that the pitch has to be constant during climb (or something like this); if the plane doesn't have this, it is not certified to fly. It wouldn't pass FAA certification without MCAS.
So what I gather is the issue was that the MAX didn't fit this requirement for steady pitch (hence the airframe problem referenced in the linked article), and MCAS was supposed to be the band-aid to fix this essentially by automatically pushing forward on the yoke during high angle of attack.
If that's the case then the following is especially concerning:
> Boeing offered the single angle-of-attack sensor as standard equipment, and charged extra for a second along with a “disagree” indicator that would allow 737 MAX pilots to “cross-check” a faulty sensor.
Seems pretty sketchy to ask airlines to cough up extra dough for redundancy on a safety critical system. Who knows what other systems are subject to the same cost/benefit tradeoffs.
Boeing offered the single angle-of-attack sensor as standard equipment, and charged extra for a second along with a “disagree” indicator that would allow 737 MAX pilots to “cross-check” a faulty sensor.
Yeah, that's factually incorrect. All 737 NG/MAX planes have two alpha vanes to detect the angle-of-attack. MCAS only ever uses one at a time.
From comments I've read, abysmal, in the sense the Quality people are told they aren't there to keep the plane from falling out of the sky, but rather to generate audit trail so that if a disaster happens, the cause is traceable.
My personal approach to Software QA is generally more in line with the role fulfilled by a Systems Engineer, so I'm intensely uncomfortable having read Quality may be culturally interpreted in such a perverse manner.
Funny thing about Quality is you find exactly what you look for. Start out making a system that only suffers traceable disasters, and you get a different product than one which focuses on not suffering disasters in the first place.
I don’t have an aeronautical background at all, but a hunch I heard from some aviation forum is that the plane must be certified to be safe when flown by merely an “average” pilot, and if the Max was more difficult because it had this tendency to stall, that might mean it needs a pilot with more skill, therefore more training for existing 737 pilots. So MCAS was needed to ensure no skill difference or training difference whatsoever for the pilots, even if a typical pilot should be capable of flying without MCAS.
The CFM Leap-1B (new engine on 737 Max) are bigger than the CFM56-7B, so Boeing had to change the mounting point of the engine (further forward and higher on the wings).
Well eetimes got some clicks, so job well done for the journalist who wrote this article about a blog post by some guy with experience flying large planes in a flight sim.
Not sure I would call it 'good' but we did sure get a lot more chatter. I wonder if collectively this is the most comments a particular issue has ever received on HN. So many articles posted, each gets hundreds of comments.
While they're is definitely a gap in the physical nature of controlling the two platforms, the fundamentals underlying both pieces of flying equipment are the same, and in most of the ways that really matter, there is no real difference between an MCAS like system on a Cessna vs a 737.
[waits for the multi-engine and fundamentally different airframe crowds to die down]
At the end of the day, strip out every system not at fault, and you have a pilot fighting a machine he isn't aware of the exact details of in terms of how it's response to various control inputs are translated to behavior of the plane.
Any pilot period (and any honest Engineer) should be capable of recognizing the 737 and the MAX 8 are entirely different planes on a fundamental level.
You may have made your comment in sarcasm, but it is kind of an Emperor's New Clothes situation going on for Boeing right now.
When the laypeople can be quickly brought up to speed that something isn't fundamentally the same, yet you (the principal engineering firm) dig in your heels to insist it is, you've committed some grievous breach of public trust.
That's pretty awesome -- and chilling. His four seat Cessna's $20k avionics upgrade (which got automatic pitch regulation same as the Max) required more training and documentation than the multi-million 737 Max that killed 157 + 189 people.
The 737 MAX is a dodgy design which probably ought to be banned.
The 757 flight simulator that I've flown was a full-motion flight simulator at UPS's Louisville training facility. I was helping conduct 6-month certification flights as co-pilot. I would say that if those simulators are good enough for UPS's 757 pilots' recurring training, they're good enough for this Cessna driver to understand that difference between his plane and a 757. Answer: Not that much.
Because the A320neo can fit the engines under instead of in front of the wings, thereby negating the nacelles contribution to leading edge flow separation.
Exactly. Except it’s not flow separation that is the problem. It’s the lift that the nacelles generate combined with their mass and moment far ahead of the center of gravity
This article appears to be fairly thinly sourced. The one named source I can find appears to be a blog post by a fellow software developer who is an instrumented-rated pilot, however has flown airliners in simulation only. The article does not claim the source is a professional pilot or that they have ever flown the 737Max.
With due respect, I am not sure whether that counts as enough expertise to qualify someone’s opinion as news worthy?
This is Gregory Travis, who wrote the original article on which the EE Times article is based. If any of you have a specific question regarding my conclusions or how I got them or want to discuss any statements of fact, I'm more than happy to engage.
In your paper, you erroneously claim that the MCAS system creates forward pressure on the control column ("pushes the pilot's control columns in the down direction".) This is incorrect. MCAS only acts on Horizontal Stabilizer Trim, and doesn't work at all when autopilot (which does provide control column forces) is active.
You also claim that the 737MAX doesn't have "mechanical connections between the pilot's controls and the things on the wings, rudder, etc." when the 737MAX does have full mechanical linkages between the control columns and the flight control surfaces. Those surfaces are normally hydraulically actuated, but if all hydraulics fail there is physical manual control of the elevator and ailerons, also known as "manual reversion."
Because of these glaring errors, your paper loses much of its credibility in critiquing Boeing designers.
It was a worthwhile read and it filled in a lot of details I didn't notice about the crash also affirming the criticism I hold of the process of software development -- where all of the decisions are deferred to the magical 'algorithms' without considering what who writes the algorithms and how decisions about them are made.
The more persistent problem would be how artificial intelligence is used to increase 'engagement' while also fueling hate, and is an example of programmers and their managers not listening to social scientists and journalists.
When planes goes down it's the case programmers and their managers don't listen to engineers and people on the ground.
Hey Greg - serious question - The A320NEO has a very similar engine placement as the Max8. Wouldn't that air frame be faulty too if the problem was solely due to engine placement being in front of the wings?
Why is this: “once this thing pitches up, it wants to keep pitching up”? And why is it more of an issue with the engines in their new position? Thanks.
Couple of things here to keep in mind. First, when the A320 came out I wrote extensively about its fly by wire system, which was highly controversial at the time (early 1990s). It’s been nearly thirty years since then and long story short, Airbus has vastly more experience with implementing cockpit automation than Boeing. Boeing simply got in far over their expertise with deadly results
Second, Airbus’ 320 airframe does not impose the same issues with larger engines that the 737 does. For starters, the 330 airframe started life in the era of large high-bypass turbofans — its initisl engines were much larger than the 737’s initisl engines
It wants to keep pitching up because the engine cowlings are now far ahead of both the center of gravity and the center of lift. And the cowlings generate significant lift themselves. Aerodynamically they act as levers that pull the nose up and the higher the angle of attack, the more they pull. That is dynamic instability and as I point out you want to have ejection seats in any dynamically unstable aircraft (ir fighter jets)
Wondering if Boeing will be able to recover from this in regards to keeping the MAX flying at all. I mean, I will always pick a different plane from now on - just not a risk I'm willing to take in the foreseeable future. Not sure if my stance is common though.
Airplane crashes are rare, and it's quite easy to remember some unique plane crashes (like I would also not fly in a Concorde). With Boeing MAX right now there is definitely some bad decisions they made safety wise, and it be not surprising that there were other tradeoffs made also.
The real question isn't whether the autists on HN will be researching every flight they schedule, it's how many Average Joes will care. Short answer: to a close approximation, zero.
At this point no 737 pilot has any excuse not to understand MCAS, including what it does and how to disable it if there's a problem, and will be watching for issues vary closely.
Remains to be seen but I am sure all the major certification authorities will be asking this now. Probably by the time it's flying again there won't be any need to make assumptions.
Many (most?) airlines list the aircraft type during the booking process. Even for those that don't, you can look up the flight number and date on a site like FlightAware or SeatGuru, which will tell you.
But, if the airline needs to, they can swap out the scheduled aircraft for another one at any time up until takeoff. So you might book with the belief that you'll be on a 737 NG, but get to the airport and find (or not even notice) that you're on a 737 MAX.
You can see it in most search engines, and after the Ethiopian crash Kayak announced they'll add an aircraft type filter. Other flight search engines will probably follow suit.
Really? My conclusion from this saga has basically been that the 737 Max is fine when flown by first-rate carriers (American, Southwest) and conversely that no plane is safe to fly with 3rd or 4th rate carriers in the third world (Lion Air).
Everything comes down to the Swiss Cheese - you're focusing on Boeing's hole, while I care more about the stack of holes that Lion Air maintained.
At the same time, Ethiopian flew a copilot with (IIRC) 200 hours of total flight experience. This changes emergency CRM from "one person flies the airplane, the other diagnoses the problem" to "Captain flies the airplane, babysits the newbie, and attempts to diagnose the problem".
As for Ethiopian Airline's 'stellar safety record', comparing these two wikipedia pages [0] [1] they seem to have a similar (recent) safety record to Aeroflot. Yeah, Aeroflot's gotten better, but not that much better. (Ethiopian has two incidents that resulted in fatalities 2000-2018, discounting passenger hijackings - Aeroflot has one. Both have one "everyone on board" incident. And while I'm not sure about relative flight frequencies, Aeroflot has a fleet of 253 planes to Ethiopian's 112. For comparison, Delta has had 0 incidents resulting in fatalities since 2000 [2] on a fleet of 896. Same goes for United's fleet of 778 if you exclude hijackings - American (with a fleet of 962) had one non-hijacking incident in 2001) So I'm honestly not sure what people are talking about there.
1) Like most people, I’m far from qualified to give aeronautical engineering advice, but as fly-by-wire technology gets more advanced, won’t this be the norm? ie: Airframes that are difficult to fly might always be more efficient, so have a computer do the hard part.
2) This part seems like the real damning misdesign:
Boeing offered the single angle-of-attack sensor as standard equipment, and charged extra for a second along with a “disagree” indicator that would allow 737 MAX pilots to “cross-check” a faulty sensor. Citing those decisions, another observer noted: “Who would design a system with a single point of failure?”
> Relaxed stability designs are not limited to military jets. The McDonnell Douglas MD-11 has a relaxed stability design which was implemented to save fuel. To ensure stability for safe flight, an LSAS (Longitudinal Stability Augmentation System) was introduced to compensate for the MD-11's rather short horizontal stabilizer and ensure that the aircraft would remain stable. However, there have been incidents in which the MD-11's relaxed stability caused an "inflight upset."
> Updates to the software package made the airplane's handling characteristics in manual flight similar to those of the DC-10, despite a smaller tailplane to reduce drag and increase fuel efficiency.
Maybe this isn't as new of a development as we think.
> “Who would design a system with a single point of failure?”
According to the original design, the MCAS was only supposed to adjust trim to a level that was easily overridden by the pilot essentially just pushing on the stick/adjusting the trim. If it had been implemented this way, sensor failure would not have been catastrophic and hence doesn't require redundancy. At some point this was either changed or implemented incorrectly so that MCAS had much more authority.
It was changed. Flight testing showed that much larger trim was required for MCAS to function, and that was implemented. The failure was not reassessing the risks after that.
That's pretty damning. I would have assumed Boeing of all people had a comprehensive change management system to automatically trigger a re-assessment in these cases.
They underestimated the aerodynamic instability and then decided just to give this auxiliary system total control. What possibly could go wrong? They should have gone back to the drawing board, putting of the launch and figuring out how to lift the airplane a bit higher. May have required them to lower the 737 prices for a while to stay competitive against the 320neo and may have cost them a fortune.
Feedback loops with high gain (strong engine pulling up, powerful MCAS pushing down) are difficult to control. Minor failures have out-sized impact. Adding rules to constrain the system introduces new operating modes all of them having the potential to confuse the pilots. And in this system the pilots are close to ground and have very little time to act.
Boeing offered the single angle-of-attack sensor as standard equipment, and charged extra for a second along with a “disagree” indicator that would allow 737 MAX pilots to “cross-check” a faulty sensor.
Unfortunately for the author that's not an accurate representation of reality. I'm a bit surprised as I thought that EETimes was a credible news source.
today i first time saw the 737 MAX frontal view. Initially i thought that it was that typical funny plane-themed photoshop. I kid you not, it is the real thing - https://i.stack.imgur.com/GFzcj.jpg
Just look at those nacelles. Deep breath. Look again. Take them in. Besides clearly visually screaming that this Frankenstein thing was quickly&cheaply slapped together and wasn't properly engineered and thus should just have never seen the light of the day, these nacelles obviously add more lift than normal symmetrical ones. So:
1. the engines placed more forward than pre-MAX 737 - that results in additional pitching up moment as the engines are below the centers of pressure, gravity, etc.
2. the engines are 2x higher-by-pass than pre-MAX 737 and thus the center of thrust is shifted even more forward and lower - as a result it adds even more of the pitching up
3. these asymmetrical nacelles generate more lift just due to the shape - and again due to the position of the engines that lift results in the additional pitching up moment.
Basically that thing just can't really fly steady straight, and looking at all this some people at Boeing decided that a bandaid software patch would just fix it. Sounds like it were the same people who did the "curl" fix in today Cisco story https://news.ycombinator.com/item?id=19508472 :)
I will not comment on (1) and (2) but (3) is wrong. I don't think that's even a 737 MAX on the picture.
737 was originally designed to be low to the ground to make it easier for ground crew to "bulk" load i.e. just throw stuff into the cargo area. The reason why 737 could be so low to the ground was that they were using turbojet[1] engines. Turbojet engines are very slim compared to new turbofan engines used on later 737 generations. When they moved to turbofan engines which have bigger diameter they needed to move them higher up from the ground. So they moved the engines in front of the wings. To gain even more ground clearance they moved accessory gearbox and fuel pump from underneath the engine to the side. That's why engine appears flat on the bottom. Obviously the engine is still round because the fan is round but the lip is flattened. All this allowed Boing to fit more efficient and quieter engines to 737 without extending landing gear. The shape itself does not generate much aerodynamical lift if any.
On the 737 MAX they actually extended the landing gear and it does not feature "flattened" engine shape any more.
afaik unstable airframes are the norm for some military/experimental planes (fighter jets for example) to allow more manoeuvrability.
It's definitely not something you want (or need) on a commercial airplane, it adds a ton of complexity and you're not going to do a high G turn or a cobra manoeuvre with a 737.
If you want to fly from Germany to the Canarian Islands for 50 EUR round trip then you're going to end up with unstable airframes.
The 737 MAX is in part unstable due to repositioned engines. They are larger and thus significantly more fuel efficient, however they would not keep enough ground clearance if they were positioned the same way as on the 737-NG. MCAS corrects for the repositioning, otherwise the plane may start to climb which in turn could lead to a stall.
B follows from A because in part A makes B economically possible.
Reductions in fuel consumption factor greatly into ticket prices. It's easy to reduce services (luggage, refreshments, leg space) but reducing fuel consumption is entirely up to the aircraft manufacturers.
But you don't need instability to get fuel economy, all you need is an airframe that actually matches the current engines instead of a relic from the 1960ies that was designed for tiny inefficient low-bypass engines. A plane designed for the new engines could be perfectly benign without sacrificing efficiency. Even a more properly adapted 737 that had the changes required to solve the problems aerodynamically instead could be fuel efficient.
Things that lead to instability can help, look at Boeing's MD-11. The horizontal stabilizer was made smaller than the DC-10 for fuel efficiency but resulted in a plane that was a handful to land safely.
I can think of many ways that achieve the same goal. Yes, most require re-cert, as they should. Trying to make an old tool do things it wasn't meant to do (see: Internet w.r.t security, 737 MAX w.r.t. engine placement) has a compounding effect that very rarely works out as well as a clean slate design with the right compromises in mind.
This is an example of a tragedy of the commons. Market forces found a solution to a problem (for one actor) that externalized long term costs for the sake of sealing a short-term deal, that in the end, was not guaranteed to stick around.
This appears to be a thinly sourced article, based on a blog post opinion by someone who is a software developer and instrumented-rated (hobby?) pilot who has flown airliners in simulation only.
With due respect, not sure whether that constitutes enough expertise to to qualify and opinion as news-worthy?
It would be interesting if these kind of companies (aviation, car companies) were forced to publicly disclose the patch they are applying in order to fix a broken piece software.
Maybe then they'd be more careful because of the extra scrutiny and the potential leaking of secrets.
On the other hand, maybe then they'd patch as little as possible, although in this case, if a second patch would be required, a very hefty fine could be forced onto the company, or possibly force a full disclosure of all the relevant source code.
Maybe the Blockchain could be used for some accountability here, where hashes of the blobs of all the software in the system, including the secret one, could be used as a means to prove that only a specific section of a codebase has been altered.
Everything made in US has become extremely expensive. As a product manufacturer you have to pay a lot more than your Asian competitors. You can open an R&D subsidiary in Asia to reduce costs, but in very short time you will see your technology has diffused and now you have even more pressure from competitors.
More bugs and design faults. Growth, innovation, effectiveness, all in a shorter time. All with increasing costs.
And even more complexity, and more pressure. iCloud leaks, empty root password, reboot by WhatsApp message, Meltdown, Spectre, 737 MAX, etc.
As an example of the seriousness of approach to stall, buffeting and proper recovery (you don't fall out of the sky like a rock, but recoveries take hundreds and possibly a couple thousand feet), this two year old serious incident in a 747 involving inadvertent stall while entering a hold, just had its final report issued.
http://avherald.com/h?article=4a787699&opt=0
You can't really fix any hardware fault in software. The best you can do is a workaround, but the whole will never be as solid as a properly designed system would be.
My bet is that Boeing will spend some time banging on this and then the 737 goes back to being a money maker for them for years to come. I'm guessing that at this point they know what is the problem and are probably pretty far done with coming up with many solutions and picking one that gets the job done. The way this works is that they have to do something because they are grounded.
This may or may not involve installing some extra hardware but it will most certainly involve a software update.
The big question is ... who will trust the Boeing - FAA duo after this? The 777X is coming, there surely will be rather pointed questions from airlines, the EASA and more.
I think many do not understand typical practices of regulatory agencies. As a related example, what do you think the FDA requires in terms of genetically engineered foodstuffs? Many seem to think there's extensive oversight and safety testing. There isn't. They treat genetically engineered products and natural products identically. If a company has all their regulatory issues in order to market e.g. corn, they can cook up a new genetically engineered corn in the lab and bring it to market with literally 0 additional oversight necessary. All the FDA offers here is a completely voluntary consultation, and that in turn basically is little more than the company signing off on some checkboxes.
This leads to a bemusing and disconcerting run around.
Monsanto: "The Food and Drug Administration (FDA) is responsible for the safety and appropriate labeling of food and feed products grown from GM crops." [1]
FDA: "It is the manufacturer's responsibility to ensure that the food products it offers for sale are safe and otherwise comply with applicable requirements." [2]
Sound similar? It'll be the exact same story if/when a company inadvertently releases a harmful genetically engineered product. The assurance of safety provided by regulatory agencies is often illusory. As an aside, this is all clearly described on the FDA's page as well. [3] But the phrasing is designed to mislead consumers. They state repeatedly that it is unlawful to ship unsafe food to consumers without ever directly clarifying that they themselves never actually test the foods. Inventions go straight from Monsanto's lab to your plate. Obviously they have a major incentive to ensure their products are safe, but they have a long history of failing in that obligation yet remain a multi billion dollar company.
Food safety isn't a function of relatively small changes in the genome of plants you eat, this is pseudo-scientific nonsense. The "natural" corn or animals you eat also experience genetic drift, and the FDA isn't tasked with sequencing them and certifying each "change".
If the purpose of aircraft was to feed them to giants who'd digest them for their raw materials Boeing wouldn't need to certify the 737 MAX either. But aircraft are flown, so minute changes to their construction can make a lot of difference. This comparison of yours makes no sense.
Boeing obviously felt there was a basically 0% chance of their decision being in anyway unsafe. And they are, arguably, the most qualified people on this Earth to decide this. Of course they probably got blinded by profit a bit, but it's not like this was a Ford Memo moment. A single plane going down is a catastrophe. Two planes going down is something much worse. They obviously felt everything was perfectly safe; they were wrong. Lots of people died. Even though the most likely outcome is they'll get a slap on the wrist, I think there's no way they would have gambled on this.
The reason I mention this is because I don't believe you believe it's impossible to create an unsafe product as you are implying in your statement. Genetic engineering technology enables us to hybridize anything. As a not entirely random example you could combine an orange with genomic data from an arbitrary virus or perhaps certain aspects of various plants in the nightshade family, if you so wished. You can theoretically do great things with genetic engineering, and you can certainly also do awful things. And there is no doubt that you can also accidentally do awful things. And I don't think short term safety is the real concern. You're not going to drop dead after drinking a cola because of some genetically engineered corn syrup in it. My concern would be longterm unforeseen consequences.
For instance weight gain, fertility, and even cognitive and psychological factors are all connected to what we consume in various ways that remain poorly understood. And we're currently running a compulsory experiment in that nearly all foodstuffs in the US now contain substantial components of genetically engineered products. The rest of the world works as a control, to varying degrees, due to radically less consumption of engineered products. What will be the longterm consequences of this? Perhaps we're already seeing them. Or perhaps the issues plaguing the US are caused by something altogether different. The point I was making is that it's ultimately up to the individual to come to their own decisions here. If you're happy to consume any genetically engineered product in full faith then I fully respect your view, even if I might disagree with the soundness of it [1]. I'd ask for nothing but comparable treatment.
> I fully respect your view, even if I might disagree with the soundness of it. I'd ask for nothing but comparable treatment.
I'm not claiming you have to eat GMO food, or food that's been exposed to cell phone tower waves or whatever.
But you weren't expressing a personal preference. You were suggesting that a government organization like the FDA should be regulating something based on a hypothesis that the current scientific consensus shows is baseless.
At that point you aren't asking for your view to be respected, you're suggesting that government policy should be changed to enforce it on the rest of us.
I did link to an overview of much of the current state of the science. You linked to a pop science article written with the impartiality and professionalism of a Breitbart article, though it does in turn reference something meaningful. Here [1] is the actual report from the NAS that that page references. They comment directly on our little discussion. Page 513: "FINDING: Not having government regulation of GE crops would be problematic for safety, trade, and other reasons and would erode public trust."
It also goes into detail on the problems with "weak" regulatory regimes. I put "weak" in quotes as any genetic engineering specific regulatory regime would be stronger than the US' reliance on self regulation. For instance in one study referenced (page 194) scientists ran a typical regulatory test (90 day whole food study) with rice that was genetically engineered specifically to be toxic. And indeed it was toxic. But over the standard 90 day test, no ill effects were found. This is a quite a serious problem.
And the one final thing I'd hit on is that much of the research on genetic engineering is driven by the companies that stand to profit from proving everything is safe and beneficial. Similar to how at one time nearly all science on e.g. leaded fuel was driven by interests that had a motivation to prove that everything was safe and beneficial, and so that's exactly what they did.
The NAS paper when discussing rat studies mentions, "Some found no statistically significant differences [from consumption of genetically engineered feed], but quite a few found statistically significant differences that the authors generally did not consider biologically relevant, typically without providing data on what was the normal range." later emphasizing again after discussing various dismissed abnormalities detected in rodent studies that "There was no presentation of standards used for judging what would be a biologically relevant difference or for what the normal range was in the measurements." In other words statistically significant differences were simply completely dismissed as "biologically irrelevant", without ever defining what would actually be considered biologically relevant. That's not good science, to say the least - but it's the typical pattern in much of the research for GE products, which tends to rely heavily on direct or indirect industry funding.
And, I think you'll find your view that negligible regulation is acceptable to be something very few outside of those directly connected to the genetic engineering industry would find satisfactory. The only reason more people do not voice concern is because they're generally completely unaware of the lack of safety inspections for these products. This state of 'regulatory subterfuge' is itself reason for a significant degree of cynical skepticism. You want to regulate? Ok. You don't want to regulate? Ok. You don't want to regulate, but strongly imply that you are? That's not ok.
The key failure of the MCAS system I have not seen discussed is that if it is overriden, and triggers again, it cranks the trim another notch. Trigger it five times, and each time it makes the plane less flyable. When Lion Air crashed, it had been triggered many times.
Making MCAS pay attention to two sensors might help a bit, but the disaster is still latent. Once it trims, it should never trim again without a full reset back to baseline. There are standards relating to this sort of thing in flight assist, about how much "authority" an automated system may assert, in total, and they were ignored, apparently because they did not treat it as part of the autopilot system.
If the standard had been observed, the bad sensor could not have had much effect on the flyability of the plane. The pilots would have needed to apply some force to keep the nose up, but would have succeeded, long enough to discover a fix or to turn around and land.
Somewhat off-topic but this has been bothering me: Several years ago I was in a program review and ended up in an argument with the lead Boeing software QA person for a particular group. The disagreement was because the person made a blanket statement that their QA process ensures there are no defects in their flight software. My response was that such a statement is absurd and that all software beyond some minimal complexity has defects. (A statement that I still agree with even though it is hyperbolic.) None of this has any direct relationship to the 737 Max issues as this wasn't even an airplane program but I think it points to what might be a cultural flaw if this attitude is widespread.
> ensures there are no defects in their flight software.
"Testing only proves the presence of defects, not the absence"
However, I think you are talking past each other here. The person you taled to was likely defining "no defects" to mean "all 1000 boxes ticked on the spec/testing protocol". They should call it "known defects".
What if it doesn't? I can see two ways in which they risk ending up with "major" problems
- They need to make changes that means it's no longer a 737 for certification.
- They need to make changes that delay production and even mean recalls of built plane, say moving engines rearward which in turn would mean big airframe/landing gear changes to manage ground clearance. These would be very expensive, cause tons of cancelled orders, and possibly also cause the same issue with certification as the above.
Any of those chnages would maybe kill the whole MAX program. And that would leave Boeing without a competitive plane in the most common class?
Is this a possibility?
Why does an active MCAS system need to exist in order to tamp down on pitch up? I wonder if modifying the surface of the wing and or tail could achieve this. Eg, make a computer model of the plane, verify that it captures the real pitch up behavior, and “evolve” a wing that counteracts it. I suppose the hard part is that we are searching for a perhaps complicated non-linear response, that needs to behave differently across speed, atmospheric conditions, pitch and turning. But maybe there are enough degrees of freedom when evolving a surface that it can capture it all?
The source article is attracting some heat. But, this aside, does anyone else think the meta-questions about the FAA and their relationship to Boeing, and the 'same type' certification process begs questions?
I "get" that people wanted this. But a regulator has to ask a subtly different question: is this actually in the wider public interest which is not neccessarily what Boeing wants.
>> “By laziness, I mean that less and less thought is being given to getting a design correct, and simple – up-front,” he wrote.
Competitors have more less expensive engineering resources. You simply cannot produce high-quality, safe, stable complex systems in shorter time if you have limited human-engineering very expensive resources. This is not only related to aircraft-manufacturing.
"Among Boeing’s critics is Gregory Travis, a veteran software engineer and experienced, instrument-rated pilot who has flown aircraft simulators as large as the Boeing 757."
Ok, what? I'm a veteran software engineer and I've flown (MS) flight simulators such as the 747 (badly), and even I know that none of that gives me any grounds to weigh in on this situation.
What causes such inaccurate AoA readings? 'Vanes' freezing?
From what I understand MCAS is about pointing up/down, so although AoA can of course be a more sophisticated angle in 3D space, couldn't the measurement for MCAS purposes be accomplished with a sensor based on gravity that could be entirely internal to the aircraft and so perhaps more reliable?
> Suspended mass inside container fixed to plane structure, measure angle of mass relative to container?
Unfortunately not! This measures acceleration of the airframe, not gravity. Was a huge challenge in the early days of flight, and there’s a great history online that I can’t find right now. Anyway, this might be useful reading: https://en.wikipedia.org/wiki/Attitude_indicator
> That doesn't sound desirable though, is it?
Given that wing-angle-to-airflow is what matters when it comes to stalls, I’d say it’s eminently desirable?
That is how lift is created. Air currents are not consistent over the plane. Sometimes they could more downward which would require a pitch up of the wings to catch and create lift. This is not the same as perpendicular to gravity which is easily measured in the plane. This is an entirely different animal.
What's this about "engineering incompetence", "engineering ethics" and "“cultural laziness” within the software development community"?
Do we have any reason not to believe that management is at fault? That management forced engineering to do things quicker/cheaper and cut corners and hack out a solution in those interests?
Management pressure is not an excuse for faulty engineering. As an engineer, your first duty is to the public, then your client, then your employer. That's straight out of the engineering ethics handbook. Your boss comes third.
When you sign-off on a design, it is your approval. Your name goes on the document. There's no passing that off as someone else's fault. If you are unable or unwilling to say "no" to your employer, then you do not meet the criteria to be a professional engineer.
Your employer may find another engineer to sign-off, but perhaps that person will think on it carefully, knowing that someone else refused. If something later went wrong, it could not be passed off as a simple oversight.
While management is definitely culpable, aviation is not like most software development shops in that legally speaking, your engineers do have final say. If they don't sign, nothing ships. An engineer is expected to do more than acquiesce to management's demands. It is, in fact, an engineer's job to say "no" to any implementation plan that jeopardizes the public.
There is a case to be made whether it is reasonable to expect an engineer to branch their scrutiny beyond what small slice they are given, in which case there was a failure at a higher level to distribute appropriate units of work such that the entire thing was thoroughly vetted. The problem is, that is extremely difficult to prove in a court of law, and it is not uncommon for high level management to hide unethical practices behind indirect communication methods without committing them outright to paper.
Call me a romantic, but at some point, as a bunch of engineers on a project, you need to be able to fully grok and model the entirety of the system as a whole. It seems that level of integration would have brought these problems to light much faster, and without the bloodshed. I can't imagine that work not happening unless it was being actively discouraged. If one or several engineers had pushed back, or a manager been thinking more about product quality/safety rather than delivery/profit, that work may have been done, and we wouldn't be talking about this now.
This article is clearly clickbait. I don't think they've done a good investigative job. Even though I don't have any pilot license, I have flown "full" 747, 767, 777 and 787 simulators, but this doesn't give me the authority to make broad statements about an airframe being faulty or not.
It's odd to call the airframe "faulty" when (a) it flies and (b) the FAA requirements for positive static stability in the airframe itself don't forbid the 737 MAX from being flown.
Does the author mean the FAA airframe acceptance criteria are faulty?
you can only make so many systems redundant before the weight increase and lower capacity outweighs any potential risk of systems failing, or this is how it seemed to me when i compared cargo carriers to other aircraft. you need constant dedicated maintenance and regular troubleshooting, i don't understand how more ptoblems like this haven't already occured. there is only so much you can do without a reliable pilot to disable anything causing problems. and having seen firsthand the problem solving methods and goto solutions for maintainers, everytime a plane lands successfully it's a miracle
I can't understand how a switch this simple could fail, frankly. Wouldn't a Mercury switch with an arc'ed path and several electrical contacts work? It seems blazing simple and idiot-proof. We are not being told the entire story! STAY TUNED!
Modern fighter aircraft are often designed to be aerodynamically unstable. This allows the removal or reduction of control surfaces, design of an airframe that is shaped to produce a small radar signature, and by naturally being unstable the craft wants to move around wildly so it has increased pitch, roll and yaw potential.
The software is then used to automatically move control surfaces to maintain stability. This is a deliberate design decision and results in a stealthier, faster, more manoeuvrable aircraft.
Passenger aircraft on the other hand are designed to be as stable as possible, to ensure stability and control authority are maintained by default as a failsafe.
Changing the design of an aircraft that is assumed to be stable by default and introducing software to compensate is a very different proposition to building an airframe from scratch with the explicit intent of it being unstable.
The failure mode of an airframe that's presumed stable should be that it will at the very least be able to glide down a reasonable descent slope. The failure mode of an unstable airframe is pilot ejection and loss of the airframe typically, although military aircraft failure tolerances are quite different (the computer should be able to compensate for damaged components no longer responding as intended etc).
So in answer to the question, it's not really comparing apples with apples. I would say that a passenger airliner should be able to maintain reasonable balance without constant interjection of a computer.
But moreover, if a change in the airframe causes an imbalance that only software can correct, the manufacturer should inform and train the purchasing airlines and their pilots in these new software systems. The fact they apparently did not has caused the deaths of hundreds of people.
For that reason I'd say as a whole the 737-Max is excessively, perhaps criminally faulty.
Even the F-16 was controlled by software. I'm not sure if the F-16 was aerodynamically unstable, but some newer jet-fighters are, specifically for more maneuverability, and they rely on software to keep it flyable.
However, those tend to be designed with redundant systems in order to avoid a single point of failure. Having a single poorly tested system rely on a single sensor to fix a fatal flaw in the airframe is the real issue, I think.
But maybe a passenger plane shouldn't be designed to be unstable in the first place; it's not meant to be a jet-fighter.
Fighters also have ejection seats, which provide the final fail safe mechanism. So you can use more radical and risky solutions without unduly putting the pilot at risk. I imagine the calculus works differently for bombers, cargo, and helicopter aircraft.
The F-16 requires specifically trained pilots to fly safely.
The Boeing in question was flown by pilots who were not trained for or even made aware of significant modifications to the plane's behavior.
A training wheel on a motorbike is not very hard to make well or drive with safely, but if you suddenly discover it during a sharp turn at speed, it's not going to go well.
Further to my comment above, here's the line from the Wikipedia entry, which I vaguely recalled:
The F-16 is the first production fighter aircraft intentionally designed to be slightly aerodynamically unstable, also known as "relaxed static stability" (RSS), to improve maneuverability.
1. The F-16 is the first production fighter aircraft intentionally designed to be slightly aerodynamically unstable, also known as "relaxed static stability" (RSS), to improve maneuverability.
The author mentions this engine in the article and there is a neat video of the behemoth engine. Pure mechanical engineering excellence ! https://www.youtube.com/watch?v=5CytG5M5Jcs
#/media/File:Tail_of_a_conventional_aircraft.svg){kind=link}
One possibility is incompetence. But Boeing engineers are smart people, so I'm not convinced by this. The elephant in the room is the requirement to maintain a common type rating with older 737 models.
Suppose they did originally do what the fixed software does now, and disable MCAS if the AoA sensors disagree. The problem Boeing face is that with MCAS disabled when this occurs, the plane no longer flies like an older 737. They'd need to announce to the pilots an AoA disagree, and announce that MCAS was disabled. Now what? A pilot certified and trained on the older 737 would not know how the Max now differs from what they trained on. If they'd done this, they'd have needed to provide additional training, and this must have concerned Boeing management that it might jeopardize the common type rating. Hence it seems likely they didn't add the AoA sensor comparison for this reason, reasoning that it was unlikely to be a problem anyway. We now know that reasoning was flawed.
What does this mean going forwards? Will EASA and other CAAs refuse to certify the modified 737 Max under the same type rating as the older 737? This certainly seems possible. If they did require a separate type rating, this would likely kill 737 sales, regardless of whether the plane is now safe.