Boeing made erroneous assumptions on pilots’ response to alerts in 737 MAX, NTSB says

The National Transportation Safety Board (NTSB) on Thursday criticized Boeing’s testing of the 737 MAX for evaluating in isolation the flight-control system that later went haywire, rather than testing it with an overload of warning messages in the cockpit as occurred during the two fatal crashes of the jet.

In a report based on the MAX accident investigations, the NTSB also said modern jets need to have much better designed pilot alerting systems, going beyond what current jets have — and much beyond the legacy systems on 737s.

“We’ve identified a gap in the human/machine or human/airplane interface,” said Evan Byrne, the NTSB’s chief of human performance and survival factors in the Office of Aviation Safety, in a press briefing. “We are trying to close that gap as it relates to multiple alerts going off simultaneously.”

The NTSB made seven recommendations for better testing that takes more account of pilot responses. Such tests should be conducted on the MAX before it’s allowed to fly again and on future airplane-certification programs, said Dana Schulze, director of the Office of Aviation Safety.

She added that Boeing’s original testing of the MAX followed the requirements laid down by the Federal Aviation Administration (FAA). And in the aftermath of the two fatal MAX accidents, she said, Boeing may already have done the enhanced tests the NTSB is asking for during testing of its upgraded flight-control system.

When the 737 MAX was tested in 2016, Boeing was required to test only systems that had changed since the previous 737 NG model, including the new Maneuvering Characteristics Augmentation System (MCAS).

This system was designed to “trim” the airplane by swiveling the plane’s horizontal tail, or stabilizer, to push the jet’s nose down and counter a tendency to pitch up.

Boeing pilots conducted tests in a flight simulator in which the stabilizer was induced to move just as if MCAS had activated once, as designed, and the test pilots then demonstrated their ability to maintain control. “They did not look at all the potential flight deck alerts and indications the pilots might face,” said Schulze.

In the accidents on Lion Air Flight 810 last October and Ethiopian Airlines Flight 302 in March, MCAS was activated by a faulty signal from an angle-of-attack sensor. But before MCAS kicked in, that same fault set off a barrage of cockpit alerts.

The captain’s control column began vibrating (a “stick shaker”), while a master caution light illuminated to indicate system faults and other lights indicated both the altitude and airspeed readings were unreliable. A computer-generated voice warning of low speed may also have sounded at some point.

On the Ethiopian flight, the pilots also heard repeated, loud “DON’T SINK” warnings that the jet was too close to the ground and a “clacker” making a very loud clicking sound to signal the jet was going too fast.

“We want the FAA to ensure Boeing takes a look at all these different failure conditions,” Schulze said.

Schulze said Boeing assumed in its testing that pilots would “immediately identify (the MCAS) unintended trim action” and then “immediately take action” to counter it, by using thumb switches on the control column to pull the nose back up and if necessary to hit two cutoff switches and stop all automatic stabilizer movement.

Boeing’s assumptions proved wrong. With those distractions in the cockpit— and in the case of the Lion Air flight, with the crew having no prior knowledge of MCAS, which Boeing had omitted from the flight manuals — the pilots did not diagnose the problem they were facing and failed to respond as Boeing expected.

When MCAS re-activated repeatedly with just five-second breaks, which wasn’t anticipated in Boeing’s design, it eventually overwhelmed  them.

Najmedin Meshkati, a professor of engineering at the University of Southern California who has taught a core course on Human Factors in Aviation Safety since 1989, said in an interview that Boeing’s “poor human-systems integration practices, testing and evaluation” contributed to the bad design of MCAS.

“Uncertainty in a cockpit … breeds unknown reactions,” he said. “It changes the nature of the task at hand, deforms operators’ mental model, erodes confidence, causes delay and could lead to confusion.”

Meshkati cited a 1996 FAA report that identified weaknesses in flight crew management of automation so that they lost situation awareness.

“This needs to be re-addressed all over the globe, as similar scenarios continue to be present — and sometimes fatal — in the aviation industry,” he said.

Schulze emphasized that the NTSB report does “not address specific issues of pilot performance” in the two crashes. Instead, “we’re looking at issues in the U.S. process used to certify” the MAX, she said.

Following the crashes, Boeing has updated the MCAS software to address various design flaws and has been extensively flight-testing it to win regulatory approval so the planes can fly again after being grounded for more than six months. Schulze said the NTSB understands that Boeing in the process is reevaluating “all the safety assessments for the flight controls overall.”

If so, it’s likely the jetmaker has already completed what the NTSB is calling for.

The NTSB recommendations go beyond the MAX. They are also asking the FAA to re-evaluate whether other airplanes besides the 737 MAX may also have major system vulnerabilities undetected during certification testing.

And the NTSB also recommends that in future airplane certification programs the FAA work with specialists in human factors to develop tools and methods that will better gauge the response to system failures in real-world situations.

Schulze said that could involve bringing in random samples of line pilots from around the world to help with testing, which is something Boeing and the FAA did this year during testing of the MCAS update.

All Boeing airplanes besides the 737 have a more modern centralized pilot alerting system called EICAS, or Engine-Indicating and Crew-Alerting system. This state-of-the-art system provides pilots visual, aural and tactile warnings as well as written messages on the main flight display when anything goes wrong, and then also recommends the remedial action needed.

Schulze said the NTSB is also recommending that the FAA develop new improved alerting systems that give pilots a clear priority of what to do, for example, telling them which checklist they need to run first when different error messages pop up at the same time.

“EICAS does some of that. We think we need to go to the next level,” Schulze said. “We want to see the FAA work with the manufacturers and human factors experts to develop a better design standard.”

That’s a future aspiration.

However, Boeing’s 737, its oldest jet, doesn’t even have EICAS. Behind its sleek-looking pilot flight displays, the jet’s legacy avionics systems have been upgraded piecemeal over 50 years, and the overall system architecture won’t support EICAS.

Installing EICAS on the 737 “would be challenging,” said Mike Carriker, Boeing’s chief pilot for product development, in a brief interview. “There aren’t enough sensors on the 737.” Even if it were possible, it would require a new type certificate and new pilot training.

During development of the MAX, Boeing’s customer airlines made clear they don’t want to pay for such an upgrade.

Boeing spokesman Chaz Bickers did not immediately comment on the difficulty of upgrading the 737 ‘s systems.

“We value the role of the NTSB in promoting aviation safety,” Bickers said via email. “We are committed to working with the FAA in reviewing the NTSB recommendations.

The FAA in a statement welcomed the NTSB recommendations and said the lessons learned from the MAX crashes “will be a springboard to an even greater level of safety.”

“The agency will carefully review these and all other recommendations as we continue our review of the proposed changes to the Boeing 737 MAX,” the FAA said.