It was in manual mode. Should it forcibly take away control from the user that gave it a direct command, because it "thinks" they're not paying attention?
Letting a developer drop the production database[1] is not good either, but trusting a person to do their job correctly is baked into every organization and level. Still, accidents happen - there is no fool-proof way to stop humans from behaviours that are ultimately harmful. Elaborate systems will have elaborate loopholes.
For certain you cannot prevent all accidents, and I am not trying to say that we can/should.
I think having a sole operator in a prototype vehicle drastically increases the odds of these accidents, and that companies like Waymo should run them with two operators and some drowsiness detection systems (which are common on luxury cars). That’s a relatively simple fix.
It’s important that testing with multi ton moving vehicles be handled with far more cars than most production databases are, speaking to your metaphor. The consequences of mistakes is much much higher with cars than it is with most databases.
It never should have allowed the vehicle to proceed with a fatigued driver, and the company should have policies in place to prevent this (maxmimum hours, dual operators, etc.)
