Comparing and contrasting
In 2010 NASA was called in by the National Highway Transport Safety Administration to help in figuring out the reason for reported unintended Toyota Camry accelerations. They subsequently published a report including a dedicated software annex. What’s interesting to me is the different outcome and conclusions of the two reports regarding software. The NASA’s team concluded that it was not possible to engineer a test (for an unintended throttle movement) nor that any place was found in the software where a single memory/variable corruption could results in a unintended acceleration. But, perhaps wisely, the NASA team did not conclude that the software was safe, simply that they could not find a possible software fault that would induce an unintended acceleration.
Examination of the code found that throttle control variables are protected from corruption by storing multiple copies…Extensive software testing and analysis was performed on TMC 2005 Camry L4 source code using static analysis, logic model testing, recursion testing, and worse case execution timing. With the tools utilized during the course of this study, software defects that unilaterally cause a Unintended Accelerations were not found…
NESC Assessment #: TI-10-00618
In contrast Michael Barr’s team following on from the NASA assessment found that some critical variables were not protected from corruption via mirroring (apparently NASA did not know this), there was no hardware protection provided against single bit errors by hardware (again NASA did not know this) there were multiple possible software causes of memory corruption (neither Toyota or NASA knew that stack usage was actually sitting at 91% rather than 41%). Thus a single bit flip or software stack overflow could kill a safety critical task which would result in an unintended acceleration and the Barr team subsequently verified this failure mode through testing.
You heard Mr. Ishii say that NASA had trouble reading Toyota’s source code. That wasn’t to do with them not following MISRA, it’s because it was badly written and badly structured source code. And that’s spaghetti.
Barr testimony, Bookout vs Toyota trial transcript
On the surface the NASA report certainly seems a comprehensive and thorough effort. But in reality the analysis was conducted under constraints of time and access to information, which were not explicitly identified in the conclusions in the report. Reading through the NASA report and especially the Annex you can also see that while the information is there the whole lacks what I’d call ‘cut through’, that is the ability to step away from the detail and establish a clear appreciation of the overall situation.
How often have I said to you that when you have eliminated the impossible, whatever remains, however improbable, must be the truth?
Sherlock Holmes, The Sign of Four
Coupled with this lack of a clarity was an error of approach that I think the NASA team fell into. In essence they made an conclusion of ‘not guilty’ or ‘not proven’ on the basis of lack of a smoking gun, when what they were actually dealing with was a situation where there was a great deal of persuasive circumstantial evidence, in the form of violated coding standards, use of known dangerous constructs, poorly structured code and other software faults. All of which spoke to the guilt of the software. That coupled with the necessity to rely upon the assurances of Toyota alone for the presence of critical safety features (given Toyota’s demonstrated poor performance) should have resulted in a very different conclusion. In essence the NASA team seemed to have adopted a scientific approach to the evidence, where a more legal and sceptical approach would have served them better.