But, we tested it? Didn’t we?
Earlier reports of the Boeing 787 lithium battery initial development indicated that Boeing engineers had conducted tests to confirm that a single cell failure would not lead to a cascading thermal runaway amongst the remaining batteries. According to these reports their tests were successful, so what went wrong?
The unfortunate problem you strike in conducting such experiments is that evidence is never independent of theory. In effect good evidence is produced by good experimental techniques but what’s a good technique is only determined by whether it determines the results that we expect.
This problem of regression is particularly important in new areas of research (like large scale Li-Ion battery designs for aircraft) where there’s no recognised consensus about the relative value of competing theories and error sources are poorly understood.
In engineering research the problem is exacerbated because most engineers don’t spend a lot of their time working on development projects while the body of engineering knowledge and the engineering community are in reality quite fractured.
For example, even if large scale Li-Ion batteries have been developed for road transportation it’s difficult given the parochial nature of most industries for that knowledge to travel out of that industry and into another (1). Within an industry engineering knowledge can be further siloed into competing organisations. Does Boeing tell Airbus about it’s development problems? I don’t think so, and so getting a consensus in engineering is quite a difficult thing to achieve ‘on the fly’.
In this case the Boeing engineers expected that their battery design would work, they conducted a test and because they got the results they expected they stopped right there. This may be appropriate for an incremental improvement to an existing design within a well established certification regime, but not so much for a cutting edge project.
We could also view their error as a form of confirmation bias but if that’s true we should also remember that the problem of experimenters regress had set them up for such an error.
1. Having worked in aerospace, marine and rail this is something to which I can personally testify. 🙂