Battery tests and experimenter’s regress


787 Lithium Battery (Image Source: JTSB)

But, we tested it? Didn’t we?

Earlier reports of the Boeing 787 lithium battery initial development indicated that Boeing engineers had conducted tests to confirm that a single cell failure would not lead to a cascading thermal runaway amongst the remaining batteries. According to these reports their tests were successful, so what went wrong?

The unfortunate problem you strike in conducting such experiments is that evidence is never independent of theory. In effect good evidence is produced by good experimental techniques but what’s a good technique is only determined by whether it determines the results that we expect.

This problem of regression is particularly important in new areas of research (like large scale Li-Ion battery designs for aircraft) where there’s no recognised consensus about the relative value of competing theories and error sources are poorly understood.

In engineering research the problem is exacerbated because most engineers don’t spend a lot of their time working on development projects while the body of engineering knowledge and the engineering community are in reality quite fractured.

For example, even if large scale Li-Ion batteries have been developed for road transportation it’s difficult given the parochial nature of most industries for that knowledge to travel out of that industry and into another (1). Within an industry engineering knowledge can be further siloed into competing organisations. Does Boeing tell Airbus about it’s development problems? I don’t think so, and so getting a consensus in engineering is quite a difficult thing to achieve ‘on the fly’.

In this case the Boeing engineers expected that their battery design would work, they conducted a test and because they got the results they expected they stopped right there. This may be appropriate for an incremental improvement to an existing design within a well established certification regime, but not so much for a cutting edge project.

We could also view their error as a form of confirmation bias but if that’s true we should also remember that the problem of experimenters regress had set them up for such an error.


1. Having worked in aerospace, marine and rail this is something to which I can personally testify. 🙂

One response to Battery tests and experimenter’s regress

    Thomas Perkins 13/06/2013 at 6:22 am

    Indeed. Experimental regression is a constant foe that developmental engineering have to fight against. Even for something as “simple” as conduction heat transfer or moisture diffusion within an aircraft fuselage, theoretical models are only of limited practicality. Especially is this problematic given the high level of safety requirements mandated by the FAA on the aerospace industry. Some handy questions to answer are:
    1. What is the theory behind the experiment, i.e. why I am choosing to measure particular data?
    2. What is the basis of confidence for my results? Is it based off of observed phenomena, numerical simulation, theory, or just a gut feeling of how things should work?
    3. When unanticipated results are obtained, what are the missing factors? Is the experiment setup suspect or are there fundamental parts of the theory that I am missing?

    Of course these questions should be second nature to any science or engineering undergrad, however I think that this kind of introspection is often lost in a high pressure developmental project such as the 787 Li battery.