Lies, damned lies and statistics

07/03/2011 — Leave a comment

20110122-093121

Why we may be carrying an order of magnitude greater risk from aircraft engine rotor bursts than we thought

One of the ‘implicit’ conclusions of the 2010 AIA study on the threat posed by jet engine rotor bursts was that the fleet of modern aircraft designed to meet FAA circular AC 20-128A also met the FAA established safety targets of a 1 in 20 likelihood of a catastrophic loss, in the event of a engine rotor burst.

To quote the authors of the AIA report:

The high bypass turbofan fleet, as a whole, has experienced 58 disk uncontainment events over the time period considered, three of which resulted in loss of the airplane. The results are consistent with the 1 in 20 criterion used during certification analysis, even though many (75%) of the events occurred on airplanes designed and certified before introduction of this criterion.

AIA report (Vol 1 Page 10).

But can we really trust the numbers? A simple statistical check should suffice to confirm whether the AIA were justified in making such an assertion. Assuming a random rate of catastrophic loss of  f= 1/20 we then have a MEBF = 1/f = 20 rotor bursts (MEBF = Mean Events Between Failure).  Calculating the lower one-sided confidence interval for a confidence level of (95% (1))

The number of catastrophic failures (r) = 1 (2).

Degrees of freedom (d.f) = 2r + 2 = 4

Alpha (α) = 0.05

From Crow (1960) Chi Squared (χ exp 2)< 9.488

Therefore.

MEBF (lcl) ≥ 2 x MEBF / χ exp 2

MEBF (lcl) ≥ 2 x 20 / 9.488

MEBF (lcl) ≥ 4  (rounding up)

As MEBF is the reciprocal of f this gives a lower confidence limit for f of no greater than a 1 in 4 chance of a catastrophic event occuring as a consequence of an uncontained rotor burst. So being as charitable as possible (3), we can only say with confidence of 95% that the catastrophic accident rate is less than 1 in 4.

Even  if we assume, as the AIA has, that current generation of aircraft have exhibited no catastrophic rotor events (note this was before QF32) then using the above method we still end up with a 3 in 20 failure rate as the lower confidence limit. Closer but still no cigar.

So we are forced to conclude, and contrary to the AIA, that field data is not consistent with the 1 in 20 safety target of AC 20-128A, unless we use ‘consistent’ in an effectively meaningless fashion. In the worst case this analysis points to a possible worst case 1 in 4 likelihood of a catastrophic accident upon a rotor burst event.

The problem the AIA report authors ran into illustrates the problem of dealing with complex man made systems that sit between the two extremes of purely random behaviour (e.g. gas laws) and purely deterministic (such as structural design). Their behaviour is complex enough so that deterministic ‘classical’ analysis does not work and yet there are not enough that statistical analyses can get much traction. The report also exhibits what I would term a subtle confirmatory bias where the authors failure to rigorously analyse the data allows them to draw conclusions based on ‘ambiguous’ data.

Notes

1.  This is an arbitrarily assigned value but most analysts use a figure that sits ‘around’ 90% value.

2.  Of the three ‘catastrophic’ events only one fits the level 1 severity criteria of the AIA report. To be as charitable as possible this is taken as the single catastrophic loss.

3.  A long standing tradition in the criticism of any logical argument. The goal is to give the most charitable, and therefore strongest version of the argument on the author’s behalf that we can to avoid prematurely rejecting it on spurious grounds e.g. because of confusion or ambiguity rather than a deeper flaw.

References

1. AIA Working Group, AIA Report On High Bypass Ratio Turbine Engine Uncontained Rotor Events And Small Fragment Threat Characterization Volume 1, January 2010.

2.  Crow, E.L., Davis, F.A, Maxfield, M..W. Statistics Manual, NAVORD Report 3369 NOTS 948, Dover Publications New York, 1960.

3. Federal Aviation Administration (FAA) AC 20-128A Design Considerations for Minimizing Hazards Caused by Uncontained Turbine Engine and Auxiliary Power Unit Rotor Failure, 25 March 1997.

No Comments

Be the first to start the conversation!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s