STS 134 Mission Patch (Image Source: NASA)

For the STS 134 mission NASA has estimated a 1 in 90 chance of loss of vehicle and crew (LOCV) based on a Probabilistic Risk Assessment (PRA). But should we believe this number?

Given that we have (unfortunately) real accident data we can run a straight forward confidence check on NASA’s conclusions.

Using the LOCV data (two losses over 133 flights or r=2) and selecting an arbitrary 90% confidence interval (a=0.1) for both upper and lower confidence levels.

Giving us a lower confidence level (1) for mean missions between LOCV:

• LCL = 2 X 133/9.49 = 28 missions ( with a 1 in 28 chance of LOCV per mission)

While the upper confidence level for LOCV is:

• UCL = 2 X 133/0.71 = 375 (giving a 1 in 375 chance of LOCV per mission)

A point estimate of the likelihood of mission loss for STS 134 results in a likelihood of 1 in 67 (based on 2 losses over 133 missions).

So based on the statistical evidence we can say that NASA’s current risk estimate of 1 in 90 for the shuttle lies within the 90% confidence interval defined by the likelihood range of 1 in 28 to 1 in 375.

What’s interesting is how NASA’s mission risk sits close to the point estimate, which is good as a sanity check on the much more complex Probabilistic Risk Assessment (PRA) upon which NASA’s conclusions are based.

The NASA presentation asserts that safety has grown across the program by an order of magnitude from a 1 in 9 chance of LOCV at STS 1 to the current risk level of a 1 in 90 LOCV at STS 134. This increase has taken time, money and often operational constraint to achieve.

What’s also interesting is that NASA’s PRA is (by definition) based on an ensemble of hazards that have been identified. Therefore the epistemic risk posed by unidentified hazards cannot be addressed in NASA’s PRA. So perhaps we should say that as long as we haven’t overlooked something the risk assessment is a reasonable one.

This bounded conclusion lies in stark contrast to the unreality of pre-Challenger risk assessments of NASA exposed by Richard Feynman.

The men and women of the astronaut corp earn their pay every launch. I wish the crew of STS 134 good luck and a safe return.

References

1. Canga, M. (Presenter) NASA Space Shuttle Safety and Mission Assurance Office, Space Shuttle Risk Review Presentation (Excerpt), Johnson Space Flight Center, Houston, 20 January 2011.

Notes

1. Where 9.49 is the Chi squared value for a/2, df=2r=4, while 0.71 is the Chi squared value for (1-a/2) and df = 4.