### Archives For software safety

A short tutorial on the architectural principles of integrity level partitioning,  I wrote this a while ago, but the fundamentals remain the same. Partitioning is a very powerful design technique but if you apply it you also need to be aware that it can interact with all sorts of other system design attributes, like scheduling and fault tolerance to name but two.

The material is drawn from may different sources, which unfortunately at the time I didn’t reference, so all I can do is offer a general acknowledgement here. You can also find a more permanent link to the tutorial on my publications page.

I was cleaning out my (metaphorical) sock drawer and came across this rough guide to the workings of the Australian Defence standard on software safety DEF(AUST) 5679. The guide was written around 2006 for Issue 1 of the standard, although many of the issues it discussed persisted into Issue 2, which hit the streets in 2008.

DEF (AUST) 5679 is an interesting standard, one can see that the authors, Tony Cant amongst them, put a lot of thought into the methodology behind the standard, unfortunately it’s suffered from a failure to achieve large scale adoption and usage.

So here’s my thoughts at the time on how to actually use the standard to best advantage, I also threw in some concepts on how to deal with xOTS components within the DEF (AUST) 5679 framework.

Enjoy 🙂

For those interested, here’s a draft of the ‘Fundamentals of system safety‘ module from a course that I teach on system safety. Of course if you want the full effect, you’ll just have to come along. 🙂

From Les Hatton, here’s how, in four easy steps:

1. Insist on using R = F x C in your assessment. This will panic HR (People go into HR to avoid nasty things like multiplication.)
2. Put “end of universe” as risk number 1 (Rationale: R = F x C. Since the end of the universe has an infinite consequence C, then no matter how small the frequency F, the Risk is also infinite)
3. Ignore all other risks as insignificant
4. Wait for call from HR…

A humorous note, amongst many, in an excellent presentation on the fell effect that bureaucracies can have upon the development of safety critical systems. I would add my own small corollary that when you see warning notes on microwaves and hot water services the risk assessment lunatics have taken over the asylum…

The QF 72 accident illustrates the significant effects that ‘small field’ decisions can have on overall system safety Continue Reading…

How do ya do and shake hands, shake hands, shake hands. How do ya do and shake hands and state your name and business…

Lewis Carrol, Through the Looking Glass

You would have thought after the Leveson and Knight experiments that the  theory that independently written software would only contain independent faults was dead and buried, another beautiful theory shot down by hard cold fact.  But unfortunately like many great errors the theory of n-versioning keeps on keeping on (1).

I was cleaning up some of my reference material and came across a copy of the ESA board of investigation report into the Ariane 501 accident. I’ve added my own personal observations, as well as those of other commentators, to the report. Continue Reading…

Recent incidents involving Airbus aircraft have again focused attention on their approach to cockpit automation and it’s interaction with the crew.

Underlying the current debate is perhaps a general view that the automation should somehow be ‘perfect’, and that failure of automation is also a form of moral failing (1). While this weltanschauung undoubtedly serves certain social and psychological needs the debate it engenders doesn’t really further productive discussion on what could or indeed should be done to improve cockpit automation. So let’s take a closer look at the Airbus protection laws implemented in the flight control automation and compare it with how experienced aircrew actually make decisions in the cockpit.

Authors Note. Below is my original post on the potential causes of the AF 447 cabin altitude advisory, I concluded that there were a number of potential causes one of which could be an erroneous altitude input from the ADIRU. What I didn’t consider was that the altitude advisory could have been triggered by correct operation of the cabin pressure control system, see  The AF 447 cabin vertical speed advisory and Pt II for more on this.

The last ACARS transmision received from AF 447 was the ECAM advisory that the cabin altitude (pressure) variation had exceeded 1,800 ft/min for greater than 5 seconds. While some commentators have taken this message to indicate that the aircraft had suffered a catastrophic structural failure, all we really know is that at that point there was a rapid change in reported cabin altitude. Given the strong indications of unreliable air data from other on-board systems, perhaps it’s worthwhile having a look for other potential causes of such rapid cabin pressure changes.

#### DO-178B and the B-777 9M-MRG Incident

In August 2005 a Boeing 777 experienced an in-flight upset caused by the aircraft’s Air Data Inertial Reference Unit (ADIRU), generating erroneous acceleration data. The software fault that caused this upset raises questions in turn about the DO-178 software development process. A subsequent investigation of the accident by the Australian Transportation Board (ATSB) identified that the following had occured:

• accelerometer #5 failed on the first of June in a false high value output mode,
• the ADIRU excluded accelerometer #5 from use in its computations,
• the ADIRU unit remained in service with this failed component (1),
• power to the ADIRU was cycled (causing a system reset),
• accelerometer #6 then failed in-flight,
• accelerometer #6 was excluded from use by the ADIRU,
• erroneous acceleration values were output to the flight computer.

The statement by, AirBus regarding the robustness of the AirBus AOA voting logic disclosed in the ATSB QF72 accident report raises some interesting questions as to what was actually meant by the term robustness.

The QF 72 in flight pitch upset demonstrates the vulnerability of a redundant and presumed fault tolerant systems to situations where the real world does not accord with the assumptions made by designers.

The use of integrity levels to achieve ultra high levels of safety has become an ‘accepted wisdom’ in the safety community. Yet I, and others, remain unconvinced as to their efficacy. In this post I argue that integrity levels are not scientific in any real sense of that term which leads in turn to the reasonable question of whether they work.

### Testability and disconfirmation

The basis of science is empirical observation and inductive reasoning. For example we may observe that swans are white and therefore form a theory that all swans must be white. But as Hume pointed out inductive reasoning is inherently limited because the premises of an inductive argument support but cannot logically entail the conclusion. For example, in our original example a single black swan is sufficient to refute our theory, despite there being a thousand white swans…This does not mean that a theory cannot be useful (that is it works), but just because a theory has worked a number of times does not mean that it is proven to be true. For example, we can build ten bridges that stay up (our theory is useful) but there is nothing to say that the eleventh will not fall down due to effects not considered in the existing theory of how to design bridges. As any test of a theory cannot prove the truth of a theory only disprove it, when we say a theory is testable we are not saying that we can prove it, only that there exists an opportunity to disprove or falsify it. This concept of disproof is very much akin to the legal principal of finding a person ‘not guilty’, rather than ‘innocent’ of a charge. Which leaves us with a problem as to how science really works, if we presume that it’s science’s job to prove things.

The response of the philosopher Karl Popper to this problem of induction was to accept this inability to absolutely prove the truth and conclude that because we can never prove the truth of a scientific theory, science has to advance on the basis of the falsification of existing theories and replacing them with theories that better explain the facts (Popper 1968). From Popper’s perspective a good theory is one that offers us ample opportunity to falsify it. Conversely a theory which is not refutable by any conceivable means is non-scientific. Irrefutability is in fact not a virtue of a theory (as people often think) but a vice (Popper 1968).  To achieve falsifiability, according to Popper, a theory therefore needs to be:

1. precisely stated (i.e. unambiguous),
2. wide ranging, and
3. testable in practical terms.

As a corollary if a theory does not satisfy these criteria it should not be considered scientific (Popper 1968). For example we could develop a design hypothesis  as to how we could design a bridge to span the straits of Gibraltar using as yet undeveloped hyper-strength materials, but as we have no practical way to test such a theory this should not be considered as scientific. Another way to look at it is that a ‘good’ scientific theory is a prohibition: it forbids certain things to happen. The more a theory forbids, the better it is because it gives us greater scope to falsify it. For example we build a very slender bridge using deflection theory (we do X) the design hypothesis forbids the bridge from falling down under specific deck loads (If X then not Y) which is eminently testable. Confirmations should also count only if they are the result of risky predictions. That is if based on our original theory or understanding, we expect an event that is incompatible with some new hypothesis then that would refute our new hypothesis. If on the other hand a new hypothesis predicts pretty much the same results as the accepted theory then there’s not much at risk. So in our bridge example if our new theory of bridge construction predicts that such a bridge will not fail under an known load, whereas the older theory based on traditional techniques predicts that it will, then there’s a clear, and testable, difference.

From an engineering perspective this means that the confirmation of a theory comes when it allows engineers to do something beyond the current state of the art. For example, we could use a new bridge deflection theory to design a lighter and more slender bridge span for given wind loads. If the bridge stands under the loads then the the results would count as confirmation of the new hypothesis as our old theory would have predicted failure. Confirming evidence should not count except when it is the result of a genuine test of our hypothesis and it can be presented as a serious, but unsuccessful, attempt to falsify the hypothesis and whose results corroborate the evidence (Popper 1968). Essentially our theories and hypotheses need to have some, ‘skin in the game’. Again using the bridge design example if both the original theory and the new design hypothesis predict the survival of a bridge this does not represent a genuine test of the new theory. But if that new bridge could only be built using the new hypothesis and the bridge subsequently falls down then that new hypothesis will inevitably be scrutinised and either rejected or adjusted, as happened with the Tacoma Narrows disaster, our new hypothesis has in effect lots of epistemic ‘skin in the game’.

### The scientific theory of Software Integrity Levels (SILs)

As the concept of a safety integrity levels is most entrenched within the software community I’ll stick with them for the moment, noting that the issues raised below are just as valid for safety integrity levels when applied to hardware. The theory of safety integrity levels for software can be expressed as follows:

1. Software failures are ‘systematic’ that is they result from systematic faults in the software specification, design or production processes,
2. As such software failures are not random in nature, given the correct set of inputs or environmental conditions the failure will always occur,
3. The requirement for ultrahigh reliability (for example 10E-9 per hour) of safety functions makes traditional reliability testing of software to demonstrate such reliability impossible,
4. The use of specific development processes will deliver the required reliability by reducing the number of latent faults that could cause a software failure but this comes at a cost,
5. Therefore based on an assessment of risk an ‘integrity level’ is assigned to the safety function. The higher the risk the greater the integrity level assigned,
6. This integrity level represents the required reliability of the safety function, and
7. To achieve the integrity level a set of processes are applied to the specification, design & production processes, these are defined as an associated software integrity level.

### Problems with SILs as a scientific theory

#### SIL’s are fundamentally untestable

Unfortunately even the lowest target failure rates for safety functions (e.g. 10E-5 per hr) are already beyond practical verification (Littlewood-Strigini 1993) therefore we have no practical independent and empirical way to demonstrate that application of a SIL (or any other posited technique) will achieve the required reliability (freedom from accident). So we end up with a circular argument where we can only demonstrate achieving a specific SIL by the evidence of carrying out the processes that define that SIL level (McDermid 2001).

#### SIL allocation is non-trivial

A number of different techniques can be used to allocate integrity level requirements ranging from the Consequence/Autonomy models of DO-178B and MIL-STD-882  to the Risk Matrices of IEC 61508 (1). Because of these differences SIL allocation cannot be said to be a consistent and therefore precisely defined activity this makes refutation of the theory difficult as a failure could be argued, ad hoc, as being due to the incorrect allocation of SILs rather than SILs themselves.

#### SIL Activities are inconsistent from standard to standard

The many SIL based standards vary widely in the methods invoked and the degree of tailoring that a project can apply. DO-178B defines a basic development process but focuses upon software product testing and inspection to assure safety. Other standards such as DEF STAN 00-55 focus on the definition of safety requirements that are acquitted through evidence. Some standards, such as DEF AUST 5679, emphasise the use of formal methods to achieve the highest integrity levels while others, such as IEC 61508, invoke a broad range of techniques to deliver a safety function at a required integrity level. There is as a result no single consistent and therefore wide ranging, ‘theory of SILs’, but each is specific to the project and company ‘instance’.

#### SIL activities are applied inconsistently

The majority of SIL standards allow a degree of tailoring of process to the specific project or company. While this is understandable given the range of projects and industry contexts it results in an inherently inconsistent application of processes across projects. As an example from aviation, within that industries software community there has been a vigorous debate over the application of various methods of achieving the Modified Condition/Decision Coverage criteria of DO-178B (Chilenski 2001). Because of this variability of application it is impossible to say with precision that a specific standard has been fully applied. The lack of precision then makes it difficult to argue that should an accident occur that the standard failed because it could always be argued after the fact that it was a fault of application rather than an inherent fault in the process standard that caused the failure. This is what Popper calls a conventionalist twist, because it can be used to explain away inconvenient results. This problem of application is further exacerbated by the standardisation bodies expressing their requirements in terms of recommendations (IEC 61508) or guidance (DO-178) rather than requirements and thereby allowing process variance without either justification or demonstration of equivalence.

#### SIL activities are ambiguous as to outcome

While the SIL standards are intended to deliver both intermediate and final products with low defect rates the logical argument as to how each process achieves or contributes to such a process is not so clear. The problem becomes worse as the process moves away from proximal activities that directly impact the final delivered product and towards the distal activities of managing the process. For example DO-178 Table A-1 requires the preparation of a plan for the software aspects of certification. While planning a process is certainly a ‘good thing’, the problem is that it is difficult to link the quality of overall planning to a specific and hazardous fault in a product. All that can be said about a plan is that it represents a planning activity and ensures that, if adhered to, subsequent efforts are carried out in a planned way and are auditable against the plan. Having developed a software product to the SIL requirements we then find that the end product behaves much like software that has not been developed to such a standard. In essence SIL’s make no risky predictions given as noted above that the purported reliability of the software is not empirically testable. Even should a latent software fault exist as long as the correct set of circumstances never arise in practice the software will operate safely.

### Conclusions

Given the problems identified above we must conclude that, however much SIL’s have become the accepted wisdom, they do not satisfy the requirements of a scientific theory. They may have a seductive simplicity but they are, it seems, closer to astrology than to science or engineering. Unfortunately while the software community continues to cling to such concepts it stifles serious investigation into the real question of what constitutes safe software.

### References

Chilenski, J. J. (2001), An Investigation of Three Forms of the Modified Condition Decision Coverage (MCDC) Criterion, FAA Tech Center Report DOT/FAA/AR-01/18.

Fowler, D., Application of IEC 61508 to Air Traffic Management and Similar Complex Critical Systems – Methods and Mythology, in Lessons in System Safety: Proceedings of the Eighth Safety-Critical Systems Symposium, Anderson, T., Redmill, F. (ed.s), pp 226-245, Southampton, UK, Springer Verlag.

Littlewood, B. & Strigini, L. (1993), Validation of Ultra-High Dependability for Software-based Systems. Comm. of the ACM, 36(11):69–80.

McDermid, J. A, Pumfrey, D.J Software Safety: Why is there no Consensus? Proceedings of the International System Safety Conference (ISSC) 2001, Huntsville,System Safety Society, 2001.

Popper, K.R. , Conjectures and Refutations, Third Ed. Routledge Pub., 1968.

Redmill, F., Safety Integrity Levels – Theory and Problems, Lessons in Systems Safety, in Lessons in System Safety: Proceedings of the Eighth Safety-Critical Systems Symposium, Anderson, T., Redmill, F. (ed.s), pp 1-20, Southampton, UK, Springer Verlag

##### Notes

1. There have been a multitude of qualitative and quantitative methods proposed for SIL assignment, so many that it sometimes seems that safety professionals take a perverse delight in propagating new techniques. Some of the more common include (with source):

• Consequence (control loss) (MISRA),
• Software authority (MIL-STD-882),
• Consequence (loss severity) (DO 178),
• Quantitative risk method (IEC 61508),
• Risk graph and calibrated risk graph (IEC 61508/IEC 61511),
• Hazardous event severity matrix (IEC 61508),
• Hybrid consequence and risk matrix (DEF STAN 00-56),
• Semi-quantitative method (IEC 61511),
• Safety layer matrix method (IEC 61511), and
• Layer of protection analysis (IEC 61511).