When All Else Fails


Re-imagining the role of operator in highly automated robust systems

In a previous post I discussed why in HOT systems the operator will be asked to intervene in situations that have not been planned for by the designer and whose uncertainty inherently demand the operator to exercise high level cognitive skills.

Therefore instead of simply supporting skill and rule based operator behavior (Rasmussen 1986) a HOT system also needs to provide decision and problem solving support for such ad hoc interactions.

To me this is simply an extension of Kopetz’s maxim that any fault tolerant system should have a ‘never give up’ strategy that is deployed when the fault hypothesis is violated. In this case the system fails back on to the operator and the system HMI needs to effectively support the operator in diagnosing and then executing control.

Research into naturalistic decision making has shown that in circumstances of uncertainty expert decision makers employ two distinct stages of cognitive processing, the first can be viewed as being recognitional primed decision making in nature and the second (triggered by new or uncertain circumstances) is a metacognitive (2) explanatory driven process of reviewing and critiquing the evolving model/understanding (Cohen et al. 1996).

For example in explaining suspect air data a highly experienced aircrew would create a narrative ‘story’ or hypothesis to explain the situation, integrate new information as it came available and critically review and amend their story in the face of conflicting evidence, up-to and including thinking about alternative explanations. Finally experienced crews will know when to think critically and when to act and be able to perform all of this under time compressed and high stress conditions, a challenging task.

Cohen (1996) suggests that in order to support this type of cognitive process the system and its interface should provide:

  1. support for causal understanding of events,
  2. supporting identifying and explaining conflicting evidence,
  3. facilitation for the generating and evaluation of alternative stories, and
  4. support for managing time and attention.

Ultimately if a system and HMI is built around rule based responses to a set of pre-defined events is simply not designed to provide this sort of cognitive support. Worse yet the system does not provide defences against the weaknesses of these types of cognitive strategies that can lead to knowledge based errors, such as availability or confirmatory biases. Importantly such errors are especially prevalent amongst experts (such as aircrew) who inherently rely on recognition and explanatory heuristics to a great degree.

Unfortunately because supporting knowledge based  problem solving is difficult from a design perspective and because the likelihood of error in such circumstances is assessed as being very likely, an equally traditional response is to say that we will deal with the risk by preventing it from occurring.

Because mistakes at the knowledge-based level are practically inevitable and difficult to recover, instead of trying to develop related error management strategies, the principle in aviation is simply to prevent crews from getting into such situations. The whole aviation system has been built accordingly.

Airbus – Human Performance & Error Management

Such an uncertainty avoidance approach presumes that preventing operators from getting into such situations is actually possible to achieve. But of course we know that the fundamental challenge of operating a HOT system is that when they fail it is because of unanticipated sequences of events, design errors or extreme environmental inputs. The very unexpected nature of such events inherently requires a knowledge based approach to problem solving.

As a counter to such a uncertainty avoidance strategy we might point to such accidents as Sioux City (UA 232) and Bagdhad (DHL OO-DLL) as illustrating where a robust system was overwhelmed and in which the operators successfully ‘learned as they went’ how to control the resultant ad hoc system. Similarly accidents such as AeroPeru 603, BirgenAir 301 illustrate where the crew were overwhelmed by the cognitive demands of an unprecedented situation.

This avoidance approach also contradicts the overall conclusions of current work on decision support systems, that aiding the decision maker is indeed possible. See for example the USN’s Tactical Decision Making Under Stress (TADMUS) Decision Support System (Morrison 1997).

Another consequence of an uncertainty avoidance approach is that operator training becomes rule (procedure compliance) and skill (performance) based. But as Cohen et al also point out operator training that is focused upon rules, performance and perfection inherently inhibits deeper comprehension and may also increase the likelihood of error in uncertain circumstances.

My conclusion is that we need to critically think both how the current system HMI are designed and how training is delivered to ensure effective operator decision making in circumstances of uncertainty and stress that are a fundamental aspect of of operating HOT systems.


Cohen, M.S., Freeman, J.T., & Wolf, Steve. Meta-Recognition in Time Stressed Decision Making: Recognizing, Critiquing, and Correcting. Human Factors, 38 (2), pp. 206-219. 1996.

Morrison, J.G., Tactical Decision Making Under Stress (TADMUS) – Science Brief, USN Space and Naval Warfare Centre, San Diego, 1997.

Rasmussen, J., Information processing and human-machine interaction: An approach to cognitive engineering. Wiley, 1986.

Rassmussen, J., Skills, Rules and Knowledge: Signals, Signs and Symbols, and Other Distinctions in Human Performance Models, IEEE Trans. on Systems, Man and Cybernetics, Vol.smc-13, No. 3, May 1983.


1. Metacognition is the individuals knowledge of the states and processes of their own mind and/or their ability to control or modify these states and processes.

One response to When All Else Fails

    Richard Airbus 20/06/2011 at 1:28 pm

    A fantastic summary. Thank you.