And not quite as simple as you think…
The testimony of Michael Barr, in the recent Oklahoma Toyota court case highlighted problems with the design of Toyota’s watchdog timer for their Camry ETCS-i throttle control system, amongst other things, which got me thinking about the pervasive role that watchdogs play in safety critical systems. The great strength of watchdogs is of course that they provide a safety mechanism which resides outside the state machine, which gives them fundamental design independence from what’s going on inside. By their nature they’re also simple and small scale beasts, thereby satisfying the economy of mechanism principle.
Watchdogs can, at least in principle, provide us with a highly assured defence against undetected design faults or circumstances that we have not anticipated, in other words they’re a good defence against ontological hazards. I’d stress though that WDT won’t catch all errors, but they’re very good for certain lockup states, or combinations thereof, and can form part of a broader never give up strategy. Watchdogs can also provide protection in concurrent systems via the an associated watchdog timer task (1). Although you do need to be very careful about the trade off between capability and robustness/economy.
Despite their abilities watchdogs don’t in my opinion receive the attention they deserve, perhaps because many designers see them as an admission that their code could be incorrect. And a poorly implemented watchdog is worse than useless as Toyota found out. So the following questions are intended to be a checklist of issues to consider when designing your watchdog, and no it’s never quite as simple as you might imagine. Herewith…
- What’s the maximum time to respond?
- How does the maximum time relate to the time the system can tolerate before it’s damaged?
- What’s the minimum time needed to prevent false triggering?
- If the watchdog is an internal on chip device has its control registers been protected?
- Have you considered software timing variability due to interrupts, context switching and caching?
- if an internal watchdog have you considered the scenario of a software bug resetting the timer?
- Do you use a single bit or an ISR to reset the timer, why?
- If the timer requires two back to back writes to reset have you considered an interrupt occurring between them?
- If you’re resetting the timer in multiple threads or processes how does this correlate to the timer interval?
- If an MMU is present have you masked all access to the watchdog, other than by a designated watchdog task?
- if multiple tasks can access the watchdog timer how do you assure a code error will not?
- How do other tasks update a watchdog task with their status? On start? Close? Each time through a loop?
- Does the timer initiate a full system reset or a Non-Maskable Interrupt (NMI)?
- If you’re using an NMI, justify why you think you can trust the CPU after a watchdog has fired?
- Have you reset all peripherals? If not what are the potential consequences?
- Are the peripherals resets software or hardware, if software why would you trust them after the watchdog has fired?
- How trusted is the clock used by the watchdog?
- What sort of system growth do you need to allow for?
- What margin do you need to allow for failure of the timer itself due to an SEU upset?
- Do you need to be able to disable the timer for test, diagnostics or software updates?
- Do you need to disable the timer operationally? If it fails?
- how reliable is the power supply to the timer, what happens if the supply is disabled?
- What happens when the power cycles?
- Have you stored the fact that the timer has triggered (and the number of times)?
- Have you used a saturating counter?
- Is timer history stored in non-volatile (and always powered) memory?
- Have you considered the timer as a SPOF?
- What is the disabling mechanism, stored or a hardware discrete command? Why?
- For stored commands under what scenarios the timer should resume? After test? Comms loss? If a SW load got bad?
- For discrete commands what’s the mechanism and why do you trust it?
- What’s the default timer state (enabled or disabled)?
- If it’s default enabled have you considered the tradeoff between simplicity and autonomy vs recovery difficulty?
- If it’s default disabled have you considered how to reset the timer and any difficulties?
- Is there a jumper used to enable or disable the watchdog, is it left in?
- If a jumper is used does it allow the watchdog to run in isolation during testing?
- When is the jumper removed and how do you know?
1. To which other tasks write their status and which then resets the hardware timer based upon the status of these tasks.