Archives For Philosophy

The philosophical aspects of safety and risk.


On Artificial Intelligence as ethical prosthesis

Out here in the grim meat-hook present of Reaper missions and Predator drone strikes we’re already well down track to a future in which decisions as to who lives and who dies are made less and less by human beings, and more and more by automation. Although there’s been a lot of ‘sexy’ discussion recently of the possibility of purely AI decision making, the current panic misses the real issue d’jour, that is the question of how well current day hybrid human-automation systems make such decisions, and the potential for the incremental abrogation of moral authority by the human part of this cybernetic system as the automation in this synthesis becomes progressively more sophisticated and suasive.

As Dijkstra pointed out in the context of programming, one of the problems or biases humans have in thinking about automation is that because it ‘does stuff’, we find the need to imbue it with agency, and from there it’s a short step to treating the automation as a partner in decision making. From this very human misunderstanding it’s almost inevitable that the the decision maker holding such a view will feel that the responsibility for decisions are shared, and responsibility diluted, thereby opening up potential for choice shift in decision making. As the degree of sophistication of such automation increases of course this effect becomes stronger and stronger, even though ‘on paper’ we would not recognise the AI as a rational being in the Kantian sense.

Even the design of decision support system interfaces can pose tricky problems when an ethical component is present, as the dimensions of ethical problem solving (time intensiveness, consideration, uncertainty, uniqueness and reflection) directly conflict with those that make for efficient automation (brevity, formulaic, simplification, certainty and repetition). This inherent conflict thereby ensuring that the interaction of automation and human ethical decision making becomes a tangled and conflicted mess. Technologists of course look at the way in which human beings make such decisions in the real world and believe, rightly or wrongly, that automation can do better. What we should remember is that such automation is still a proxy for the designer, if the designer has no real understanding of the needs of the user in forming such ethical decisions then if if the past is any guide we are up for a future of poorly conceived decision support systems, with all the inevitable and unfortunate consequences that attend. In fact I feel confident in predicting that the designers of such systems will, once again, automate their biases about how humans and automation should interact, with unpleasant surprises for all.

In a broader sense what we’re doing with this current debate is essentially rehashing the old arguments between two world views on the proper role of automation, on the one side automation is intended to supplant those messy, unreliable humans, in the current context effecting an unintentional ethical prosthetic. On the other hand we have the view that automation can and should be used to assist and augment human capabilities, that is it should be used to support and develop peoples innate ethical sense. Unfortunately in this current debate it also looks like the prosthesis school of thought is winning out. My view is that if we continue in this approach of ‘automating out’ moral decision making we will inevitably end up with the amputation of ethical sense in the decision maker, long before killer robots stalk the battlefield, or the high street of your home town.

Toyota ECM (Image source: Barr testimony presentation)

Comparing and contrasting

In 2010 NASA was called in by the National Highway Transport Safety Administration to help in figuring out the reason for reported unintended Toyota Camry accelerations. They subsequently published a report including a dedicated software annex. What’s interesting to me is the different outcome and conclusions of the two reports regarding software.  Continue Reading…

Waaay back in 2002 Chris Holloway wrote a paper that used a fictional civil court case involving the hazardous failure of software to show that much of the expertise and received wisdom of software engineering was, using the standards of the US federal judiciary, junky and at best opinion based.

Rereading the transcripts of Phillip Koopman, and Michael Barr in the 2013 Toyota spaghetti monster case I am struck both by how little things have changed and how far actual state of the industry can be from state of the practice, let alone state of the art. Life recapitulates art I guess, though not in a good way.

The enigmatic face of HAL

The enigmatic face of HAL

When Formal Systems Kill, an interesting paper by Lee Pike and Darren Abramson looking at the automatic formal system property of computers from an ethical perspective. Of course as we all know, the 9000 series has a perfect operational record…

Easter 2014 bus-cycle accident (Image Source: James Brickwood)

The limits of rational-legal authority

One of the underlying and unquestioned aspects of modern western society is that the power of the state is derived from a rational-legal authority, that is in the Weberian sense of a purposive or instrumental rationality in pursuing some end. But what if it isn’t? What if the decisions of the state are more based on belief in how people ought to behave and how things ought to be rather than reality? What, in other words, if the lunatics really are running the asylum?

Continue Reading…

On being a professional

Currently dealing with some software types who, god bless their wooly socks, are less than enthusiastic about dealing with all this ‘paperwork’ and ‘process’, which got me to thinking about the nature of professionalism.

While the approach of ‘Bürokratie über alles’ doesn’t sit well with me I confess, on the other side of coin I see the girls just wanna have fun mentality of many software developers as symptomatic of a lack of professionalism amongst the programming class. Professionals in my book intuitively understand that the ‘job’ entails three parts the preparing, the doing and the cleaning up, in a stoichiometric ratio of 4:2:4. That’s right, any job worth doing is a basic mix of two parts fun to eight part diligence, and that’s true if you’re a carpenter or a heart surgeon.

Unfortunately the fields of computer science seems to attract what I can only call man children, those folk who like Peter Pan want to fly around never land and never grow up, which is OK if you’re coding Java beans for a funky hipster website, not so great if you’re writing an embedded program for a pacemaker, and so in response we have seem to have process*.

Now as a wise man once remarked, process really says you don’t trust your people so I draw the logical conclusion that the continuing process obsession of the software community simply reflects an endemic lack of trust, due to the aforementioned lack professionalism, in that field. In contrast I trust my heart surgeon (or my master carpenter) because she is an avowed, experienced and skillful professional not because she’s CMMI level 4 certified.

*I guess that’s also why we have the systems engineer. 🙂

…and the value of virtuous witnesses

I have to say that I’ve never been terribly impressed with ISO 61508, given it purports to be so arcane that it require a priesthood of independent safety assessors to reliably interpret and sanction its implementation. My view is if your standard is that difficult then you need to rewrite the standard.

Which is where I would have parked my unhappiness with the general 61508 concept of an ISA, until I remembered a paper written by John Downer on how the FAA regulates the aerospace sector. Within the FAA’s regulatory framework there exists an analog to the ISA role, in the form of what are called Designated Engineering Representatives or DERs. In a similar independent sign-off role to the ISAs, DERs are paid by the company they work for to carry out a certifying function on behalf of the FAA.

Continue Reading…