Or how not to do risk assessments
To set the scene. Here in Australia there is a a group called the Australian Technical Advisory Group (ATAGI) that provides advice to the federal government’s health Minister on the safety and use of vaccines. Part of their job during the pandemic has been advising on how the various vaccines should be rolled out, to what age groups and so on. ATAGI originally recommended that for 50 years and under that Pfizer was preferred against Astra Zeneca due to clotting risks in younger individuals and that, “given there is currently no or limited community transmission in Australia” we could afford to wait for Pfizer to turn up. On the 17th June 2021 they updated their advice and extended their recommendation to the 50 to 69 age cohort due to several incidents of thrombosis and thrombocytopenia syndrome, TTS for short. Again they again reaffirmed that it was based on no or limited transmission in Australia.
So what is wrong with this advice? Well quite a lot, one of the principle rules of risk assessment is that you should always be careful to to compare risks that are equivalent, technically this is called mensurability. So can we compare the risk of getting Covid 19 with that of having a blood clot as a result go a vaccine? Well, let’s start with another example drawn from earlier in this pandemic. Early on the lockdown sceptics were pointing out that your risk of drowning in a pool, in California, was much higher than that of dying from Covid 19 so why to worry? If you feel this is intuitively wrong, in fact wronger than wrong, then yes you’re on the right track. The two probabilities that underpin these risks are in fact radically different. In the first case if I drown in my pool it’s not going to have any affect on the probability of my neighbour drowning, we can say these events are independent and so are their probabilities. But, on the other hand if I contract Covid 19 you’ll find that the probability of my neighbour also getting Covid 19 is actually dependent on the probability that others (including me) are infected. In the first scenario we can truck along with a constant rate (and risk) of drowning events each unaffected by each other, but in our second scenario the events can affect each other’s probabilities and the risk can suddenly blow up. Thus these two risks are fundamentally immensurable because of the underlying difference between dependent and independent probabilities. In the case of the ATAGI estimate they compared exactly these two different types of probability, and risk, and made exactly the same error. That is the risk of clotting is clearly independent while that of dying from Covid 19 is very much dependent, just like our prior example these are not the same sort of risks and just like our prior example you cannot compare them.
So could ATAGI retrieve their risk assessment? In order to do that they would have to meaningfully assess an individuals risk while they wait, patiently, for Pfizer. We can in theory apply what’s called Pascal’s parallel worlds theory to the problem. That is we imagine all the different parallel worlds (or scenarios) in which an individual might be infected and the consequences as well as those in which they are not infected and based on the relative proportions of outcomes we can assign probabilities. This all sounds fine except for the small problem, called the ergodic fallacy, that we don’t actually live in these parallel worlds, you and I and everyone else live our lives on a single timeline where getting Covid can kill us and it doesn’t matter that a thousand other ‘us’ in other timelines have not. Where there is risk of ruin these sort of exercises can and do deliver alarming underestimates of risk. We might for example consider the young woman who recently died of Covid 19 in Sydney and whether, if knowing what lay ahead, she would have preferred to have take Astra Zeneca (2). I think perhaps so. So no, from an individual’s perspective such prognostication about the future cannot rescue ATAGI’s assessment. As N. Taleb points out when there’s risk of ruin in the house we shouldn’t try to estimate the probability of such uncertain and unknowable events we should just focus on eliminating the risk.
Adding further to the problem when ATAGI actually got around to performing a cost versus benefit study (ATAGI advice 21 June 2021) they compared the rate of TTS against the rate of hospitalisations and deaths for Covid 19. But this makes no sense as a TTS is an event that can have a range of outcomes, so their risk assessment is misleading in that it’s comparing a naive event rate on the one hand with the various rates of loss outcomes on the other. Nowhere in their study do they identify exact what are the probabilities of death or ICU stays if a TTS does occur, which on the face of it skews the risk assessment and make TTS appear to be a bigger risk than it is (2). Further compounding the confusion, ATAGI then go on to point out that their probability estimates for TTS are uncertain as they’re based on a small number of people vaccinated with Astra Zeneca under 50 in Australia (3). Well OK, but in that case would it be too much to ask for a confidence interval on their estimate? Or to explicitly compare it to international data?
The result of ATAGI’s sloppy thinking and analysis is that it encouraged the ‘waiting for Pfizer’ mindset even in those who were eligible for Astra Zeneca according to ATAGI’s own recommendations. And as we all now know it was one unvaccinated limousine driver, who’d declined Astra Zeneca because he was waiting for his Pfizer, that sparked off the current Delta outbreak in Australia.
Here we are, thanks in no small measure to ATAGI.
1. Also in the age group 18-49 their estimating zero deaths due to Covid 19 are zero for a moderate outbreak. Now that can’t be right as we’ve had deaths already in this age cohort. It’s rare, perhaps not so rare with Delta, but it does happen. One meta-study estimates a median Infection Fatality Rate of 0.002% at age 10 and 0.01% at age 25. How they arrived at zero is a mystery, risk doesn’t just magically go to zero even for a moderate outbreak. So again it appears that AGATI are underplaying the risks of Covid 19 for the younger cohorts.
2. I went and pulled the prior ATAGI advice and the risk of death is 3% if you get a clot. So the risk of death due to TTS risk is 0.081 per 100,000 in the age group 50-59. Compared to their risk of death in a moderate outbreak of 0.1 per 100,000. I mean really ATAGI?
3. Where there is uncertainty in an estimator we should do more than just point at it and say look, it’s uncertain. And there are well tried statistical techniques that can do just that.