The Magic of Statistics in Action: Tamiflu Is Pretty Useless
I discussed the relationship between effect size, sample size and significance in my last article on methodology. Those who remember the key facts know: even the smallest effect can be made significant if it is present, provided you have the appropriate means. What all readers should always ask themselves: Is the effect worth the money? Is the size of the effect large enough for practical purposes? In addition, the question always arises with small effects: Are all studies really known and taken into account? Because clearly, with small effects, if they are summarized in meta-analyses or reviews, the absence of a few negative or less strongly positive studies makes a big difference.
How this all adds up can be seen in a recent example: the Cochrane review on the efficacy of neuraminidase inhibitors for the treatment of influenza and prevention of influenza complications, which has just been submitted in a new version by Jefferson and colleagues [1]. We recall:
In 2009, a new wave of swine flu broke out, caused by the A/H1N1 virus. I remember it well and I was in England at the time. The newspapers reported almost hysterically.
It is worth revisiting the original audio today, from a five-year distance.
One reads that a pandemic will most likely break out. In other publications, the WHO is issuing highly-official warnings of a pandemic – and is imploring governments to stockpile enough Tamiflu. This drug is even preferred over vaccination in the question-and-answer publication of the English daily newspaper “The Independent”, which I followed regularly at the time. The reason: making a vaccine is difficult and the result cannot be introduced in time for everyone. Therefore, the logical alternative was Tamiflu. This neuraminidase inhibitor from Roche, just like the competing product Relenza from GlaxoSmithKline (GSK), supposedly inhibits the spread of the virus particles. This prevents infection, thus it can also be given prophylactically. Will there be a pandemic? Yes, most likely. But that’s no problem: we have Tamiflu and Relenza in stock. Not enough, unfortunately, but still. That way we can get the worst problems under control. Wonderful, you think. Or shouldn’t we just take vitamin C? It’s cheaper, isn’t it? The clever answer from the off: “Alternative health practitioners of homeopathy, herbal remedies and nutritional medicine are recommending measures to protect against the flu. They are about as useful as a water pistol against a forest fire.” Sure. Only pharmacology, “evidence based”, with known principles of action, corroborated by experiment, can help here.
Armed with the cover of such publicity, under pressure from the WHO and the public, our governments set about investing an estimated CHF 7.6 billion in stockpiling Tamiflu. How much money went over the counter at GSK for Relenza is not known to my knowledge. To cut a long story short for impatient readers: the money was thrown out of the window. Firstly, the A/H1N1 flu did not turn into an epidemic, at least not one that would have been worse than other flu waves. Secondly, Tamiflu has only a very limited effect against flu once you have it and practically no preventive effect. And thirdly, the effect is unlikely to result from the claimed mechanism.
This is made clear by the review by Jefferson and colleagues [1]. Peter Gøtzsche, the head of the Nordic Cochrane Centre in Copenhagen, even says in this context: “Roche has committed what looks to me like the biggest theft in history, but no one has yet dragged the company to court” [2, p. 28].
Strong words. While I disagree with Gøtzsche – the greatest robbery in modern history probably was Henry VIII’s expropriation of the monasteries in England, or perhaps the robbery of the gold treasures of the indigenous peoples of the Americas – let’s not get into these detailed skirmishes. The fact is: in a mirror battle, Roche and GSK seem to have fooled us by initially publishing only those studies that supported the thesis of efficacy. Even the first analysis by the group of authors in 2009 raised doubts [3]. The effects were significant. Symptom reduction by about one day in experimentally laboratory-induced infections. That was not much, but it was still significant. Even then, it was obvious that the claim that neuraminidase inhibitors could prevent influenza altogether was false. Then suddenly the argument appeared in the debate: but you could prevent complications, hospitalizations, pneumonia and the like. That’s something (although with 7 or so billion francs you could treat whole armies with pneumonia in hospital).
But now all that gold plating is melting away into cheap knock-offs in the hands of the Cochrane reviewers. The new review has now included all the tangible studies, including those that Roche has long withheld and that have not been published for a long time, including those that were not available through the regulatory agencies. This is thanks to Peter Doshi and some people in the authors’ group who tirelessly put pressure on Roche via the British Medical Journal and the public, so that Roche finally made all the documents available. The authors of the review then had to dig through thousands and thousands of pages of poorly edited reports, as they themselves say. It is lost in the brittle text of the official Cochrane publication, but those who have ears hear very clearly: some reports were bad, the studies were full of errors, protocols were subsequently changed, target criteria were changed during ongoing studies, evaluations were adjusted, outcome variables were redefined. The whole pandemonium of methodological errors is encountered on the first 20 pages of the description of the included studies.
This went so far that even a whole series of the released studies were not usable at all because the data analysis was too fragmentary. And in the end, a downright embarrassing, if not tragic, result emerges: It is true that, seen across all studies, neuraminidase inhibitors, Tamiflu and Relenza, have a statistically significant effect. But what does that mean in concrete terms? The result comes from pooling data from 46 studies, 20 of which used oseltamivir (Tamiflu; 9623 patients) and 26 of which used Zanamivir (Relenza; 14,628 patients). The reduction of the time with symptoms was 16.8 hours: instead of 7 days, people were only sick for 6.3 days. With Zanavir it was similar: reduction of time with symptoms was 0.6 days, thus resulting in a reduction of the mean symptom duration from 6.6 to 6 days. Hospitalizations and prophylaxis, which were advertised? No effect. Complications? No effect. Pneumonia? Only insofar as self-reported symptoms were evaluated, not when radiologically, i.e. hard-examined pneumonia was evaluated. The effect was small: you have to treat 100 patients to prevent pneumonia. Only Zanavir reduces the risk of bronchitis. 56 people have to be treated before one can experience this benefit. Oseltamivir does not show this benefit, and neither drug lowers the risk of getting otitis media or sinusitis, in children or adults.
On the other hand, there are the side effects: Tamiflu increases the risk of nausea (one in 28 treated experience this) and vomiting (one in 22 treated). Prophylactic effects are present, but small. The authors conclude:
“Oseltamivir and Zanamivir have small, non-specific effects on reducing the time to alleviation of influenza symptoms in adults, but not in asthmatic children. Using either drug as prophylaxis reduces the risk of developing symptomatic influenza. Treatment trials with oseltamivir or Zanamivir do not settle whether the complications of influenza (such as pneumonia) are reduced, because of a lack of diagnostic definitions. The use of Oseltamivir increases the risk of adverse effects, such as nausea, vomiting, psychiatric effects and renal events in adults and vomiting in children. The lower bioavailability may explain the lower toxicity of Zanamivir compared to oseltamivir. The balance between benefits and harms should be considered when making decisions about use of both NIs for either the prophylaxis or treatment of influenza. The influenza virus-specific mechanism of action proposed by the producers does not fit the clinical evidence.” [1, p.3]
In plain English: it works so-so, but definitely not as one would think based on theory – and certainly not as strongly as claimed. And important questions, such as whether pneumonia can be prevented as a complication, are still open. In view of the danger, for instance also the possible problems of psychiatric diseases or kidney diseases, and the small effect sizes, one should think twice about whether public institutions should finance these substances. What was that again about vitamin C and the recommendations of complementary medicine practitioners? Water pistols against forest fires? That does indeed seem to apply to Tamiflu and co. and the authors of the review then also ponder aloud whether old anti-inflammatories like good old aspirin might not be better in the end.
By the way, the studies analysed here are only part of the existing ones. Because the data for many studies were not reported in sufficient detail, they could not be analysed at all. A total of 121 studies on Oseltamivir were available, 83 of which were provided by Roche, so the 20 Oseltamivir studies represent only a fraction of the data. I wonder how much all this cost? The reports do not give any information about that. If we calculate an average of 2 million euros per study, which is probably too cheap, then we end up with more than 240 million euros for Tamiflu alone, probably more. But still far below the more than 7 billion Swiss francs that Roche has collected with Tamiflu.
What do we learn from this debacle, because there is no other way to describe it?
Effect sizes are among the most important variables to look at, not significance.
When it comes to drug trials and other products where money or other interests are involved, it is reasonable to assume that important information is being withheld by vested interests. It pays to be sceptical and conservative until proven otherwise.
I have made it a habit, especially when the mainstream media hype or hype the new solution, or the announcement of a new problem, to first assume the opposite on a trial basis until I could convince myself through data that the claim is true.
Any effect must be seen in relation to the cost. Costs include not only monetary ones, but also side effects and other problems. Note that in the example here, the effect sizes of the side effects are much larger than the effect sizes of the benefits.
With enough money, any effect can be “scientifically” proven. This scientific prostitution will only stop when we move away from significance and look at effect sizes.
Scientifically, we seem to be reaching the limit of the existing system of regulatory research with this example. The authors of the review keep saying that the current system of scientific assessment is useless. It allows you to publish only the data that suits you and hide the rest. Even study registers, which now exist, are of only limited help. Because the data are the property of the company that generated them. And the ethical standard that obliges a scientist to publish his data is not enforceable. At most, public pressure, as in this case, can lead to a company releasing unpublished data. Only if it were to become a self-evident standard that all data be made publicly available, via platforms, would companies or individuals who do not do this make themselves untrustworthy. But who could be the guardian and caretaker of the data? Who would fund the platforms? Who would enforce the standard? When one thinks at length about all possible solutions, it seems to me that one quickly comes back again and again to the paradox that ethical and moral behaviour cannot be enforced by rules.
Literature
- Jefferson, T., Jones, M. A., Doshi, P., Del Mar, C. B., Hama, R., Thompson, M. J., et al. (2014). Neuraminidase inhibitors for preventing and treating influenza in healthy adults and children. Cochrane Database of Systematic Reviews, CD008965(April).
- Gøtzsche, P. C. (2013). Deadly Medicines and Organised Crime: How Big Pharma Has Corrupted Health Care. London: Radcliff.
- Jefferson, T., Jones, M., Doshi, P., & Del Mar, C. (2009). Neuraminidase inhibitors for preventing and treating influenza in healthy adults: systematic review and meta-analysis. British Medical Journal, 339, b5106.