Evidence Check Evidence Check (or; What The Papers Say)


Over the last couple of weeks, the Commons Committee on Science and Technology held a couple of their “evidence check” sessions, looking at homeopathy.  Sessions such as this are held to examine whether there is evidence to support government policy.

The oral hearings take the form of witnesses with relevant backgrounds being quizzed by committee members.  Witnesses for the first of these sessions included the legendary Ben Goldacre, Edzard Ernst from the University of Exeter, and Tracey Brown from the charity Sense About Science.  Speakers in favour of homeopathy included Paul Bennett from Boots, Peter Fisher from the Royal London Homeopathic Hospital, and Robert Wilson from the British Association of Homeopathic Manufacturers.

The big thing that came out of this hearing, from a rhetorical point of view, was the admission by Paul Bennett that Boots did not believe homeopathy to be effective – but they sell it anyway because of consumer demand.  This lead to us here at Merseyside Skeptics drafting An Open Letter to Alliance Boots, calling upon them to withdraw the product.  If you haven’t done so already, or even if you have, please check out the letter.  Digg it, tweet it, repost it, write about it.  Help up make some noise!

Ahem.

The pro-homeopathy witnesses, when challenged, mentioned a number of studies which they claimed supported the idea that homeopathy has strong effects beyond placebo.  So I thought I’d look up a few of the studies mentioned and see what those studies actually say.

The first study mentioned in the oral evidence was what Robert Wilson referred to as the “Witt Trial”.  When challenged to give a single homeopathic product for which there is good evidence to support efficacy, Wilson said:

Arnica, which is for bruising, and is extremely useful in post-operative care. There was a major trial done on arnica and, indeed, there is one that has just been published, the Witt Trial, which was done by the Charity Hospital in Berlin. It was a large trial – 3,700 patients involved – and that has shown clearly that there is a strong benefit in homeopathic use to these patients with long-term chronic conditions. One of the subjects of that trial was arnica.

For a kick off, the so-called Witt Trial is not a clinical trial.  It was a cohort study, published in December 2008, in the journal BMC Public Health.  3709 consecutive patients who were referred to a predefined selection of homeopaths for a consultation were recorded. Eight years later, they went back to the same 3709 and people. And guess what had happened?

Some of them had got better. Therefore, homeopathy works.  Makes sense to me!

On top of that, their criteria for “better” was based upon asking the patient “do you feel better?”, rather than any objective measure.  It was actually more standardised than that (please rate your condition today on a scale of 0 to 10 where 0 means you are 100% better, etc) but ultimately that’s what it came down to.

There was no control over what else the patients did in that time.  Many of them could have, and likely did, take other medicinal products during that time, be they complementary or evidence-based.  This lead to the study’s authors commenting:

As patients were allowed to use conventional therapies and other complementary therapies during the study period, the observed improvements cannot be attributed to homeopathic treatment alone.

There is no mention in the study, that I could see, of arnica, which is what Wilson claimed that the study showed effectiveness for.  There is no mention of any specific treatments actually, so although Arnica may well have been studied, I have no idea what for.

At the end of the eight year study, only 33% of the patients were still taking homeopathy.  29% had stopped taking homeopathy because they had recovered and 26% stopped taking homeopathy because they thought it wasn’t working.  Interestingly, twice as many children than adults had stopped treatment because they thought their condition had improved.  Adults were more likely to have stopped because they thought the treatment was ineffective.

The same patients were also quizzed two years after their visit to the homeopath.  The study says that there was practically no difference between the figures after two years and after eight years; and that in children, no relevant difference was found between those who stopped homeopathy and those who continued.

the differences in the outcome between those patients who stopped treatment and those who still continued were small

This seems to have been taken as a sign that the effects of homeopathy are long-lasting, rather than the arguably more sensible view that they’re non-existant.

I don’t claim to have any special expertise in reading clinical trials.  At best, I have a layman’s understanding.  Perhaps the data presented here is actually very interesting and relevant for a hundred different reasons.  But what this study certainly doesn’t do is provide strong support for the efficacy of homeopathic arnica, which is how Robert Wilson presented it in his evidence.  On the contrary, the study actually says:

The aim of this study […] was not to test the effectiveness of homeopathic drug treatment

The next study Wilson mentions was referred to as the Möllinger Trial.  This was published in the German journal “Research in Complementary Medicine”, edited by the psychologist Harald Walach.  This is some cause for concern, as Walach is also listed as co-author of the study.  That’s doesn’t have to be a reason to suspect foul play, but it is a red flag.

I couldn’t find the full text of this trial freely available online so there is a limit to what I can talk about, but I was fortunate enough to find some skeptical commentary from people who have read the full study; and of course we have the abstract.

This was described as a double-blind, randomised, placebo-controlled trial.  Which is a good start.  The participants – all healthy individuals – were divided into three treatment groups and studied over four days.

The first group was given 30C Arsenicum Album (which is the fancy-shmancy homeopathy way of saying Arsenic); the second group was given 30C Natrum muriaticum (a fancier-shmancier way to say table salt).  A third group was given placebo.

The idea is that, if homeopathy is true, because the patients are all healthy, then those in the Arsenic group should develop the same symptoms that homeopathic arsenic is used to treat.  Similarly, the patients in the salt group should develop the symptoms that homeopathic salt is used to treat.  And the people in the placebo group wouldn’t develop anything at all, or at least what they do develop would be unrelated.

There is another name for this type of study: a homeopathic proving.  Back to Robert Wilson for a moment:

a homeopathic proving is a technical term for when homeopathic medicines are assessed. It is not a way of doing a trial.

Sorry – why are you citing this paper then?  Arguably, citing provings only makes sense if the mechanism of action for homeopathy was plausible. Which is it not.  Leaving that aside for a moment, let’s look at the data.

Group Average number of Arsenic-related Symptoms Average number of Salt-related Symptoms Average number of non-specific Symptoms
Arsenic 6 0 0
Salt 0 5 0
Placebo 0 0 11

The data seems very clear cut… perhaps too clear cut.  What about the trial’s methodology?  The number of participants is very important because, as Robert Wilson said in his evidence:

Any sample of fewer than 500 is not going to be statistically relevant.

If Wilson believes this, when why is he citing the Möllinger trial?  This trial included – wait for it – no fewer than twenty-five patients.  I’ll give that to you again – twenty five patients only.  By Wilson’s own flawed view of statistics, this trial is not statistically relevant.  I say flawed view because, as Evan Harris MP pointed out Wilson during the hearing:

whether the sample size is statistically significant depends on the frequency of the outcome you are measuring

And after the hearing, Ben Goldacre made a similar comment on his blog:

if you have a pill that cures everyone from an incurable condition then 40 people is fine, hell, a dozen would do

Another red flag is that all twenty-five subjects were actually trainee homeopaths, not that that has to make a difference.  As I say, just another red flag.  Though there does seem to be an increasing number of them!

Looking at the data itself – no non-specific symptoms in the two homeopathy groups, but eleven in placebo?  Surely you would expect to get similar numbers of non-specific symptoms across all groups?  This being the very point of the control group.  Absolutely no non-specific symptoms in the homeopathy groups? That is very suspicious.

Worse than that, they don’t just say an average of zero symptoms.  They say 0±2.  How can you get a deviation of ±2 without at least one patient having a negative symptom?  I don’t understand.  Genuinely!  Have I missed some subtle feature of statistics?  Answers on a postcard, please.

Next up is the randomisation code for this study.  This is the secret code used to blind the patients and clinicians from knowing which group is which.  The code for this trial was created by Rainer Schneider, though worryingly Scheider is also the person who did the analysis of the data, even though he knows (because he wrote the code) what the codes refer to.  This represents a flaw in the blinding.  I’m not accusing Schneider of deliberately introducing bias, but it’s another red flag to add to the list.

The results of this study do seem to be too good to be true.  One co-author is the editor of the journal which published the study, another co-author generated the blinding codes then performed supposedly-blind analysis of the data.  The result set itself doesn’t make any sense, with its ±2 symptoms and zero non-specific symptoms on the homeopathic groups.  It all seems very fishy.

Even if it were all above board, the study would still need to be repeated, with tighter protocols and more participants before it can overturn the mountain of evidence which suggests homeopathy doesn’t work.  Moreover, as mentioned earlier, this was never a test of efficacy, but a homeopathic proving!

The last paper I’m going to look at was referred to by Peter Fisher, who talked about an Italian trial done comparing homeopathy for the prevention of flu against placebo for the prevention of flu. He says:

quite a lot of people who actually got the homeopathic medicine got flu-like symptoms but did not actually get flu.

This was cited as a situation where homeopathy was known to have side effects.  If homeopathy was merely a placebo, then you wouldn’t expect to find side effects, says Fisher.

Except that’s bullshit.

I wasn’t able to locate this paper.  I did find a paper called “A randomized trial in the prevention of influenza-like syndromes by homeopathic management“, which was published in France, but the lead author was Italian.  This may or may not have been the paper to which Fisher was referring, but as I can’t find even an abstract for this paper online, there really isn’t much I can say about it.

What I can do, however, is query Fisher’s assertion that placebos do not have side effects.  Placebos can have side effects as much as they can have effects – it’s all down to the patients expectations.  If they expect a tablet to make them nauseous, then there is a good chance they will feel nauseous!

After the hearing, Ben Goldacre posted a note up on his website, which I think very accurately summarises the hearing.  Over to you, Ben:

One thing that will never get old for the homeopaths, it seems, is the old practise of pulling out a single trial and saying “ah, but look, pish to your meta-analyses, here is a trial where homeopathy works”. No matter how many times you point out why this is foolish and wrong, they will always think you’re just being picky, and that is why they will always give us joy.


This article was written using the raw transcript of the evidence check session from the Parliament website. Witnesses and members have not had the opportunity to correct the record.

, , , , ,

  1. #1 by AexMagd on December 7, 2009 - 13:40

    I think a lot of these flimflam merchants (great phrase) rely on the fact that most people won’t be able to read the papers they cite. Certainly these days if you’re proposing something that science deems ridiculous you needn’t worry about what the experts say – quote some papers and throw it out to the public and before long you’ve got 60% of people who don’t believe in global warming and about a third who think evolution’s a fairy tale. It’s all about open access!

  2. #2 by Michael on December 7, 2009 - 21:00

    Isn’t that the reason for OUR existence then AexMagd?

    We need to be in that public sphere to lay down the law in 3 second soundbites just like the flimflammers(?) do in a shouty way so they have no comeback.
    I was talking to my GP tonight, he doesn’t think that any more money will be spent by the gub’mint on Homeopathy. And my GP is pretty ‘down to Earth’, however he didn’t know that Boots was selling their own homeopathetic (new word) witchcraft lotions. I went to Boots to get my prescription just after seeing my GP. The misses met me in there and we both scanned around for the homeopathetics and by George we found them. They had strengths between 6C and 30C. They were tiny little pills (the 30C belladonna were blue and about 2 mm diameter and 1 mm thick) and there were about 50 of these tiny sugar pills in a small tube with a price of £6.99(!!!!!!!!!!!!!). OK so what is the ‘mark-up’ on those then. Say packaging and manufacture works out at 10p per bottle. That is ‘laughing-all-the-way-to-the-bank’ money. Odd thing was that the 6C solution pills cost £4.99. So you pay more for less-just in keeping with the homeopathetic methodology. Why don’t we invent our own branch of medicines (with nothing in them-of course) and say that they are vetted by the infamous, non-medically trained, MSS? We could make tens of pounds in a year!

  3. #3 by Yorkshire Skeptic on December 7, 2009 - 21:55

    Cracking take down of homeopathy!
    I’d say it was impressive but the advocates for homeopathy seem to be doing everything they can to help skeptics poke holes in their already hole-filled case! 😛

  4. #4 by AexMagd on December 8, 2009 - 13:24

    Oh I definitely agree that we – and all other critically thinking people – need to speak up about this kind of thing, but it’s not really a longterm solution. Not sure what is mind you – my own utopian dream is a world in which science journalism is based on facts and not sensationalism, thus creating within the public a desire to actually go and look at the science which will, of course, be available in open access journals. There will also be jetpacks and monkey butlers

  5. #5 by Stu on December 8, 2009 - 22:04

    I agree with Michael and AexMagd (what does that mean?) . We really should be a bit more vocal about quackery!

    Why not try an open letter to the papers (Mail, Express?) that try to pass press releases off as science? At least if it’s published in a couple of them their readers might start to think a bit more . Pressure on the Press Complaints Commission in the form of open letters and block complaints by the sceptical community – a petition properly disseminated maybe – would be well supported. At least by the twenty people who’ll read this!

    And if all that fails to raise consciousness then we hit them where it really hurts! An all out T Shirt campaign!!*

    Just a thought.

    *I know – nicked from Red Dwarf.

  6. #6 by Andy Wilson on December 9, 2009 - 13:41

    A T-shirt campaign. Hmmmm.

    I think it might just work 🙂

    Andy
    10:23

  7. #7 by Andy Wilson on December 9, 2009 - 13:42

    Where’s my bloody photo gone webmaster?

  8. #8 by Mike on December 9, 2009 - 14:13

    Happy?

  9. #9 by Michael on December 9, 2009 - 15:24

    Have any of you read Professor Colquhoun’s memo sent in to the HSCoSaT? Here it is http://www.publications.parliament.uk/pa/cm200910/cmselect/cmsctech/memo/homeopathy/ucm0202.htm. Talk about scathing, it is well worth reading as he is the World No1 when it comes to evidence based medical trials (he invented the method used today).

  10. #10 by Stu on December 11, 2009 - 14:10

    Brilliant memo from Prof Colquhoun. And not one use of the word ‘allegedly’ which is how we all should be able to speak when discussing quackery.

    I’ll make good use of the story from Australia when friends/colleagues keep coming out with the mantra-like assertion that complementary therapies are harmless.

(will not be published)