How to Summarise the Evidence Base

Earlier this week, a survey of the content of local acupuncturists websites performed by Daniel Ryan, a computer developer, was published in the New Zealand Medical Journal. The article was full of inaccuracies of all shapes and sizes and its successful journey through the peer-review process left many scratching their heads.

As one small example, Ryan writes ‘The UK’s National Institute for Health Care Excellence (NICE) no longer recommends using acupuncture for the treatment of any health conditions.” In reality (where the rest of us live), a number of NICE Guidelines do indeed recommend acupuncture (for instance, this one on the management of tension-type headache).

But what’s more worrying is his response when the errors in his publication were pointed out: “Of course they would say there were mistakes. I’ve backed up all my statements . . .” In the above example, he backed up his statement about what all NICE guidelines recommend by citing one single guideline that does not. Even if you don’t have a research, medical or science background (which, incidentally is a pretty good description of Daniel Ryan), obviously the reference wasn’t a valid or logical way to support the point. If the New Zealand medical journal wants to allow submissions from lay people off the street, they should consider some sort of mentoring program or additional writing support for non-researchers.

To offer a counter-opinion, the media contacted Kate Roberts of Acupuncture NZ, who holds a Masters and is currently undertaking a PhD. She provided a fully referenced and eloquent counter-point to Ryan’s piece. Poor quality and unsubstantiated claims aside, the article sparked some media attention in New Zealand where the consensus is that the lay PseudoSkeptics came out looking, if it’s possible, even sillier than usual.

Skeptics of a Feather

All that said, what’s more interesting is the response of esteemed Yale neurologist and die-hard Skeptic Steven Novella, MD to the publication and how he weighs in on how to evaluate ‘what the evidence says’ about an intervention, in this case, acupuncture. Amazingly, he seems to defend Ryan’s shoddy attempt at scholarship, demonstrating that he will pretty much support literally anything that agrees with his views on acupuncture.

In his response to the NZ Acupuncture Website-aganza, he accuses Roberts, of ‘mischaracterizing’ the medical literature when she points out that as a whole, the clinical research supports the use of acupuncture for a wide variety of conditions. In contrast to her ‘distortion’ of the literature, he offers some lessons in how to provide a thorough, unbiased and accurate review of the clinical and scientific evidence for acupuncture.

So how do you accurately characterize the scientific literature for an intervention, according to the ‘science-based’ way of doing things? On what basis can Dr Novella demonstrate that Kate Roberts is ‘simply wrong’ about what the medical literature says? Here are the steps.

1) The science-based ‘because I said so’ gambit

Novella starts out teaching us about the acupuncture literature thusly: “A science-based review of acupuncture finds that the claims made for it are highly implausible and there is no single indication for which there is robust evidence of efficacy.” Well, that’s the end of it, eh? I guess we can all go home now, a ‘science-based review of acupuncture’ says that acupuncture definitely doesn’t work.

Curious as to how I had missed such a definitive evidence-based debunking and clicking on the link to the review in question, I’m actually taken to a reprint of an opinion piece written by Steven Novella himself and co-authored by David Colquhoun. Well, that’s funny, I wonder how Dr Novella could confuse an essay he wrote about his own opinion with a piece of independent, high-quality, science-based scholarship? It’s almost as if Dr Novella thinks his op-ed piece is actually a ‘science-based review.’

So let’s take a little segue into the misleading use of the word ‘review’ to make something sound objectively scientific that really isn’t. There are systematic reviews (like the ones we will touch on in a moment) that give very little leeway into what is included in order to arrive at a roughly accurate picture of what the literature shows about a particular clinical area. Then there are non-systematic or ‘narrative reviews,’ and these too should have some modicum of rigour (a textbook might be considered an example) but do allow for cherry-picking.

Novella’s anti-acupuncture Op-Ed piece (appearing alongside a pro-acupuncture opinion piece, which for some reason Novella never mentions) can hardly even be described as a narrative review. It can only be considered a ‘scientific review’ in the same way my nephew’s Christmas list can be considered a ‘scientific review’ on his opinion of what he’d like Santa to bring him. There are no rigorous or objective criteria for what is included and no a priori methods. It’s tantamount to an essay about the author’s feelings on the subject which is fine, but ‘science-based review of acupuncture’ it certainly is not. It’s actually a little bit incredible that Novella still references this essay, where he rests his entire argument on excluding the entirety of the basic science literature into acupuncture’s mechanisms before cherry-picking a couple of studies and then concluding that acupuncture only works a little.

2) Do a ‘random search of Pubmed, use ‘what comes up’ to make generalizations about the literature

Novella’s next step in ‘fairly and rigorously’ evaluating the clinical evidence of acupuncture is to take a ‘random’ stroll down Pubmed where, to everyone’s total amazement, he finds some systematic reviews of acupuncture that are negative. (Gasp!) This is the type of exercise that non-researchers who agree with the point the author is making love but will leave actual researchers cringing. There’s probably a reason why I’ve never seen the methods section of a systematic review start out by saying ‘we decided to pop on over to Pubmed and do a little looky-loo and, by gosh, wouldn’t you know! The first bunch of studies that we happened to look at agreed with our hypothesis. <Mic drop>’

Of course, when I just did a ‘random’ search of Pubmed for systematic reviews of acupuncture, the first reviews were positive. But of course, like Dr Novella’s odd and fairly ironic exercise into lit review, this doesn’t tell us what the literature as a whole shows. And while Dr Novella indicates that he understands that this sort of ‘search’ is complete bologna, it doesn’t stop him with wasting our time with it rather than looking at actual high-quality reviews of the literature as a whole.

So, what evidence does Kate Robert and other researchers use to support the position that acupuncture is backed by clinical evidence?

According to Steven Novella, ‘acupuncture does not work.’ This is scientific fact based on an opinion piece that Novella wrote in 2013, which he amazingly agrees with, and a quick search on Pubmed. Unbiased, rigorous scientific review at its finest.

But Kate Roberts, myself and others (NICE, the NIH, the Royal College of Obstetrics and Gynaecologists, the Cleveland Clinic, Harvard . . .) claim that as a whole, the clinical literature of acupuncture supports its use for a variety of conditions.

Novella muses: “I won’t speculate about the honesty of proponents like Roberts – I don’t know what she really believes, but that does not really matter. It is possible that she simply has a very different idea of what constitutes scientific evidence than I do. I (sic) my experience when a proponent of alternative medicine claims that a treatment is evidence-based or backed by science they mean that there is some study somewhere that was positive.”

So what is Roberts basing her position on? A single study? Well, fortunately, Kate, who holds a Masters of Science and is currently undertaking a PhD, told us what she was basing the interview that Novella is referencing. She based her assertions about what the clinical evidence for acupuncture shows, not on an essay she wrote about her own opinion or a ‘random’ search of Pubmed, like Novella suggests we do, but on the most recent review of all systematic reviews for acupuncture to date.

The review, conducted this past January by Stephen Janz, who holds a Masters in Public Health and Dr John McDonald, PhD, “draws on two prior comprehensive literature reviews, one conducted for the Australian Department of Veterans’ Affairs (DVA) in 2010 and another conducted for the United States Department of Veterans Affairs (USVA) in 2013. The research identified by these reviews was pooled, then a search of further literature from 2013 to 2016 was conducted. Trials were assessed using the National Health and Medical Research Council (NHMRC) levels of evidence, with risk of bias assessed using the Cochrane GRADE system. Results have been tabulated to indicate not just the current state of the evidence, but to indicate how the quality and quantity of evidence has changed from 2005 to 2016. In this review, 122 conditions across 14 broad clinical areas were identified and, of these, only five conditions found ‘no evidence of effect’ for acupuncture. The level of evidence was found by this review to have increased for 24 conditions.” 1

Parting thoughts

Why Dr Novella would omit a recent review of all systematic reviews of acupuncture from a discussion of what the evidence shows, never mind fail to mention that this (amongst other publications) is what Mrs Roberts supported her assertions, is unclear. Perhaps it’s for a similar reason as to why his opinion piece omits all basic science literature on how acupuncture works before pretending to attempt a discussion of the evidence. If we weren’t talking about misinforming the public about a treatment that’s more effective and safer than what’s routinely offered to them, one could even try to find some humour in it.

Novella leaves us with this thought: “We are left with a situation in which acupuncture proponents claim that acupuncture works for a long list of medical indications, and claiming that this is supported by evidence. Meanwhile the actual evidence, fairly and rigorously evaluated, is negative.”

Yes, acupuncture proponents, as well as dozens of independently conducted biomedical guidelines and the top medical institutions in the world, find that the scientific evidence as a whole does indeed support its use for a growing number of conditions, based on the best available published evidence. Meanwhile, ‘the actual evidence’ (by which Novella presumably means his 2013 op-ed piece) ‘fairly and rigorously evaluated’ (by Novella, himself!) ‘is negative.’