The Trouble with your analysis, Dr Labos. Point by point.

This past Sunday, the Montreal Gazette published an opinion piece by a local doctor, Christopher Labos. The main thrust was a rehash of arguments made by a small group of self-proclaimed skeptics who are intent on explaining away the evidence of acupuncture’s effectiveness, such as the Cochrane Systematic Reviews below.

Below, I respond to Dr Labos, point by point.

“Acupuncture has been particularly popular recently and its interest was bolstered by a recent Cochrane review suggesting it might help with tension type headaches.”

Actually, there have been three recent Cochrane Systematic Reviews that support acupuncture’s efficacy. One is the Acupuncture for Tension Type Headache review you mention, which was published in April. Acupuncture for neck disorders was published in May and the other was Acupuncture for Preventing Migraine Attacks published in June. All were rigorous and overwhelmingly positive.

“However, this review had a number of statistical problems with it.”

Interesting, please share. It’s always helpful when researchers can offer their critical review skills.

“First, the data listed in the meta-analysis didn’t match the data presented in the published studies.”

. . . Errrr, is that the end of your first point? That’s fine if you’ve found a typo in the data table (out of curiosity, where was it? Do you have a page or table number?). But in order for this ‘statistical problem’ to be relevant to your argument against acupuncture, I’m curious to know, does correcting the error actually change the outcome? I’m thinking that if it did, you would have mentioned it.

“Second, the review used a fixed-effects rather than a random-effects model. This is a subtle statistical point, but it means that a less rigorous statistical approach was used.”

Huh, I’ve never heard of fixed-effects referred to as being “less rigorous” before, it’s really just a question of using the appropriate model for the particular review. My understanding is that this model is used when heterogeneity is assumed to be low. And indeed, the authors of the review write: “For the comparison versus sham we calculated pooled fixed-effect estimates, their 95% confidence intervals, the Chi-squared test for heterogeneity and the I-squared statistic. If the P value of the Chi-squared test for heterogeneity was < 0.2 and/or the I-squared > 40% we reported random-effects estimates in addition.” 1

So basically, they started with the assumption that there was low heterogeneity and then they tested this assumption. If this assumption was incorrect, they also reported Random Effects. The application of different statistical approaches is somewhat subjective and there’s no indication that their approach lacked rigour. And of course, there’s the obvious question that I’ll repeat yet again: if they used a random effects model, would that have changed the conclusion of the results? If not, then it isn’t really a problem for acupuncture, is it?

“My big problem with the paper, though, was that the reported benefit was largely driven by one trial. This 2007 study examined traditional acupuncture against “sham” acupuncture where needles were inserted in random locations.

There was no significant difference between the two groups in terms of the trial’s primary endpoint. Roughly 33 per cent and 27 per cent of patients responded, respectively. However, the study’s authors then proceeded to redefine what constituted success, and the success rates jumped to 66 per cent and 55 per cent. I become worried when endpoints change and negative studies become positive.”

Well gosh, I’m sorry if this study worried you. But in a systematic review, it’s not really as important how the study authors defined success so much as how the reviewers define success. For the primary endpoint, this study defined “a responder” as someone who had a 50% reduction in headaches. But, they would be defined as a “non-responder” even if they did have a 50% reduction in headaches if they changed their medications. Using the stricter definition that turned responder into non-responders, they had non-significant results. But, they noticed positive results for their secondary endpoints, which made them wonder why the primary endpoint was negative. So they did a re-analysis using the criteria set forth by the International Headache Society, which is a >50% reduction in headaches but doesn’t include all the additional exclusions.

But, to get to the relevant bit, how did the Cochrane Reviewers define a responder? “The main efficacy outcome measure was response (at least 50% reduction of headache frequency)”2

So it really doesn’t matter if the endpoint was primary, secondary or 98th in the study, if it was original or changed 15 times. What matters is what the reported results were for the outcome of interest of the review, which is clearly the one using the definition set out by the IHS and the ones that the reviewers extracted.

“How can the response rate be so spectacularly high?”

Well there’s a question!! Mr Occam tells us that the simplest answer is usually the correct one. The evidence suggests that the intervention is effective. At least, that’s the unbiased interpretation.

“It’s worth noting a quote from another study by the same research group. In that study, on low-back pain, the authors stated: “Effectiveness of acupuncture, either verum (true) or sham, was almost twice that of conventional therapy.

In other words, no matter what was done — whether traditional acupuncture or some guy randomly sticking needles into your back — people felt better. In that study, the response rate was 47.6 per cent for true acupuncture, 44.2 for the sham group, and 27.4 per cent for those who had nothing done to them.

The truth is, most of the benefits of acupuncture likely stem from the placebo effect.”

Ok, let’s break this down a bit. First, you’re saying that there are benefits to acupuncture. And you’ve provided good evidence that this benefit is substantial. Second, if most of the benefits are from placebo (which is true of most pharmacological interventions) then the rest of the benefits are from the acupuncture itself. I’m glad we’ve cleared that up.

But it needs to be pointed out that ‘some guy randomly sticking needles into your back’ is not a placebo control. Indeed, the sham protocol itself, sticking needles superficially in the back on either side of the spine, is not distinct from what many acupuncturists do as part of their treatment. As this is not a placebo control, lack of significant difference between the two groups at 6 months does not tell you if acupuncture is better than placebo. Furthermore, if we assume a true difference in effect size between acupuncture and sham of .2 (based on the Vickers 2012 individual patient meta-analysis using data from nearly 18,000 patients)3, the study you mention was grossly under-powered to detect this difference. One simply cannot conclude that it doesn’t matter where you stick the needles based on this study because it was too small to address this issue.

Now, both acupuncture and sham acupuncture were twice as effective as guideline-based conventional treatment. So what did this treatment involve?

“Conventional therapy included 10 sessions with personal contact with a physician or physiotherapist who administered physiotherapy, exercise, and such. Physiotherapies were supported by nonsteroidal anti- inflammatory drugs or pain medication up to the maximum daily dose during the therapy period. Rescue medication was identical to that for the acupuncture groups.”4

Ok, so explain your theory to me again. Chronic low back pain is a notoriously difficult condition to treat and one that does not tend to spontaneously get better. Compared to 10 sessions with a caring physician or physiotherapist administering physio and exercise, patients who had an intervention involving very limited communication with a practitioner sticking small needles into their back that stimulate nerve endings, release endogenous opioids and alter local tissue perfusion had twice the rate in successful outcomes, both in terms of pain reduction, function and reduction in pain medication.

So how do you propose that this treatment is able to achieve such a high placebo response? The placebo response in getting 10 treatments from a physio or doc with free-flowing caring attention while popping NSAIDs is phenomenally high. Can you elaborate on your theory? You seem to be using the ‘it’s mostly placebo’ line as some sort of magic wand to sweep away results that go contrary to your personal beliefs and opinions without applying any critical thinking or evidence to support your theory. Could you please explain how an intervention can be twice as effective as 10 treatments with a doctor, exercise and medication through the power of placebo alone? And can you provide some evidence to back it up?

“Proponents of traditional acupuncture say inserting needles along meridians redirects the Qi, a life force energy that flows from organs to the skin. Having looked inside many human bodies over my career, I can tell you that no such meridians exist.”

Huh, that’s interesting. Can I just ask, when you were looking inside of these bodies, did you ever see alpha-waves in the brain? How about a QRS complex? Or a neuronal action potential? These are all realities that are pretty important to human functioning and yet impossible to see when looking inside of a body. Are you saying, that these don’t exist?

Experimental research suggests that acupuncture’s channels are functional channels through which signals (chemical, electrical, and mechanical) preferentially flow. They tend to be close to fascial planes richer in interstitial fluid and mast cells, aiding chemotaxis, as well as fine nerve endings. It seems that a channel is not so much an object (like a bone or an eyeball) as it is where you find a movement of biochemical and bioelectrical factors that aid communication to keep the body healthy.5

There is experimental evidence that this same communication network is involved in embryological development. For example, have you ever considered how in foetal development, a bud that’s growing into an arm somehow “knows” where the upper arm ends and the forearm begins? This information isn’t in the DNA, as this contains the complete set of information for all body parts and proteins. Each part of the developing embryo “knows” where it is and what proteins to make due to an information network involving biochemicals (particularly morphogens) and bioelectric signals.6 As this bio-information network has been proven to exist and yet is largely ignored by conventional medicine (with the notable exception of oncology), some circumspection is required.

“The larger problem is that it is impossible to completely tease out the placebo effect when it comes to acupuncture. It is not like trials with medications, where the placebo group is given sugar pills.”

I couldn’t agree more and thank you for providing some actually useful commentary here. Due to the nature of randomised trial design, it is simply impossible to separate treatment and placebo effects for any non-pharmacological intervention, such as surgery, physiotherapy, talking therapy or acupuncture, which is why it’s so puzzling that you so confidently attribute acupuncture’s overwhelming success in these studies to placebo when you are fully aware that you simply cannot determine that experimentally.

“A number of placebo techniques have been tried. For example, needles can be inserted into the patient randomly or only superficially. There are also special retractable needles that pop back into the shaft of the device after piercing the skin. However, patients who’ve undergone acupuncture know what the traditional procedure is supposed to feel like, and where the needles are supposed to go. At the same time, the practitioners obviously know whether they are using real needles or inserting them differently, and their behaviour often clues in the patients as well.

Not surprisingly, this has led to the finding that much of the effect of acupuncture is guided by the acupuncturist’s style and bedside manner more than anything else.”

Yep, practitioner style and bedside manner influences much of the effect of all interventions, including pharmacological interventions 7(Vase 2015). That’s a feature of treating humans with health problems, not a feature of acupuncture.

“Some will use the “what’s the harm” argument and suggest that any benefit — even that of a placebo — is worth the effort. I disagree.

For one thing, acupuncture is not entirely risk-free. Complications like infections and punctured lungs have been reported, as Edzard Ernst details in his 2011 paper in Pain.

As well, the practice uses precious health care resources we can scant afford to waste in a time of repeated budget cuts.

Finally, the medical profession decided long ago that selling placebos to patients was unethical, no matter how convenient or profitable it might be.”

No, no, no. Please listen closely and pay attention. This really is not complicated. The question is not “What’s the harm?” The question is “what is the risk to benefit ratio and how does it compare to other available treatments for the condition in question?” As you have explained, acupuncture is much more effective than available alternatives for a number of conditions and that these effects are not entirely explained by placebo effects. These available alternatives, such as NSAIDs and opioids, are extremely dangerous and contribute to healthcare costs in a disastrous way. Acupuncture, on the other hand, is exceedingly safe.8

Based on the available research, recommending against this treatment is unethical and pretty darn silly if your aim is to simultaneously help patients get better and reduce the risk of harm you expose them to in the process.

In fact, let me ask you a question. You’ve looked at a rigorous Cochrane Systematic Review of acupuncture for tension type headaches, and other than some invalid quibbles that don’t affect the conclusion, you couldn’t find anything substantively wrong with it. This review, considered to be the pinnacle of the evidence-based medicine hierarchy, found that acupuncture was both efficacious (better than sham) and effective (better than usual care) and yet in spite of this high quality evidence, you still recommend against acupuncture for this condition. So, what do you specifically recommend for tension type headache and what research evidence do you base this on? It’s all well and good to sit there saying that acupuncture offends your firmly held beliefs in spite of the clinical evidence-base, but what do you tell your patients to do instead? And based on what?

Your move, Doc.