14% of patients improve from therapy?

The first point is about the statement that 14% of patients improve from therapy. I assume that Solem refers to an estimation of the NNT (numbers needed to treat) of seven. Unfortunately, Solem’s interpretation of the NNT is simply wrong. An NNT of seven does not mean that 14% of patients improved after therapy. The NNT is based on the risk difference, which indicates the difference between the proportion of people improving in therapy minus the proportion of people in the control condition. If that difference is 14%, then the NNT is indeed seven (the NNT is one divided by the risk difference). But that can mean very different things; if nobody in the control group improves, then indeed only 14% of the people in treatment improve, but if 80% of the people in the control group improve, then 94% of the people in treatment improve. In both cases the NNT is seven, but from a clinical perspective it means something completely different. For depression, the response rate in control groups has been estimated to be 16%–17%, compared to 42% in therapy, which means a risk difference of ~25% and an NNT of four.

Solems comments on meta-analyses

The second point I want to make is that Solem does not have a strong point in his comments on meta-analyses. He would have to go back decades in time to find critics of the methods of meta-analyses (Eysenck in the 1970s, who called meta-analyses ‘mega-silliness’ because he could not accept that people did not agree with his simplistic methods to conclude that therapies were not effective; and Samuel Shapiro, who talked about ‘shmeta-analysis’ but only in the context of meta-analyses of observational studies). Of course this does not mean that meta-analyses can be problematic. He gives a good example by referring to a meta-analysis saying that the effects of CBT have declined over time. This meta-analysis was methodologically very flawed, but that was not even seen by the reviewers and editor of the journal. Unfortunately, that is the reality of research. However, the same is true for randomised trials and any other research conducted. It is important that readers remain critical of what they read, whether it be meta-analyses, randomised trials or the research that Solem proposes.

What has happened since our publication of 2014

The third point: Solem wonders what has happened since our publication of 2014 that seemingly changed the effects of therapies. Our 2014 paper was a small meta-analysis comparing therapies with pill placebo. That is a highly specific subject and can only be examined in trials in which pharmacotherapy is also a treatment option. The difference between therapy and pill placebo has not changed since 2014. However, this difference provides a very narrow look at the effects of therapy, because comparisons with other control groups have been made in hundreds of trials, while only 10 trials compared therapy with pill placebo. Based on the broader range of studies, one can say that an NNT of four is more appropriate.

My fourth and final point is that I am sorry that Solem insists on looking at differences between trials while a huge body of research shows that these differences are not related to the outcome. This is a rigid and conservative position, suggesting that we do not know enough about this specific therapy in this context in this population by this therapist. It ignores the fact that we have a wealth of knowledge about which therapies work, for whom, and in which context, and that this knowledge is ready to be used in innovating practice to help patients improve.