Category: Further Statistics

In an accompanying editorial at http://www.bmj.com/content/348/bmj.g3148, I have outlined how the network meta-analysis combines evidence from the included randomised trials of maintenance treatments for adults with asthma.

The strength (and weakness) of the network approach is that treatment effects estimated from the direct randomised comparisons within each trial are combined with indirect non-randomised comparisons of the treatment effects between the trials. Although the authors tested the consistency of the direct and indirect comparisons, the power to find inconsistency is low when there is only a small amount of direct evidence. This is pointed out by the authors and demonstrated by the wide confidence intervals in Figures 5 and 6. Note that these figures are on a log scale, so 2 on the scale represents a Rate Ratio(RR) of e², which is equivalent to RR = 7.39!

The problem is that indirect comparisons are subject to potential confounding by differences between the participants, outcome measurements and trial designs between the individual trials. This assumption is that the patients are similar enough that it would be reasonable to assume that they could been found with equal likelihood in any of the trials? The technical term for this is a “transitivity assumption”.

One way to assess whether the network meets this transitivity assumption is to look at the asthma exacerbation (attack) rates across both arms of each of the trials to see if these are broadly similar. The Loymans network is very well reported and includes full documentation of the exacerbation rates in each trial in supplementary table S1. This shows considerable discrepancy between the attack rates in the different trials.

The variation in the frequency of the asthma attacks between studies causes two problems. Firstly the results cannot readily be applied in clinical practice, as the treatments have been compared in a wide variety of different severities of asthma. Secondly, since the frequency of asthma attacks is likely to be an important effect modifier when comparing the effects of the different maintenance treatments, the indirect comparisons may be confounded by this.

One example of the possible impact of such confounding is explored further in my editorial, and may explain why combination with lower dose of inhaled corticosteroids (ICS) is ranked in the network as likely to be more effective than combination therapy with higher dose ICS. In general the trials using higher dose ICS were on people who had more severe asthma and more frequent attacks (as you might expect), so this is not a fair indirect comparison of the higher and lower dose combination therapy. There was very little direct evidence comparing the high and low dose ICS combination treatments (i.e. randomised within the same trial).

Finally, it is also not too surprising that current best practice comes out as least likely to lead to withdrawal of treatment. The authors suggest that this outcome gives an indication of the safety of the asthma treatments. However, it may simply be that all the other treatments required some change for the participants. When treatments are changed we would expect some people to dislike the new inhaler for one reason or another. Again this is not really a fair comparison.

It is also a shame that the safety of the different treatments is discussed in the text without much mention of serious adverse events, which are in relegated to a supplementary table!

For a more in-depth discussion of the methodology of the network approach, you might find the following helpful: Cipriani A, Higgins JPT, Geddes JR, Salanti G. Conceptual and technical challenges in network meta-analysis.Ann Intern Med 2013; 159: 130-7.