Study of the heterogeneity and inconsistency in networks of multiple interventions

Abstract

Aims: Network meta-analysis requires the agreement between direct and indirect evidence, defined as consistency. The objective is to evaluate empirically the prevalence of inconsistency in full networks using various approaches for the assessment of consistency and to explore factors that might control its statistical detection.Methods: I evaluated inconsistency in 40 published networks with dichotomous data published in PubMed from March 1997 until February 2011 and involved at least four treatments and at least one closed loop. The networks included 303 loops of evidence, 362 trial designs -studies involving different sets of treatments- and 348 comparisons. I employed five approaches: 1) loop-specific (LS): I evaluated each loop in the network separately by contrasting direct and indirect estimates 2) separating one design from the rest: I evaluated the agreement between studies of a particular design and the remaining network 3) separating indirect and direct evidence: I evaluated the agreement between a particular comparison and the remaining network 4) Lu and Ades model: I jointly assessed all possible inconsistencies in the network to obtain an omnibus test 5) Design-by-Treatment interaction model (DBT): I evaluated the agreement between estimates from different designs in the network in an omnibus test. In LS and DBT approaches I used different effect measures, and various estimators and assumptions for the heterogeneity. I also carried out a simulation study to estimate the performance of the LS-test.Results: Inconsistency Results: Inconsistency was prevalent in 1) Inconsistency was prevalent in 1) Inconsistency was prevalent in 1) Inconsistency was prevalent in 1) Inconsistency was prevalent in 1) Inconsistency was prevalent in 1) Inconsistency was prevalent in 1) Inconsistency was prevalent in 1) Inconsistency was prevalent in 1) Inconsistency was prevalent in 1) between 2% and 10% of the tested loops depending on the effect measure, assumption and estimation method for heterogeneity, 2) 9% of the 9% of the 9% of the 9% of the tested designs, 3) 11% of the total comparisons, 4) tested designs, 3) 11% of the total comparisons, 4) tested designs, 3) 11% of the total comparisons, 4) tested designs, 3) 11% of the total comparisons, 4)tested designs, 3) 11% of the total comparisons, 4) tested designs, 3) 11% of the total comparisons, 4) tested designs, 3) 11% of the total comparisons, 4)tested designs, 3) 11% of the total comparisons, 4) tested designs, 3) 11% of the total comparisons, 4)tested designs, 3) 11% of the total comparisons, 4) tested designs, 3) 11% of the total comparisons, 4) tested designs, 3) 11% of the total comparisons, 4) tested designs, 3) 11% of the total comparisons, 4) tested designs, 3) 11% of the total comparisons, 4) tested designs, 3) 11% of the total comparisons, 4) tested designs, 3) 11% of the total comparisons, 4) tested designs, 3) 11% of the total comparisons, 4) tested designs, 3) 11% of the total comparisons, 4) maximum seven (18%) networks depending on the parameterisation of the multi-arm studies, and 5) between 13% and 28% of the networks depending on the effect size and estimator for heterogeneity. Important heterogeneity was associated with a small decrease in statistical inconsistency, but different effect measures had no statistically significant impact on detecting inconsistency. The simulation study showed that the LS-test has generally low power that is positively associated with sample size and frequency of the outcome and negatively associated with the presence of heterogeneity. Type I error converges to the nominal level as the number of individuals in the loop increases. Coverage is close to the nominal level in most cases.Conclusions: This study suggests that a sensitivity analysis in the assumptions and estimators of heterogeneity is needed before concluding the absence of statistical inconsistency, particularly in networks with few studies. Investigators should interpret every test result very carefully and always consider the comparability of the studies in terms of potential effect modifiers.