SYSTEM Research note on: Assessing publication bias

CONTEXT: Systematic reviews aim to assess precision and internal validity of the current clinical evidence. The precision and internal validity of clinical evidence is limited by the risk of biases, one of which is publication bias.

Publication bias is created when trials, often with small sample size that have found negative or non-significant results are not being published and thus are not identified during systematic reviews as part of the of current clinical evidence.

In that way, publication bias distorts precision and internal validity of clinical evidence. The extent of such distortion thus requires the assessment of publication bias risk as part of the systematic review methodology. Funnel plots and Egger’s regression form one of the best currently available methods when assessing for publication bias risk.

PROBLEM: However, funnel plot and Egger’s regression have weak statistical power and its results may even be misleading when the number of included trials/datasets is small. They do not measure risk of publication bias directly but statistical in-between-trials heterogeneity in terms of their relation of sample size (SE) to effect size (lnRR). In that regard, the lack of published small trials is only one possible reason for any observed asymmetry. Other reasons are: (i) General in-between-trial heterogeneity and (ii) Methodological artifacts. Both will give positive results even in complete absence of publication bias risk.

SUGGESTED SOLUTION: Therefore, funnel plot and Egger’s regression need to be applied together with other considerations, which are presented in this research note, in order to obtain meaningful results.