Detecting Selection Bias in Meta-Analyses with Multiple Outcomes: A Simulation Study

Belén Fernández-Castilla, Lies Declercq, Laleh Jamshidi, S Natasha Beretvas, Patrick Onghena, Wim Van den Noortgate

Research output: Contribution to journalArticle

Abstract

This study explores the performance of classical methods for detecting publication bias—namely, Egger’s regression test, Funnel Plot test, Begg’s Rank Correlation and Trim and Fill method—in meta-analysis of studies that report multiple effects. Publication bias, outcome reporting bias, and a combination of these were generated. Egger’s regression test and the Funnel Plot test were extended to three-level models, and possible cutoffs for the estimator of the Trim and Fill method were explored. Furthermore, we checked whether the combination of results of several methods yielded a better control of Type I error rates. Results show that no method works well across all conditions and that performance depends mainly on the population effect size value and the total variance.

Original languageEnglish (US)
JournalJournal of Experimental Education
DOIs
StatePublished - Jan 1 2019

Fingerprint

Selection Bias
Meta-Analysis
simulation
trend
Publication Bias
regression
Population Density
performance
Publications
Values

Keywords

  • Meta-analysis
  • multiple effect sizes
  • publication bias
  • selective outcome reporting bias
  • simulation study

ASJC Scopus subject areas

  • Education
  • Developmental and Educational Psychology

Cite this

Detecting Selection Bias in Meta-Analyses with Multiple Outcomes : A Simulation Study. / Fernández-Castilla, Belén; Declercq, Lies; Jamshidi, Laleh; Beretvas, S Natasha; Onghena, Patrick; Van den Noortgate, Wim.

In: Journal of Experimental Education, 01.01.2019.

Research output: Contribution to journalArticle

Fernández-Castilla, Belén ; Declercq, Lies ; Jamshidi, Laleh ; Beretvas, S Natasha ; Onghena, Patrick ; Van den Noortgate, Wim. / Detecting Selection Bias in Meta-Analyses with Multiple Outcomes : A Simulation Study. In: Journal of Experimental Education. 2019.
@article{e3def56df98d470f8f39f525fa49c78f,
title = "Detecting Selection Bias in Meta-Analyses with Multiple Outcomes: A Simulation Study",
abstract = "This study explores the performance of classical methods for detecting publication bias—namely, Egger’s regression test, Funnel Plot test, Begg’s Rank Correlation and Trim and Fill method—in meta-analysis of studies that report multiple effects. Publication bias, outcome reporting bias, and a combination of these were generated. Egger’s regression test and the Funnel Plot test were extended to three-level models, and possible cutoffs for the estimator of the Trim and Fill method were explored. Furthermore, we checked whether the combination of results of several methods yielded a better control of Type I error rates. Results show that no method works well across all conditions and that performance depends mainly on the population effect size value and the total variance.",
keywords = "Meta-analysis, multiple effect sizes, publication bias, selective outcome reporting bias, simulation study",
author = "Bel{\'e}n Fern{\'a}ndez-Castilla and Lies Declercq and Laleh Jamshidi and Beretvas, {S Natasha} and Patrick Onghena and {Van den Noortgate}, Wim",
year = "2019",
month = "1",
day = "1",
doi = "10.1080/00220973.2019.1582470",
language = "English (US)",
journal = "Journal of Experimental Education",
issn = "0022-0973",
publisher = "Routledge",

}

TY - JOUR

T1 - Detecting Selection Bias in Meta-Analyses with Multiple Outcomes

T2 - A Simulation Study

AU - Fernández-Castilla, Belén

AU - Declercq, Lies

AU - Jamshidi, Laleh

AU - Beretvas, S Natasha

AU - Onghena, Patrick

AU - Van den Noortgate, Wim

PY - 2019/1/1

Y1 - 2019/1/1

N2 - This study explores the performance of classical methods for detecting publication bias—namely, Egger’s regression test, Funnel Plot test, Begg’s Rank Correlation and Trim and Fill method—in meta-analysis of studies that report multiple effects. Publication bias, outcome reporting bias, and a combination of these were generated. Egger’s regression test and the Funnel Plot test were extended to three-level models, and possible cutoffs for the estimator of the Trim and Fill method were explored. Furthermore, we checked whether the combination of results of several methods yielded a better control of Type I error rates. Results show that no method works well across all conditions and that performance depends mainly on the population effect size value and the total variance.

AB - This study explores the performance of classical methods for detecting publication bias—namely, Egger’s regression test, Funnel Plot test, Begg’s Rank Correlation and Trim and Fill method—in meta-analysis of studies that report multiple effects. Publication bias, outcome reporting bias, and a combination of these were generated. Egger’s regression test and the Funnel Plot test were extended to three-level models, and possible cutoffs for the estimator of the Trim and Fill method were explored. Furthermore, we checked whether the combination of results of several methods yielded a better control of Type I error rates. Results show that no method works well across all conditions and that performance depends mainly on the population effect size value and the total variance.

KW - Meta-analysis

KW - multiple effect sizes

KW - publication bias

KW - selective outcome reporting bias

KW - simulation study

UR - http://www.scopus.com/inward/record.url?scp=85063903485&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85063903485&partnerID=8YFLogxK

U2 - 10.1080/00220973.2019.1582470

DO - 10.1080/00220973.2019.1582470

M3 - Article

AN - SCOPUS:85063903485

JO - Journal of Experimental Education

JF - Journal of Experimental Education

SN - 0022-0973

ER -