A critical review of studies investigating the quality of data obtained with online panels based on probability and nonprobability samples1

Mario Callegaro, Ana Villar, David Yeager, Jon A. Krosnick

Research output: Chapter in Book/Report/Conference proceedingChapter

20 Citations (Scopus)

Abstract

This chapter provides an overview of studies comparing the quality of data collected by online survey panels by looking at three criteria: (1) comparisons of point estimates from online panels to high-quality, established population benchmarks; (2) comparisons of the relationship among variables; and (3) the reproducibility of results for online survey panels conducted on probability samples to panels conducted on nonprobability samples. When looking at point estimates, all online survey panels differed to some extent from the population benchmarks. However, the largest comparison studies suggest that point estimates from online panels of nonprobability samples have higher differences as compared to benchmarks than online panels of probability samples. This finding is consistent across time and across studies conducted in different countries. Moreover, post-stratification weighting strategies helped little and in an inconsistent way to reduce such differences for data coming from online panels of nonprobability samples, whereas these strategies did bring estimates from online panels of probability samples consistently closer to the benchmarks. When comparing relationships among variables, it was found that researchers would reach different conclusions when using online panels of nonprobability samples versus panels of probability samples. When looking at reproducibility of results, the limited evidence found suggests that there are no substantial differences in replication and effect size across probability and nonprobability samples for question wording experiments and when comparing students samples to other samples. It is worth noting that in pre-election polls, an area where abundant prior knowledge exists, online panels of nonprobability samples have consistently performed as well and in some cases better than polls based on probability samples in predicting election winners.

Original languageEnglish (US)
Title of host publicationOnline Panel Research
Subtitle of host publicationA Data Quality Perspective
PublisherWiley Blackwell
Pages23-53
Number of pages31
Volume9781119941774
ISBN (Electronic)9781118763520
ISBN (Print)9781119941774
DOIs
StatePublished - May 27 2014

Fingerprint

Point Estimate
Benchmark
Reproducibility
Elections
Review
Post-stratification
Effect Size
Prior Knowledge
Inconsistent
Replication
Weighting
Estimate
Experiment
Strategy
Relationships

Keywords

  • Life of an online panel member
  • Multiple panel membership
  • Nonprobability online panel
  • Point estimates
  • Pre-election polls
  • Probability online panel
  • Relationship among variables
  • Reproducibility of study results
  • Weighting

ASJC Scopus subject areas

  • Mathematics(all)

Cite this

Callegaro, M., Villar, A., Yeager, D., & Krosnick, J. A. (2014). A critical review of studies investigating the quality of data obtained with online panels based on probability and nonprobability samples1. In Online Panel Research: A Data Quality Perspective (Vol. 9781119941774, pp. 23-53). Wiley Blackwell. https://doi.org/10.1002/9781118763520.ch2

A critical review of studies investigating the quality of data obtained with online panels based on probability and nonprobability samples1. / Callegaro, Mario; Villar, Ana; Yeager, David; Krosnick, Jon A.

Online Panel Research: A Data Quality Perspective. Vol. 9781119941774 Wiley Blackwell, 2014. p. 23-53.

Research output: Chapter in Book/Report/Conference proceedingChapter

Callegaro, M, Villar, A, Yeager, D & Krosnick, JA 2014, A critical review of studies investigating the quality of data obtained with online panels based on probability and nonprobability samples1. in Online Panel Research: A Data Quality Perspective. vol. 9781119941774, Wiley Blackwell, pp. 23-53. https://doi.org/10.1002/9781118763520.ch2
Callegaro M, Villar A, Yeager D, Krosnick JA. A critical review of studies investigating the quality of data obtained with online panels based on probability and nonprobability samples1. In Online Panel Research: A Data Quality Perspective. Vol. 9781119941774. Wiley Blackwell. 2014. p. 23-53 https://doi.org/10.1002/9781118763520.ch2
Callegaro, Mario ; Villar, Ana ; Yeager, David ; Krosnick, Jon A. / A critical review of studies investigating the quality of data obtained with online panels based on probability and nonprobability samples1. Online Panel Research: A Data Quality Perspective. Vol. 9781119941774 Wiley Blackwell, 2014. pp. 23-53
@inbook{6cae20f26a4042c281b0dc85a6d1b103,
title = "A critical review of studies investigating the quality of data obtained with online panels based on probability and nonprobability samples1",
abstract = "This chapter provides an overview of studies comparing the quality of data collected by online survey panels by looking at three criteria: (1) comparisons of point estimates from online panels to high-quality, established population benchmarks; (2) comparisons of the relationship among variables; and (3) the reproducibility of results for online survey panels conducted on probability samples to panels conducted on nonprobability samples. When looking at point estimates, all online survey panels differed to some extent from the population benchmarks. However, the largest comparison studies suggest that point estimates from online panels of nonprobability samples have higher differences as compared to benchmarks than online panels of probability samples. This finding is consistent across time and across studies conducted in different countries. Moreover, post-stratification weighting strategies helped little and in an inconsistent way to reduce such differences for data coming from online panels of nonprobability samples, whereas these strategies did bring estimates from online panels of probability samples consistently closer to the benchmarks. When comparing relationships among variables, it was found that researchers would reach different conclusions when using online panels of nonprobability samples versus panels of probability samples. When looking at reproducibility of results, the limited evidence found suggests that there are no substantial differences in replication and effect size across probability and nonprobability samples for question wording experiments and when comparing students samples to other samples. It is worth noting that in pre-election polls, an area where abundant prior knowledge exists, online panels of nonprobability samples have consistently performed as well and in some cases better than polls based on probability samples in predicting election winners.",
keywords = "Life of an online panel member, Multiple panel membership, Nonprobability online panel, Point estimates, Pre-election polls, Probability online panel, Relationship among variables, Reproducibility of study results, Weighting",
author = "Mario Callegaro and Ana Villar and David Yeager and Krosnick, {Jon A.}",
year = "2014",
month = "5",
day = "27",
doi = "10.1002/9781118763520.ch2",
language = "English (US)",
isbn = "9781119941774",
volume = "9781119941774",
pages = "23--53",
booktitle = "Online Panel Research",
publisher = "Wiley Blackwell",
address = "Japan",

}

TY - CHAP

T1 - A critical review of studies investigating the quality of data obtained with online panels based on probability and nonprobability samples1

AU - Callegaro, Mario

AU - Villar, Ana

AU - Yeager, David

AU - Krosnick, Jon A.

PY - 2014/5/27

Y1 - 2014/5/27

N2 - This chapter provides an overview of studies comparing the quality of data collected by online survey panels by looking at three criteria: (1) comparisons of point estimates from online panels to high-quality, established population benchmarks; (2) comparisons of the relationship among variables; and (3) the reproducibility of results for online survey panels conducted on probability samples to panels conducted on nonprobability samples. When looking at point estimates, all online survey panels differed to some extent from the population benchmarks. However, the largest comparison studies suggest that point estimates from online panels of nonprobability samples have higher differences as compared to benchmarks than online panels of probability samples. This finding is consistent across time and across studies conducted in different countries. Moreover, post-stratification weighting strategies helped little and in an inconsistent way to reduce such differences for data coming from online panels of nonprobability samples, whereas these strategies did bring estimates from online panels of probability samples consistently closer to the benchmarks. When comparing relationships among variables, it was found that researchers would reach different conclusions when using online panels of nonprobability samples versus panels of probability samples. When looking at reproducibility of results, the limited evidence found suggests that there are no substantial differences in replication and effect size across probability and nonprobability samples for question wording experiments and when comparing students samples to other samples. It is worth noting that in pre-election polls, an area where abundant prior knowledge exists, online panels of nonprobability samples have consistently performed as well and in some cases better than polls based on probability samples in predicting election winners.

AB - This chapter provides an overview of studies comparing the quality of data collected by online survey panels by looking at three criteria: (1) comparisons of point estimates from online panels to high-quality, established population benchmarks; (2) comparisons of the relationship among variables; and (3) the reproducibility of results for online survey panels conducted on probability samples to panels conducted on nonprobability samples. When looking at point estimates, all online survey panels differed to some extent from the population benchmarks. However, the largest comparison studies suggest that point estimates from online panels of nonprobability samples have higher differences as compared to benchmarks than online panels of probability samples. This finding is consistent across time and across studies conducted in different countries. Moreover, post-stratification weighting strategies helped little and in an inconsistent way to reduce such differences for data coming from online panels of nonprobability samples, whereas these strategies did bring estimates from online panels of probability samples consistently closer to the benchmarks. When comparing relationships among variables, it was found that researchers would reach different conclusions when using online panels of nonprobability samples versus panels of probability samples. When looking at reproducibility of results, the limited evidence found suggests that there are no substantial differences in replication and effect size across probability and nonprobability samples for question wording experiments and when comparing students samples to other samples. It is worth noting that in pre-election polls, an area where abundant prior knowledge exists, online panels of nonprobability samples have consistently performed as well and in some cases better than polls based on probability samples in predicting election winners.

KW - Life of an online panel member

KW - Multiple panel membership

KW - Nonprobability online panel

KW - Point estimates

KW - Pre-election polls

KW - Probability online panel

KW - Relationship among variables

KW - Reproducibility of study results

KW - Weighting

UR - http://www.scopus.com/inward/record.url?scp=84927800015&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84927800015&partnerID=8YFLogxK

U2 - 10.1002/9781118763520.ch2

DO - 10.1002/9781118763520.ch2

M3 - Chapter

AN - SCOPUS:84927800015

SN - 9781119941774

VL - 9781119941774

SP - 23

EP - 53

BT - Online Panel Research

PB - Wiley Blackwell

ER -