Opening the File Drawer: Assessing and Understanding Publication Bias in the Economic, Behavioral and Social Sciences by Utilizing two German Academic Access Panels (PubBias)
Abstract
Publication
bias can be regarded as the publication (or non-publication) of study results
due to the direction or strength. That is when, for example, statistically
significant results are more likely to be published than statistically non-significant
results. Consequently, publication bias hinders the assessment of the true
state of knowledge on a particular research question, resulting in scientific
debates and political decisions that are based on false or inadequate scientific
evidence. Several factors impact the emergence
of publication bias, most importantly the actors directly involved: authors,
editors, and reviewers. In addition, the disciplinary and cultural background affects
the likelihood and extent of publication bias as well.
In our
project, we plan to focus on the various steps of researchers’ decision-making processes,
which may lead to the publication of selected results while leaving other
results in the “file drawer.” The project has two main objectives: (I) First,
it aims to measure the extent of publication bias for the economic, behavioral,
and social sciences in Germany. This will be accomplished by contrasting
published and unpublished GESIS Panel and SOEP-Innovation Sample (SOEP-IS) study
submissions. (II) Second, building upon existing theoretical frameworks on the
causes of publication bias, we will empirically test the derived hypotheses and
(if necessary) adjust and/or extent the current theoretical models.
To investigate
researchers’ decision processes, this project will be in the unique position to
open the file drawer and observe the entire research cycle. It aims to look
closely at research practices during the way from a study proposal to the publication
stage by using data from study submissions of external researchers to the GESIS
Panel and the SOEP-IS. In these two German access panel infrastructures,
external researchers from all disciplines can submit their own studies.
However, they must include a thorough description of the theoretical
background, the hypotheses, the research design, which are then subject to a
peer review. Most successful study submissions result in a corresponding
publication; some, however, do not. In this context, 184 successful study
submissions from both panels combined allow us to investigate which study
characteristics (e.g., seniority and number of authors, experimental vs.
non-experimental research design) are associated with a higher likelihood of a successful
publication. In addition, we will conduct an author survey to collect further actor-based
data on motives, attitudes, and values with respect to the research process. Furthermore,
for a randomly selected subset of successful study submissions without any corresponding
publication, we will conduct own statistical analyses using GESIS Panel and
SOEP-IS data to investigate if they have a high likelihood of producing statistically
non-significant results, which might explain the lack of a corresponding
publication.
Runtime
2023-07-01 – 2026-06-30Funding
Partner
- SOEP – The Socio-Economic Panel, DIW – German Institute of Economic Research