Opening the File Drawer: Assessing and Understanding Publication Bias in the Economic, Behavioral and Social Sciences by Utilizing two German Academic Access Panels (PubBias)

Publication

bias can be regarded as the publication (or non-publication) of study results

due to the direction or strength. That is when, for example, statistically

significant results are more likely to be published than statistically non-significant

results. Consequently, publication bias hinders the assessment of the true

state of knowledge on a particular research question, resulting in scientific

debates and political decisions that are based on false or inadequate scientific

evidence. Several factors impact the    emergence

of publication bias, most importantly the actors directly involved: authors,

editors, and reviewers. In addition, the disciplinary and cultural background affects

the likelihood and extent of publication bias as well.

In our

project, we plan to focus on the various steps of researchers’ decision-making processes,

which may lead to the publication of selected results while leaving other

results in the “file drawer.” The project has two main objectives: (I) First,

it aims to measure the extent of publication bias for the economic, behavioral,

and social sciences in Germany. This will be accomplished by contrasting

published and unpublished GESIS Panel and SOEP-Innovation Sample (SOEP-IS) study

submissions. (II) Second, building upon existing theoretical frameworks on the

causes of publication bias, we will empirically test the derived hypotheses and

(if necessary) adjust and/or extent the current theoretical models.

To investigate

researchers’ decision processes, this project will be in the unique position to

open the file drawer and observe the entire research cycle. It aims to look

closely at research practices during the way from a study proposal to the publication

stage by using data from study submissions of external researchers to the GESIS

Panel and the SOEP-IS. In these two German access panel infrastructures,

external researchers from all disciplines can submit their own studies.

However, they must include a thorough description of the theoretical

background, the hypotheses, the research design, which are then subject to a

peer review. Most successful study submissions result in a corresponding

publication; some, however, do not. In this context, 184 successful study

submissions from both panels combined allow us to investigate which study

characteristics (e.g., seniority and number of authors, experimental vs.

non-experimental research design) are associated with a higher likelihood of a successful

publication. In addition, we will conduct an author survey to collect further actor-based

data on motives, attitudes, and values with respect to the research process. Furthermore,

for a randomly selected subset of successful study submissions without any corresponding

publication, we will conduct own statistical analyses using GESIS Panel and

SOEP-IS data to investigate if they have a high likelihood of producing statistically

non-significant results, which might explain the lack of a corresponding

publication.