https://www.gesis.org/typo3conf/ext/mindshape_cookie_hint/Resources/Public/Css/dark-bottom.css GESIS: Cognitive Pretests

Cognitive Pretests

Dr. Timo Lenzner

Survey Design and Methodology
Fragebogendesign & -evaluation

+49 (621) 1246-227
E-Mail
vCard

Dr. Cornelia Neuert

Survey Design and Methodology
Fragebogendesign & -evaluation

+49 (621) 1246-225
E-Mail
vCard

Patricia Hadler

Survey Design and Methodology
Fragebogendesign & -evaluation

+49 (621) 1246-198
E-Mail
vCard

Pretesting is generally considered to be indispensable to survey research for the successful development of a questionnaire, and for the quality of the data obtained by the survey.

The GESIS Pretest Lab supports researchers in optimizing their survey questions prior to data collection with various services. In addition, we provide researchers access to the findings of our cognitive pretests via the GESIS pretest database.

Conducting cognitive pretests

In cognitive pretests, we evaluate the comprehensibility of questions, identify problems respondents may have in answering and generate constructive suggestions for eliminating such problems. The pretests work well with all modes of questioning (personal-oral, telephone, written, online).

The additional application of eye tracking offers the possibility to follow and record (live) the eye movements of subjects. Our research findings suggest that this procedure helps to discover additional problems with questions (Neuert & Lenzner, 2015a, 2015b).

How do we conduct cognitive pretests?

  • With small samples of 10 to 30 participants
  • The individual interviews are limited to 60 to maximally 90 minutes
  • Per interview, we are able to test about 50 questions or items with the eye tracker and thereof about 20 questions or items via pre-scripted cognitive pretesting techniques.

Conducting cognitive online pretests (web probing)

Cognitive online pretests are an alternative to pretests conducted in the lab. With the probing questions used it is possible, for example, to determine whether all respondents interpret a question in the same way or whether there are systematic differences in the understanding of a question between different groups of respondents.

How do we conduct cognitive online pretests?

  • With samples of 80 to 120 participants
  • The online questionnaires are designed to take about 20 to 30 minutes to complete
  • Per questionnaire, we are able to test about 10 to 15 questions or items 

Coordination of cognitive pretests

Comparative surveys are increasingly being conducted in different countries and/or languages. We carry out cross-cultural cognitive pretests for projects of this kind, which can involve training cognitive interviewers in the participating countries, coordinating the pretests in the countries and analyzing the results of the cognitive interviews cross-nationally. 

 

Consulting on choosing and independent usage of various pretest procedures

Depending on the chosen mode of survey (personal-oral, telephone, written, online) various pretesting procedures for optimizing questionnaires are available to researchers (for example: cognitive interviewing, web probing, interviewer/respondent debriefing, focus groups, eye tracking, behavior coding). We are happy to consult with researchers in choosing and utilizing the appropriate pretesting procedures for those wanting to test their own survey instruments themselves.

Costs & Duration

Consultations on pretesting procedures are free of charge. The costs for conducting the pretests are project-specific and based on the number of test persons, number of questions being tested and the pretesting method to be used.

Example of a cognitive pretest (with/without eye tracking):

We normally conduct cognitive pretests with 16 participants. Planning, execution and evaluation of pretests such as this generally takes 8 to 10 weeks, including the written report on the results.

Example of a cognitive online pretest:

Cognitive pretests are usually conducted with 100 to 120 participants and 10 questions to be tested. Planning, execution and evaluation of pretests such as this generally takes 5 to 6 weeks, including the written report on the results.

References

Our varied client list thus far includes a number of externally funded research projects, in addition to the following survey programs, institutes and agencies: