Automatische Prüfung der Verständlichkeit von Fragebogen-Items mittels Eye-Tracking (TACT)

Questionnaires

are important data collection tools used to gather self-report information from

individuals. In the social sciences, questionnaires are regularly applied to

monitor the society. In psychology, they are commonly used to assess, e.g.,

individuals’ personality traits, emotional states, or beliefs towards

particular issues. Moreover, in relation to temporary events, such as financial

crises or the current pandemic, questionnaires are also a suitable method to

rapidly assess the population’s opinions and attitudes. To obtain valid and

reliable data, however, it is essential to construct questionnaire items (i.e.,

questions or statements) that are clear and comprehensible for the target

group. For this purpose, survey methodologists pretest the items and check

whether respondents understand them as intended. A common method to discover

problematic items is cognitive interviewing, in which participants verbalize

potential comprehension problems. However, the output of such cognitive

interviews is bounded, e.g., to the respondents’ ability and willingness to

express their thoughts. As an additional unobtrusive method, eye-tracking can

be incorporated in pretests to observe respondents’ gaze behavior while

completing questionnaires and to detect further problems that otherwise would

go unnoticed. However, little is known so far about the relationship between

different eye-tracking measures, different kinds of comprehension problems, and

the role of individuals’ general reading skills. Furthermore, analyses of

eye-tracking data in questionnaire pretests can easily become very complex

(depending on the number of considered eye-tracking measures). In this

interdisciplinary project, we will contribute to the research field of

eye-tracking in questionnaire pretesting by combining expertise in reading

research and machine learning. In three experimental user studies, we will

collect eye-tracking data of a total of 150 participants (from a quota adult

sample) answering a set of 40 questionnaire items. We will analyze the roles of

item comprehensibility (experimentally varied) and of general reading skills on

participants’ gaze behavior by means of different fine-grained eye-tracking

measures. Based on these insights, we will model and train machine learning

algorithms to detect comprehension problems automatically. These models will be

tested for their generalizability in an evaluation study with another 50 participants.

In sum, the results of the proposed project will contribute to a better

understanding of how comprehension problems in questionnaire items manifest in

respondents’ eye movements, and result in automated methods to make pretests of

questionnaire items more efficient and reliable.