Survey Operations Management: Monitoring Production, Cost, and Quality

November 17, 2015, 13:45 h
Mannheim (13:45-16:00 h at MZES in A5,6, Room A231)

Brad Edwards


Nonsampling errors and their relationship to cost have been very difficult to quantify, especially in real time. Recent advances in paradata offer an opportunity to address this problem. This talk summarizes some recent developments in computer-assisted recorded interviews (CARI), mobile technology, global positioning systems (GPS), and dashboard design that improve our ability to monitor quality of data collection activities on household surveys.

  • For CARI, we report the results of an experiment to investigate quality improvements in interviewer behavior during interviews, by providing feedback on performance in several ways.
  • For mobile devices, we will demonstrate an application deployed on smartphones that enables interviewers to record outcomes of contact attempts in near real time and to report on their time use by type of activity, and provide some results from a national study in the U.S.
  • With GPS, we can track interviewer movement throughout the work day, and link it with activity data. These data will be used to create routing maps for supervisor review with the data collector to improve efficiency. We also report on experience running GPS falsification-detection techniques alongside more traditional methods.
  • A performance dashboard gives supervisors an at-a-glance view of key performance indicators on production, cost, and quality; signals quality anomalies that need immediate attention; and supports drilling down and through paradata systems to investigate problems. The talk will address issues in quantifying nonsampling errors and costs, using visualization and process quality monitoring techniques.

Download presentation (1.41 MB)

YouTube-Video of presentation

About the speaker

Brad Edwards is a Vice President, Director of Field Services, and Deputy Area Director at Westat, with more than 30 years of experience designing and managing large, complex face-to-face and telephone surveys.  Research interests include multicultural issues, panel survey design, mobile devices, computer-assisted audio-recorded interviews (CARI), and field data collector recruitment, training, and management. He co-chaired the 2015 international conference Total Survey Error:  Improving Quality in the Era of Big Data, and is an editor of the follow-up monograph, Total Survey Error in Practice (Paul Biemer, lead editor, to be published by Wiley in late 2016). He co-edited the 2010 Wiley monograph Survey Methods in Multinational, Multiregional, and Multicultural Contexts (Janet Harkness, lead editor), which won the 2013 AAPOR Book Award, and the 2014 Cambridge book, Hard-to-Survey Populations (Roger Tourangeau, lead editor).