DReMSS Webinar_ Harnessing Generative AI for questionnaire design, evaluation and testing

This webinar examines how generative AI can be integrated into the questionnaire design and evaluation process. Rather than treating large language models as ad hoc drafting tools, we discuss structured, workflow-embedded applications that support measurement quality at multiple stages of survey development. Several use cases are discussed, along with findings from validation exercises.

First, we present an occupational coding application that maps open-ended job descriptions to SOC2020 classifications using structured prompts and validation routines, illustrating how AI can support consistent and scalable post-fieldwork coding. We also describe how the tool can be embedded in a dynamic questionnaire design and generate intelligent follow-ups to probe unclassifiable answers. Second, a “Silicon Sampling” generator is presented that simulates heterogeneous respondent personas to stress-test draft questions at scale, identifying ambiguity, response instability, and potential measurement error prior to fielding. Third, an AI-enabled cognitive interviewing tool that operationalises probing protocols to examine comprehension, retrieval, judgement, and response processes in a systematic and reproducible way. Fourth, we present an example of a scalable pipeline for LLM evaluations of draft survey questions using rule-based question evaluation frameworks designed to identify question features that cause response challenges) (in this case, the Question Appraisal System (QAS-99). We share the results of a validation exercise to assess how LLM evaluations using the QAS compare with those of human experts and novices. Finally, we discuss how such tools can be integrated into an agentic workflow to support survey questionnaire development.

Presenters:

Patrick Sturgis is Professor of Quantitative Social Science and Head of Department in the Department of Methodology at the London School of Economics. He was previously Director of the ESRC National Centre for Research Methods at the University of Southampton 2010-2019. His research focuses on applied quantitative and statistical methods, with a particular specialism in survey design and analysis. He was President of the European Survey Research Association 2011-2015 and has published widely on in leading methodology journals including Journal of the Royal Statistical Society, Public Opinion Quarterly, and Journal of Survey Methods and Statistics. He has served as Chair of the Methodological Advisory Board of the European Social Survey and the UK Household Longitudinal Survey. He is currently Principal Investigator of ‘Harnessing Generative AI for Questionnaire Design, Evaluation and Testing’, a research grant under the ESRC Survey Futures programme with Dr Caroline Roberts and Dr Tom Robinson.

Caroline Roberts is a Senior Lecturer and Researcher in the Institute of Social Sciences at the University of Lausanne, where she teaches MA courses on survey research methods and questionnaire design, and regularly gives specialist training in summer schools and short courses on related topics. She is also an affiliated Survey Methodologist at FORS, the Swiss Centre of Expertise in the Social Sciences, and a Visiting Senior Fellow in the Department of Methodology at the London School of Economics and Political Science. Her research interests in survey methodology relate to the measurement and reduction of different types of survey error. Her most recent research focuses on challenges relating to the implementation of digital data collection in high quality general population surveys, and ways to leverage generative AI in questionnaire design, evaluation and testing. She was Chair of the Methods Advisory Board of the European Social Survey from 2020-2025 and President of the European Survey Research Association from 2019-2021.