“Please speak in Hindi”: Cognitive interviews to improve measurement of digital access in North India

Abstract

Access to and use of mobile phones are increasingly a prerequisite for full participation in modern society. Yet, a significant digital gender gap persists in India, where women are less likely than men to own phones or access the internet. To measure and understand this gap, survey questions must be both comprehensible to the respondents and contextually appropriate. We conducted 101 cognitive interviews with men (n = 44) and women (n = 57) in rural Uttar Pradesh to test 118 draft survey questions developed for an upcoming survey on digital access and use. These draft questions were drawn primarily from validated global instruments, with some additional questions newly developed by our team, and were all translated into Hindi. Through team debriefs and systematic analysis of the interview transcripts, we identified seven categories of question problems: (1) inappropriate terminology, (2) overly complex wording, (3) low resonance of digital concepts (e.g., personal data, online tracking, privacy policies, hacking), (4) confusion when asking about permission and supervision of mobile phone use due to phone sharing and overlap with concept of help or support, (5) problematic question structures and response options (especially Likert scales), (6) self-practice bias, and (7) unclear time frames and recall expectations. Overall, we found that many draft survey questions were incomprehensible to respondents despite our research team’s efforts to develop clear and simple translations. Revisions were required for almost all questions. Our findings highlight the critical role of cognitive interviewing in improving data quality, particularly when conducting surveys across languages and sociocultural contexts, and amid rapidly changing digital landscapes.