DiSCoKit: An open-source toolkit for deploying live LLM experiences in survey research

Abstract

In scientific studies of human-AI interaction dynamics, researchers often need to present participants with opportunities to interact with live large-language models (LLMs). However, technical and practical challenges (from survey platform limitations and logging chat data to manipulating AI behaviors for experimental designs) often inhibit survey-based deployment of AI stimuli. We developed DiSCoKit—an open-source toolkit for deploying live LLM experiences (e.g., ones based on models delivered through Microsoft Azure portal) through JavaScript-enabled survey platforms (e.g., Qualtrics). We describe the toolkit’s scientific motivation, architecture, and operation. We also offer an example of toolkit deployment and customization, along with discussing its possibilities and limitations. Altogether, DiSCoKit gives researchers a flexible, secure, scalable solution for deploying naturalistic LLM stimulus experiences through online surveys.