Interface
[Overview] The interface created for our paper’s experiment is used for studying (1) how AI use affects critical thinking task performance and (2) how people use AI during critical thinking tasks. The critical thinking task asks participants to make a reasoned written decision based on a decision-making scenario and a set of documents of varying characteristics. The main task interface has three components: a document viewer, an LLM chatbot (if applicable for the experimental condition), and a text editor. It controls the access timing of the LLM chatbot and the time availability for task completion, enabling experiments within a two-dimensional time constraint space. After the task, participants sequentially restate their decision, complete a free recall assessment, evaluate document characteristics, and answer comprehension questions.
What it logs: The interface captures keystroke, cursor, and navigation events across all panels, enabling analysis of how participants move between reading, using the LLM, and writing throughout the task. These raw logs yield behavioral metrics at different granularities — see the data dictionary for how these metrics are derived, including those that show notable patterns in our paper.
For reuse:
The interface can be used for human-centered experiments on AI’s impact on critical thinking and other cognitive tasks involving a non-linear process of reading and writing. Researchers can replace the task scenario and documents, and revise or skip the post-task questions.
The time constraint controls can be configured independently, or researchers can use the two base versions (with or without the LLM) without any time constraints.
Interface Code
Data and Analysis