Comparative evaluation of OpenAI O1 and human performance in higher order cognition

Abstract

Abstract This study evaluates the performance of OpenAI’s o1-preview model in higher-order cognitive domains, including critical thinking, systematic thinking, computational thinking, data literacy, creative thinking, logical reasoning, and scientific reasoning. Using established benchmarks, we compared the o1-preview models’ performance to human participants from diverse educational levels. o1-preview achieved a mean score of 24.33 on the Ennis-Weir Critical Thinking Essay Test (EWCTET), surpassing undergraduate (13.8) and postgraduate (18.39) participants ( z = 1.60 and 0.90, respectively). In systematic thinking, it scored 46.1 ± 4.12 on the Lake Urmia Vignette, significantly outperforming the human mean (20.08 ± 8.13, z = 3.20). For data literacy, o1-preview scored 8.60 ± 0.70 on test “Use Data” dimension, compared to the human post-test mean of 4.17 ± 2.02 ( z = 2.19). On creative thinking tasks, the model achieved originality scores of 2.98 ± 0.73, higher than the human mean of 1.74 ( z = 0.71). In logical reasoning (LogiQA), it outperformed humans with 90% ± 10 accuracy versus 86% ± 6.5 ( z = 0.62). For scientific reasoning, it achieved near-perfect performance (0.99 ± 0.12) on the TOSLS, exceeding the highest human scores of 0.85 ± 0.13 ( z = 1.78). While o1-preview excelled in structured tasks, it showed limitations in problem-solving and adaptive reasoning. These results demonstrate the potential of AI to complement education in structured assessments but highlight the need for ethical oversight and refinement for broader applications.