Writing creativity, cohesion, and formal linguistic competence in LLMs: A comparative evaluation based on English and Chinese continuation writing

Abstract

Recent advancements in large language models (LLMs), such as GPT-3.5, GPT-4, and Chinese models like ERNIE and SparkDesk (SPARK), have shown remarkable potential in story generation. However, a direct comparison between ChatGPTs, ERNIE, and SPARK (pretrained with dominant Chinese Corpus, referred to herein as C-dominant LLMs) remains underexplored in this domain, particularly regarding narrative writing continuation ability. Identifying such difference would be beneficial to teachers to adopt LLMs in writing practice. This study addresses this issue by comparing ChatGPTs and C-dominant LLMs in story continuation tasks, with a focus on cohesion, creativity, and formal linguistic competence. We harnessed advanced analytical tools, including Coh-Metrix and CTAP for cohesion and LIWC-22 and CLIWC for creativity, to assess the performance of GPT-3.5, GPT-4, ERNIE, and SPARK across English and Chinese continuations. Results indicated that C-dominant LLMs surpassed ChatGPTs in referential and deep cohesion for English story generation, while ChatGPTs excelled in latent semantic analysis. Conversely, ChatGPTs demonstrated superior performance in cohesion across all metrics for Chinese story generation. In terms of creativity, GPT models behaved better in image for English texts and voice for both English and Chinese, whereas C-dominant LLMs excelled in image for Chinese continuations. For formal linguistic competence, GPTs showed better performance in English tasks, while C-dominant LLMs in Chinese tasks. These findings provide insights for optimizing LLMs in multilingual contexts and offer guidance for their application in writing and translation based on diverse languages.