Evaluating Uighur literary translation: A comparative study of ChatGPT, Google Translate, and Bing Translator

Abstract

This study compares generative artificial intelligence (GenAI) and neural machine translation (NMT) systems in translating Uighur literary text (قۇتادغۇ بىلىك)into English. Two NMT systems, Google Translate and Bing Translator, were evaluated alongside ChatGPT, a GenAI large language model, under two prompt strategies. Translation quality was assessed through automatic metrics (BLEU, ROUGE-N/L, METEOR, and BERT-based semantic similarity), automated error counts (grammar, spelling, style), and expert ratings across four dimensions. Qualitative examples of culturally sensitive excerpts were also examined to illustrate success and failure cases. Results show that ChatGPT, especially with a concise instruction prompt, generally outperforms NMT systems in semantic accuracy, fluency, and cultural adequacy. Bing Translator produced the highest number of errors, particularly spelling mistakes, while Google Translate demonstrated more stable but moderate performance. Statistical testing and expert evaluations supported these patterns, and case analyses revealed how NMT outputs often distorted meaning through polarity reversal and semantic shifts. The findings highlight prompt engineering as a key factor for improving GenAI-based literary translation while recognizing the complementary strengths of GenAI adaptability and NMT stability. Future research should expand language and system coverage and examine the role of human post-editing in enhancing translation quality.