Comparative analysis of AI-generated and original abstracts of academic articles on philology

Generative artificial intelligence systems have a significant impact on tasks related to natural language processing: machine translation, sentiment analysis, text generation, and summarisation, etc. The aim of the presented work was to determine the features of automatically generated academic texts in comparison with texts created by authors, and to evaluate the capabilities of different methods in relation to the task of their classification. The paper analyses two types of abstracts: collected from academic journals on computational linguistics and Germanic studies and generated from the titles of the corresponding articles using ChatGPT-4o mini. The total amount of data was 60 items. The choice of article topics is due to the fact that the texts belong to the same subject area but differ in their structure. The first group which contains original texts on computational linguistics, is similar to the abstracts of academic articles on computer science, and contains a large amount of English terminology. The second group contains texts on Germanic studies and is more descriptive-narrative in their nature. We analyzed the differences between the two types of abstracts and classified them into two categories with the help of experts, three detector systems to determine the involvement of artificial intelligence in the creation of texts (Smodin, ZeroGPT and GPTZero), as well as the ChatGPT system itself. The analysis showed that the generated texts are characterized by a clear formal structure and adherence to the rules of academic text construction in accordance with IMRAD (Introduction, Methods, Results and Discussion). They are superficial in content and they do not always follow the scientific style; there are repetitions of constructions and paraphrasing of article titles, which is not found in the abstracts written by the authors without artificial intelligence. Automatically generated abstracts need not only further editing (because in some cases lexical and syntactic coherence is broken and ambiguity is present), but also verification of the facts and terms mentioned. Among the detector systems, the highest scores in Precision, Accuracy and F1-score are achieved by Smodin tools, while the best results in Recall are achieved by ZeroGPT. The lowest results in abstract evaluation when compared with other tools were achieved by the ChatGPT system itself. Expert-assisted classification showed the highest results in the case of Germanic abstracts. The results may be useful for researchers when working with academic texts on linguistics as well as for further finetuning of neural network models.

1. Mikolov T., Chen K., Corrado G., Dean J. Efficient estimation of word representations in vector space // Proc. of the 1st International Conference on Learning Representations, ICLR 2013 — Workshop Track Proceedings. 2013. P. 1–12.

2. Mikolov T., Sutskever I., Chen K., Corrado G.S., Dean J. Distributed representations of words and phrases and their compositionality // Advances in Neural Information Processing Systems. 2013. V. 26. P. 3111–3119.

3. Orăsan C. ChatGPT for translators: a survey // Proc. of the First Workshop on NLP Tools and Resources for Translation and Interpreting Applications. 2023. P. 61–63.

4. Castilho S., Mallon C.Q., Meister R., Yue S. 2023. Do online machine translation systems care for context? What about a GPT model? // Proc. of the 24th Annual Conference of the European Association for Machine Translation. 2023. P. 393–417.

5. Rajpurkar P., Jia R., Liang P. Know what you don’t know: Unanswerable questions for SQuAD // Proc. of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). 2018. P. 784–789. https://doi.org/10.18653/v1/p18-2124

6. Bahak H., Taheri F., Zojaji Z., Kazemi A. Evaluating ChatGPT as a question answering system: A comprehensive analysis and comparison with existing models // arXiv. 2023. arXiv:2312.07592. https://doi.org/10.48550/arXiv.2312.07592

7. Alkaissi H., McFarlane S.I. Artificial hallucinations in ChatGPT: Implications in scientific writing // Cureus. 2023. V. 15. N 2. P. e35179. https://doi.org/10.7759/cureus.35179

8. Stokel-Walker C. ChatGPT listed as author on research papers: many scientists disapprove // Nature. 2023. V. 613(7945). P. 620–621. https://doi.org/10.1038/d41586-023-00107-z

9. Ide K., Hawke P., Nakayama T. Can ChatGPT be considered an author of a medical article? // Journal of Epidemiology. 2023. V. 33. N 7. P. 381–382. https://doi.org/10.2188/jea.JE20230030

10. Dwivedi Y.K., Kshetri N., Hughes L., Slade E.L., Jeyaraj A., Kar A.K., Wright R. et al. Opinion Paper: “So what if ChatGPT wrote it?” Multidisciplinary perspectives on opportunities, challenges and implications of generative conversational AI for research, practice and policy // International Journal of Information Management. 2023. V. 7. P. 102642. https://doi.org/10.1016/j.ijinfomgt.2023.102642

11. Tools such as ChatGPT threaten transparent science; here are our ground rules for their use // Nature. 2023. V. 613(7945). https://doi.org/10.1038/d41586-023-00191-1

12. Rahman M., Terano H.J.R., Rahman N., Salamzadeh A., Rahaman S. ChatGPT and academic research: A review and recommendations based on practical examples // Journal of Education, Management and Development Studies. 2023. V. 3. N 1. P. 1–12. https://doi.org/10.52631/jemds.v3i1.175

13. Herbold S., Hautli-Janisz A., Heuer U., Kikteva Z., Trautsch A. A large-scale comparison of human-written versus ChatGPT-generated essays // Scientific Reports. 2023. V. 13. N 1. P. 18617. https://doi.org/10.1038/s41598-023-45644-9

14. Gao Y., Wang R., Hou F. How to design translation prompts for ChatGPT: An empirical study // arXiv. 2023. arXiv:2304.02182. https://doi.org/10.48550/arXiv.2304.02182

15. Kolmogorova A.V., Margolina A.V. Written vs generated text: “naturalness” as a textual and psycholinguistic category // Научный результат. Вопросы теоретической и прикладной лингвистики. 2024. Т. 10. № 2. C. 71–99. https://doi.org/10.18413/2313-8912-2024-10-2-0-4

16. Черкасова М.Н., Тактарова А.В. Признаки сгенерированного текста в академическом дискурсе: проблема идентификации // Филологические науки. Вопросы теории и практики. 2024. Т. 17. № 7. С. 2226–2232. https://doi.org/10.30853/phil20240307

17. Aydın Ö., Karaarslan E. OpenAI ChatGPT generated literature review: Digital twin in healthcare // Emerging Computer Technologies 2. İzmir Akademi Dernegi, 2022. P. 22–31. https://doi.org/10.2139/ssrn.4308687

18. Mindner L., Schlippe T., Schaaff K. Classification of human- and AI-generated texts: Investigating features for ChatGPT // Lecture Notes on Data Engineering and Communications Technologies. 2023. V. 190. P. 152–170. https://doi.org/10.1007/978-981-99-7947-9_12