SEMANTIC TEXT SIMILARITY ESTIMATION ACROSS DIVERSE LARGE LANGUAGE MODELS

Vitalii Parubochyi, Oleksandr Chmykhalo, Oleh Husak, Roman Shuvar

Abstract


Background. Semantic Text Similarity (STS) estimation is one of the useful proxy-based approaches for analyzing semantic differences between responses generated by various large language models (LLMs) used in AI chats and AI agents. Various studies in recent years have attempted to estimate STS for AI-generated responses, but they typically focus on ground-truth estimation by comparing AI-generated responses with predefined answers that, in most cases, differ structurally from the generated texts. However, cross-model STS estimation can provide valuable insights into model similarity, independent of how well they align with the predefined answers.

Materials and Methods. In this paper, we estimate both types of STS, but primarily focus on the STS between responses generated by several selected LLMs from different providers. We build our estimation framework using a subset of Frequently Asked Questions (FAQs) in English from the MFAQ dataset, as it offers a wide range of topics and a clear, easy-to-use structure.

Results and Discussion. The results of our experiments are organized into two parts. In the first part, we estimate ground-truth STS scores using multiple metrics (Cosine Similarity, BERTScore, and Latent Semantic Analysis (LSA)) to analyze how closely the generated answers from each model match the ground-truth answer in the MFAQ dataset. In the second part, we estimated cross-model STS scores using the same metrics to analyze how closely models align with each other. Additionally, we measure average request latency and use publicly available information on model context size and input and output prices to explore possible relationships between these parameters and STS-based quality indicators.

Conclusion. Our results indicate relatively high semantic similarity between models from the same provider in the selected experimental setup and demonstrate that high ground-truth STS scores do not necessarily imply high inter-model semantic similarity within the selected benchmark and metrics, as seen in the analysis of cross-model STS scores. Additionally, we show that not all metrics are equally suitable for cross-model STS estimation, and that metrics like Cosine Similarity or LSA scores can provide better sensitivity than BERTScore or similar metrics.

Keywords: semantic text similarity (STS), large language model (LLM), MFAQ dataset, cosine similarity, BERTS core, latent semantic analysis (LSA)


Full Text:

PDF

References


[1] Sblendorio, E., Lo Cascio, A., Napolitano, D., Germini, F., Dentamaro, V., Piredda, M., & Cicolini, G. (2025). Assessing and Comparing Free Large Language Models’ Responses to a Clinical Case: Accuracy, Safety, and Reliability. Computational Intelligence Methods for Bioinformatics and Biostatistics, CIBB 2024. Lecture Notes in Computer Science, 15276, 134-149. https://doi.org/10.1007/978-3-031-89704-7_11

[2] Lu, D., Jeuring, J., & Gatt, A. (2025). Evaluating LLM-Generated Versus Human-Authored Responses in Role-Play Dialogues. In Proceedings of the 18th International Natural Language Generation Conference, Association for Computational Linguistics, Hanoi, Vietnam, 20–40. https://doi.org/10.48550/arXiv.2509.17694

[3] Ding, X. et al. (2024). Evaluation of LLMs and Other Machine Learning Methods in the Analysis of Qualitative Survey Responses for Accessible Engineering Education Research. 2024 ASEE Annual Conference & Exposition, Portland, Oregon, USA, 1-24. https://doi.org/10.18260/1-2--47360

[4] Wu, X., Lin, W., Akgul, O., & Bauer, L. (2025). Estimating LLM Consistency: A User Baseline vs Surrogate Metrics. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, Suzhou, China, 30530-30544. https://doi.org/10.18653/v1/2025.emnlp-main.1554

[5] Lee, P. C., Sharma, S. K., Motaganahalli, S., & Huang, A. (2023). Evaluating the Clinical Decision-Making Ability of Large Language Models Using MKSAP-19 Cardiology Questions. JACC: Advances, 2(9). https://doi.org/10.1016/j.jacadv.2023.100658

[6] Zhang, L., Wang, J., Wu, J., & Zhang, Z. (2026). RetailBench: Evaluating Long-Horizon Autonomous Decision-Making and Strategy Stability of LLM Agents in Realistic Retail Environments. ArXiv preprint. https://doi.org/10.48550/arXiv.2603.16453

[7] Brand, J., Israeli, A., & Ngwe, D. (2023). Using LLMs for Market Research. Harvard Business School Working Paper, No. 23-062, April 2023 (Revised October 2025). https://doi.org/10.2139/ssrn.4395751

[8] Wang, M., Zhang, D., & Zhang, H. (2024). Large Language Models for Market Research: A Data-augmentation Approach. Social Science Research Network (SSRN). https://doi.org/10.2139/ssrn.5057769

[9] De Bruyn, M., Lotfi, E., Buhmann, J., & Daelemans, W. (2021). MFAQ: a Multilingual FAQ Dataset. In Proceedings of the 3rd Workshop on Machine Reading for Question Answering, Association for Computational Linguistics, Punta Cana, Dominican Republic, 1-13. https://doi.org/10.18653/v1/2021.mrqa-1.1.

[10] OpenAI. Introducing ChatGPT. OpenAI, November 30, 2022. Retrieved from: https://openai.com/index/chatgpt/

[11] Resnik, P. (1999). Semantic similarity in a taxonomy: an information-based measure and its application to problems of ambiguity in natural language. Journal of Artificial Intelligence Research, 11, 95-130. https://doi.org/10.1613/jair.514

[12] Harispe, S., Ranwez, S., Janaqi, S., & Montmain, J. (2015). Semantic Similarity from Natural Language and Ontology Analysis. Synthesis Lectures on Human Language Technologies Series, 8(1). Springer Cham. ISBN: 978-3-031-01028-6. https://doi.org/10.2200/S00639ED1V01Y201504HLT027

[13] Feng, Y., Bagheri, E., Ensan, F., & Jovanovic, J. (2017). The state of the art in semantic relatedness: a framework for comparison. Knowledge Engineering Review, 32(10), 1-30. https://doi.org/10.1017/S0269888917000029

[14] Slimani, T. (2013). Description and Evaluation of Semantic Similarity Measures Approaches. International Journal of Computer Applications, 80, 25-33. https://doi.org/10.5120/13897-1851

[15] Haque, S., Eberhart, Z., Bansal, A., & McMillan, C. (2022). Semantic similarity metrics for evaluating source code summarization. In Proceedings of the 30th IEEE/ACM International Conference on Program Comprehension (ICPC '22), Association for Computing Machinery, New York, NY, USA, 36–47. https://doi.org/10.1145/3524610.3527909

[16] Cer, D., Yang, Y., Kong, S., Hua, N., Limtiaco, N., John, R. St., Constant, N., Guajardo-Cespedes, M., Yuan, S., Tar, C., Strope, B., & Kurzweil, R. (2018). Universal Sentence Encoder for English. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, Association for Computational Linguistics, Brussels, Belgium, 169-174. https://doi.org/10.18653/v1/D18-2029

[17] Conneau, A., Kiela, D., Schwenk, H., Barrault, L., & Bordes, A. (2018). Supervised Learning of Universal Sentence Representations from Natural Language Inference Data. ArXiv preprint. https://doi.org/10.48550/arXiv.1705.02364

[18] Reimers, N., & Gurevych, I. (2019). Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 3973-3983. https://doi.org/10.18653/v1/D19-1410

[19] Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., & Artzi, Y. (2020). BERTScore: Evaluating Text Generation with BERT. International Conference on Learning Representations, 1-43. https://doi.org/10.48550/arXiv.1904.09675

[20] Fu, J., Ng, S.-K., Jiang, Z., & Liu, P. (2023). GPTScore: Evaluate as You Desire. ArXiv preprint. https://doi.org/10.48550/arXiv.2302.04166

[21] Manakul, P., Liusie, A., & Gales, M. J. F. (2023). SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, Singapore, 9004-9017. https://doi.org/10.18653/v1/2023.emnlp-main.557

[22] Liu, Y., Iter, D., Xu, Y., Wang, S., Xu, R., & Zhu, C. (2023). G-Eval: NLG Evaluation using Gpt-4 with Better Human Alignment. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, Singapore, 2511-2522. https://doi.org/10.18653/v1/2023.emnlp-main.153

[23] Dumais, S. T. (2004). Latent Semantic Analysis. Annual Review of Information Science and Technology, 38, 188–230. https://doi.org/10.1002/aris.1440380105

[24] Salton, G., & Buckley, C. (1988). Term-weighting approaches in automatic text retrieval. Information Processing and Management: an International Journal, 24(5), 513-523. https://doi.org/10.1016/0306-4573(88)90021-0

[25] Golub G. H., & Reinsch, C. (1970). Singular value decomposition and least squares solutions. Numerische Mathematik, 14(5), 403-420. https://doi.org/10.1007/BF02163027

[26] Hansen, P. C. (1987). The truncated SVD as a method for regularization. BIT Numerical Mathematics, 27(4), 534-553. https://doi.org/10.1007/BF01937276

[27] Wall, M. E., Rechtsteiner, A., & Rocha, L. (2003). Singular Value Decomposition and Principal Component Analysis. In A Practical Approach to Microarray Data Analysis, 5, 91-109. https://doi.org/10.1007/0-306-47815-3_5

[28] Ventures, LLC (2026). LLM API Pricing 2026 – Compare 300+ AI Model Costs. Price Per Token. https://pricepertoken.com/




DOI: http://dx.doi.org/10.30970/eli.34.3

Refbacks

  • There are currently no refbacks.