Cross-Validated Probabilistic Bayesian BERT with Semantic Embedding Fusion for Robust Text Classification

Nabil M. Abdel-Aziz, Faculty of Computers and Informatics, Zagazig University, Zagazig, 44519, EgyptFollow
Mahmoud Ibrahim, Faculty of Computers and Informatics, Zagazig University, Zagazig, 44519, EgyptFollow
Khalid A. Eldrandaly, Faculty of Computers and Informatics, Zagazig University, Zagazig, 44519, EgyptFollow

ORCID

Nabil M. Abdel-Aziz: https://orcid.org/0000-0001-6181-097X

Mahmoud Ibrahim: https://orcid.org/0009-0000-8242-6643

Khalid A. Eldrandaly: https://orcid.org/0000-0003-0802-5563

Keywords

NLP, Text classification, Transformers, Robustness, Cross validation

Article Type

Original Article

Abstract

Sentiment Analysis is one of the most prominent branches of natural language processing. It deals with text classification to identify the public emotions and opinions which help businesses and political institutions make strategic decisions. This study proposes a sentiment classification model by integrating the Bayesian inference into the BERT transformer model, augmented with pre-trained GloVe embeddings. The primary objective is to refine sentiment analysis performance on IMDB movie reviews dataset by leveraging BERT's contextualized embeddings and the semantic richness of GloVe vectors, while incorporating Bayesian inference for uncertainty quantification. Using both 5-fold cross-validation and solo training, the model's performance was evaluated and produced interesting findings in both cases. The model in solo training gets an accuracy of 88.22%, an F1 score of 0.8820, and an AUC of 0.9496. Further evaluation through 5-fold cross-validation further validated the model's performance. The results indicated consistent improvement in performance across epochs, with Fold 5 reaching near-perfect classification performance (99.82% accuracy, 0.9989 AUC). This highlights the robustness of the model, as it achieved high performance across different dataset splits. The mean AUC value across all folds remained consistently high, exceeding 0.95. These results validate the efficient application of probabilistic frameworks and transformer-based models for pragmatic sentiment classification challenges in many different sectors.

How to Cite

Abdel-Aziz, Nabil M.; Ibrahim, Mahmoud; and Eldrandaly, Khalid A. (2025) "Cross-Validated Probabilistic Bayesian BERT with Semantic Embedding Fusion for Robust Text Classification," Sustainable Machine Intelligence Journal: Vol. 13: Iss. 1, Article 1.
DOI: https://doi.org/10.63689/3005-3617.1072