Abstract

The rapid growth of digital payment systems and e-commerce platforms has significantly increased the volume of credit card transactions worldwide, consequently raising the risk of fraudulent financial activities. Detecting fraudulent transactions remains a challenging problem due to the highly imbalanced nature of transaction datasets and the constantly evolving behavior of fraud patterns. Traditional rule-based detection systems and single machine learning models often struggle to identify both known and previously unseen fraud patterns effectively. To address these challenges, this study proposes a hybrid credit card fraud detection framework that integrates supervised and unsupervised machine learning approaches. XGBoost is employed as the primary supervised learning model to identify known fraud patterns, while an autoencoder-based anomaly detection model identifies unusual transaction behaviors by analyzing reconstruction errors. The outputs are combined using a logistic regression-based meta-classifier. The proposed hybrid system was evaluated using the BankSim transaction dataset. Experimental results demonstrate that the standalone XGBoost model achieved an Average Precision (AP) of 0.79, an F1-score of 0.719, precision of 0.743, and recall of 0.697. The final hybrid meta-classifier model achieved an AP of 0.724, F1-score of 0.703, precision of 0.689, and recall of 0.718, indicating improved recall stability and robustness in detecting fraudulent transactions.

Graphical Abstract

Novelty Statement

An empirical and systematic analysis of supervised, unsupervised, and meta-learning-based hybrid detection approaches to fraud detection, where practical trade-offs for data calibration for imbalanced data are presented.