Online financial transactions have witnessed exponential growth in recent years, leading to a parallel rise in fraudulent activities across e-commerce and digital payment systems. To address this pressing issue, we propose a robust fraud detection framework that integrates machine learning and deep learning techniques, with a primary focus on Random Forest and ensemble-based architectures. Our approach includes comprehensive data preprocessing strategies such as label encoding, normalization, and handling class imbalance through the SMOTE technique. Furthermore, advanced feature extraction is performed using auto encoders and ResNeXt, followed by sequential learning with Gated Recurrent Units (GRUs) for temporal pattern recognition. The proposed model is evaluated using three benchmark datasets IEEE-CIS, PaySim, and the European card transaction dataset. Experimental results demonstrate that our method outperforms conventional models, achieving an accuracy of 96.0%, sensitivity of 99.8%, and specificity of 93.5%. The model not only enhances detection accuracy but also adapts effectively to evolving fraud patterns, making it suitable for real-time financial fraud prevention in diverse domains such as banking, e-commerce, and mobile transactions.