A Multimodal Attention-Based Fusion Framework with Ensemble Learning for Robust Differentiation of Epileptic and Psychogenic Seizures
Keywords:
Multimodal Fusion; Attention Mechanism; Epileptic Seizures; Psychogenic Non-Epileptic Seizures (PNES); Electroencephalography (EEG); Video Semiology; Physiological Signals; Ensemble Learning; Seizure Classification; Clinical Decision SupportAbstract
Differentiating epileptic seizures from psychogenic non-epileptic seizures remains a significant clinical challenge, often leading to misdiagnosis and inappropriate treatment. We propose a multimodal attention-based fusion framework that integrates electroencephalography signals, video streams, and peripheral physiological data to address this problem. The system first extracts modality-specific features: time-frequency and nonlinear dynamical descriptors from EEG via wavelet decomposition and adaptive principal component analysis, spatiotemporal behavioral embeddings from video using a 3D ResNet, and autonomic markers from physiological signals encoded by a bidirectional long short-term memory network. A self-attention mechanism then computes context-aware weights for each modality, dynamically suppressing contributions from noisy or uninformative channels before constructing a fused representation. This fused vector is subsequently classified by a heterogeneous ensemble of a support vector machine, a random forest, and a shallow multi-layer perceptron, whose outputs are aggregated through soft voting. The key novelty lies in the attention-based fusion layer, which replaces conventional early or late fusion strategies and enables the model to adaptively prioritize diagnostically relevant modalities on a per-instance basis. Furthermore, the ensemble classifier provides robust decision boundaries even when single-modality cues are ambiguous. We expect this framework to improve diagnostic accuracy in clinical settings where video quality may be compromised or EEG artifacts are prevalent. The proposed method therefore offers a principled approach to integrating heterogeneous seizure-related signals, with potential to reduce misdiagnosis rates and guide appropriate therapeutic interventions.
References
[1] E. Gedzelman and S. LaRoche, “Long-term video EEG monitoring for diagnosis of psychogenic nonepileptic seizures,” Neuropsychiatric Disease and Treatment, 2014.
[2] U. Raghavendra, U. Acharya, and H. Adeli, “Artificial intelligence techniques for automated diagnosis of neurological disorders,” European neurology, 2020.
[3] M. Pedersen, K. Verspoor, M. Jenkinson, et al., “Artificial intelligence for clinical decision support in neurology,” Brain Communications, 2020.
[4] M. Omidvar, A. Zahedi, and H. Bakhshi, “EEG signal processing for epilepsy seizure detection using 5-level Db4 discrete wavelet transform, GA-based feature selection and ANN/SVM classifiers,” Journal of Ambient Intelligence and Humanized Computing, 2021.
[5] S. Tan et al., “Automatic detection and prediction of epileptic EEG signals based on nonlinear dynamics and deep learning: A review,” Frontiers in Neuroscience, 2025.
[6] A. Subasi and M. Gursoy, “EEG signal classification using PCA, ICA, LDA and support vector machines,” Expert systems with applications, 2010.
[7] I. Wijayanto, S. Rizal, and S. Hadiyoso, “Epileptic electroencephalogram signal classification using wavelet energy and random forest,” in AIP conference proceedings, 2023.
[8] U. Acharya, S. Oh, Y. Hagiwara, J. Tan, et al., “Deep convolutional neural network for the automated detection and diagnosis of seizure using EEG signals,” Computers in Biology and Medicine, 2018.
[9] D. Pathak and R. Kashyap, “Neural correlate-based e-learning validation and classification using convolutional and long short-term memory networks,” Traitement du Signal, 2023.
[10] D. Ahmedt-Aristizabal, C. Fookes, S. Dionisio, et al., “Automated analysis of seizure semiology and brain electrical activity in presurgery evaluation of epilepsy: A focused survey,” Epilepsia, 2017.
[11] D. Ahmedt-Aristizabal, K. Nguyen, et al., “Deep motion analysis for epileptic seizure classification,” in Annual international conference of the IEEE engineering in medicine and biology society, 2018.
[12] F. Mason, A. Scarabello, L. Taruffi, E. Pasini, et al., “Heart rate variability as a tool for seizure prediction: A scoping review,” Journal of Clinical Medicine, 2024.
[13] Y. Nagai, C. Jones, and A. Sen, “Galvanic skin response (GSR)/electrodermal/skin conductance biofeedback on epilepsy: A systematic review and meta-analysis,” Frontiers in neurology, 2019.
[14] R. Pillalamarri and U. Shanmugam, “A review on EEG-based multimodal learning for emotion recognition,” Artificial Intelligence Review, 2025.
[15] A. Vaswani, N. Shazeer, N. Parmar, et al., “Attention is all you need,” in Advances in neural information processing systems, 2017.
[16] J. Hu, L. Shen, and G. Sun, “Squeeze-and-excitation networks,” in 2018 IEEE/CVF conference on computer vision and pattern recognition, 2018.
[17] W. Hou, J. Wang, Q. Lin, X. Wang, et al., “Improving clinical foundation models with multi-modal learning and domain adaptation for chronic disease prediction,” IEEE Journal of Biomedical and Health Informatics, 2025.
[18] N. Liu, X. Li, E. Qi, M. Xu, L. Li, and B. Gao, “A novel ensemble learning paradigm for medical diagnosis with imbalanced data,” IEEE Access, 2020.
[19] V. Dharani and L. Lakshmanan, “An enhanced machine learning-based multimodal framework for seizure detection using EEG and MRI data,” Developmental Neurobiology, 2026.
[20] V. Shah, E. V. Weltin, S. Lopez, J. McHugh, et al., “The temple university hospital seizure detection corpus,” Frontiers in Neuroinformatics, 2018.
[21] M. B. Mbarek, I. Assali, S. Hamdi, et al., “Automatic and manual prediction of epileptic seizures based on ECG,” Signal, Image and Video Processing, 2024.
[22] S. Vieluf, M. Amengual-Gual, B. Zhang, R. E. Atrache, et al., “Twenty-four-hour patterns in electrodermal activity recordings of patients with and without epileptic seizures,” Epilepsia, 2021.
[23] S. Stahlschmidt, B. Ulfenborg, et al., “Multimodal deep learning for biomedical data fusion: A review,” Briefings in Bioinformatics, 2022.
[24] V. K. Tiwari, S. Bajpai, and J. Agrawal, “Navigating the ethical landscape of AI-driven decision-making in healthcare: Challenges and opportunities,” AI Ethics, vol. 6, art. no. 216, Mar. 2026, doi: 10.1007/s43681-026-01077-4.
[25] S. Tiwari, V. K. Tiwari, M. Khemariya, and V. Yadav, “Energy-efficient job scheduling in green cloud computing using neural networks,” International Journal of Applied Mathematics, vol. 38, no. 4S, pp. 533–548, Sep. 2025, doi: 10.12732/ijam.v38i4s.1.
[26] S. Lenka, S. Bajpai, V. K. Tiwari, and K. Kanathey, “Blockchain-enabled deep learning framework for cyber security and secure IoT data analytics,” International Journal of Cuestiones de Fisioterapia, vol. 53, no. 3, pp. 5407–5419, Aug. 2024, doi: 10.48047/rw0z6h06.
[27] M. K. Bagwani, S. Dwivedi, V. Kumar, and P. Koshti, "Transmitting malware through QR codes: Risk analysis and a hybrid detection method," International Journal for Multidisciplinary Research, vol. 8, no. 3, pp. 1-25, 2026.
[28] A. M. K. Bagwani and V. K. Tiwari, "Preventing malware spread through QR codes: A detection and analysis approach," Journal of Engineering and Technology Management, vol. 73, pp. 1260-1268, 2024.
[29] A. M. K. Bagwani, V. K. Tiwari, and N. Singh, "Integrating GrapesJS with AWS: Building an educational platform for web development training," An Overview of Literature, Language and Education Research, vol. 10, pp. 106-124, 2025.
[30] M. Bagwani, "Building a cloud-native microservices based web application with GraphQL and Docker," School of Advanced Computing, Sanjeev Agrawal Global Educational University, 2024.
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Priyanka Singh
This work is licensed under a Creative Commons Attribution 4.0 International License.
Articles published in the International Journal of Artificial Intelligence, Cybersecurity and Emerging Technologies (IJAICET) are licensed under the Creative Commons Attribution 4.0 International License (CC BY 4.0).
Under this license, users are free to share, copy, redistribute, adapt, and build upon the published material for any purpose, including commercial use, provided appropriate credit is given to the original author(s) and the source, a link to the license is provided, and any changes made are indicated.
Authors retain copyright of their work while granting IJAICET the right to publish, archive, distribute, and index the article. The journal supports the free exchange of scientific knowledge through open access publishing and encourages the widest possible dissemination of research.
