Emotion Recognition Based On Electroencephalogram Signals Using Deep Learning Network

Bin Wu

doi:10.6180/jase.202401_27(1).0014

Emotion Recognition Based On Electroencephalogram Signals Using Deep Learning Network

Computer Science and Information Engineering

Bin Wu

School of Computer and Artificial Intelligence, Chaohu University, Chaohu 238000, China

Received: March 26, 2022
Accepted: May 6, 2023
Publication Date: June 13, 2023

Copyright The Author(s). This is an open access article distributed under the terms of the Creative Commons Attribution License (CC BY 4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are cited.

Download Citation: ||https://doi.org/10.6180/jase.202401_27(1).0014

Deep learning networks have a high calculation volume, which is one of their problems. To solve this defect, the data of intrinsic modes obtained from the application of empirical mode decomposition to Electroencephalograph signals were used for the first time in this study. The present paper presents a method for emotion recognition using a deep learning network and electroencephalogram signal. Based on the non-stationary nature of the electroencephalogram, the intrinsic mode functions are extracted using empirical mode decomposition before selecting the first three intrinsic mode functions. Then, electrode positions are converted into pixel positions in images using suitable mapping, and the extracted features are interpreted as pixel color components. Using a deep learning network, all generated images are input into the network to determine whether they belong to the high or low valence class. Similarly, the class of arousal has been determined using the same method. This method was evaluated using the DEAP database to assess its efficiency. The results show that by selecting the image with the size of 17 × 17, the proposed method can detect valence and arousal emotions with an accuracy of 82.3% and 78.4%, respectively, which is an acceptable superiority compared to previous research.

Keywords: Deep learning network, electroencephalogram signal, intrinsic mode functions, empirical mode decomposition, emotion recognition.

[1] S. Karim, A. Qadir, U. Farooq, M. Shakir, and A. Laghari, (2022) “Hyperspectral Imaging: A Review and Trends towards Medical Imaging" Current Medical Imaging:
[2] E. Eslami, (2022) “Analytical Study of Deep Learning Methods for Road Condition Assessment":
[3] E. Eslami and H.-B. Yun, (2022) “Improvement of Multiclass Classification of Pavement Objects Using Intensity and Range Images" Journal of Advanced Transportation:
[4] E. Eslami and H.-B. Yun, (2021) “Attention-based multiscale convolutional neural network (A+ MCNN) for multi-class classification in road images" Sensors 21(15):5137.
[5] L. Wang, Y. Shoulin, H. Alyami, A. Laghari, M. Rashid, J. Almotiri, H. Alyamani, and F. Alturise, (2022) “A novel deep learning-based single shot multibox detector model for object detection in optical remote sensing images":
[6] A. Laghari, H. He, M. Shafiq, and A. Khan, (2018) “Assessment of quality of experience (QoE) of image compression in social cloud computing" Multiagent and Grid Systems 14(2): 125–143.
[7] A. Laghari and S. Yin, (2022) “How to Collect and Interpret Medical Pictures Captured in Highly Challenging Environments that Range from Nanoscale to Hyperspectral Imaging" Current Medical Imaging:
[8] Z.-M. Wang, J.-W. Zhang, Y. He, and J. Zhang, (2022) “EEG emotion recognition using multichannel weighted multiscale permutation entropy" Applied Intelligence 52(10): 12064–12076.
[9] N. S. Suhaimi, J. Mountstephens, and J. Teo, (2022) “A dataset for emotion recognition using virtual reality and EEG (DER-VREEG): emotional state classification using low-cost wearable VR-EEG headsets" Big Data and Cognitive Computing 6(1): 16.
[10] M. M. Hassan, K. T. Hasan, I. A. Tonoy, M. M. Rahman, and M. A. B. Abedin. “Student Attention Base Facial Emotion State Recognition Using Convolutional Neural Network”. In: Proceedings of International Conference on Data Science and Applications:
ICDSA 2022, Volume 1. Springer, 2023.
[11] G. Li, D. Ouyang, Y. Yuan, W. Li, Z. Guo, X. Qu, and P. Green, (2022) “An EEG data processing approach for emotion recognition" IEEE Sensors Journal 22(11): 10751–10763.
[12] T. Tuncer, S. Dogan, M. Baygin, and U. R. Acharya,(2022) “Tetromino pattern based accurate EEG emotion classification model" Artificial Intelligence in Medicine 123: 102210.
[13] X. Li, Y. Zhang, P. Tiwari, D. Song, B. Hu, M. Yang, Z. Zhao, N. Kumar, and P. Marttinen, (2022) “EEG based emotion recognition: A tutorial and review" ACM Computing Surveys 55(4): 1–57.
[14] Z. Xie, J. Pan, S. Li, J. Ren, S. Qian, Y. Ye, and W. Bao, (2022) “Musical Emotions Recognition Using Entropy Features and Channel Optimization Based on EEG" Entropy 24(12): 1735.
[15] M. Chernykh, B. Vodianyk, I. Seleznov, D. Harmatiuk, I. Zyma, A. Popov, and K. Kiyono, (2022)“Detrending Moving Average, Power Spectral Density, and Coherence: Three EEG-Based Methods to Assess Emotion Irradiation during Facial Perception" Applied Sciences 12(15): 7849.
[16] O. Almanza-Conejo, D. L. Almanza-Ojeda, J. L. Contreras-Hernandez, and M. A. Ibarra-Manzano, (2023) “Emotion recognition in EEG signals using the continuous wavelet transform and CNNs" Neural Computing and Applications 35(2): 1409–1422.
[17] F. Zheng, B. Hu, X. Zheng, C. Ji, J. Bian, and X. Yu, (2022) “Dynamic differential entropy and brain connectivity features based EEG emotion recognition" International Journal of Intelligent Systems:
[18] V. Kanuboyina, T. Shankar, and R. Penmetsa, (2022) “Electroencephalography based human emotion state classification using principal component analysis and artificial neural network" Multiagent and Grid Systems 18(3-4): 263–278.
[19] Q. Li, Y. Liu, Y. Shang, Q. Zhang, and F. Yan, (2022) “Deep Sparse Autoencoder and Recursive Neural Network for EEG Emotion Recognition" Entropy 24(9): 1187.
[20] X. Li, D. Song, P. Zhang, G. Yu, Y. Hou, and B. Hu. “Emotion recognition from multichannel EEG data through convolutional recurrent neural network”. In: 2016 IEEE International Conference on Bioinformatics and Biomedicine (BIBM). IEEE. 2016, 352–357.
[21] M. Yu, S. Xiao, M. Hua, H. Wang, X. Chen, F. Tian, and Y. Li, (2022) “EEG-based emotion recognition in an immersive virtual reality environment: From local activity to brain network features" Biomedical Signal Processingand Control 72: 103349.
[22] A. Topic, M. Russo, M. Stella, and M. Saric, (2022) “Emotion Recognition Using a Reduced Set of EEG Channels Based on Holographic Feature Maps" Sensors (Basel) 22(9): 3248.
[23] M. Malathi, G. A. A. Mary, J. Kumar, P. Sinthia, and M. Nalini. “An estimation of PCA feature extraction in EEG-based emotion prediction with support vector machines”. In: Proceedings of Data Analytics and Management: ICDAM 2021, Volume 1. Springer. 2022,
1–7.
[24] A. D.Wibawa, B. S. Y. Mohammad, M. A. K. Fata, F. A. Nuraini, A. Prasetyo, and Y. Pamungkas. “Comparisonof EEG-Based Biometrics System Using Naive Bayes, Neural Network, and Support Vector Machine”. In: 2022 International Conference on Electrical and Information Technology (IEIT). IEEE. 2022, 1–6.
[25] Y. Wang, S. Qiu, D. Li, C. Du, B.-L. Lu, and H. He, (2022) “Multi-modal domain adaptation variational autoencoder for eeg-based emotion recognition" IEEE/CAA Journal of Automatica Sinica 9(9): 1612–1626.
[26] Ł. Furman,W. Duch, L. Minati, and K. Tołpa, (2023) “Short-time Fourier transform and embedding method for recurrence quantification analysis of EEG time series" The European Physical Journal Special Topics 232(1): 135–149.
[27] E. Hancer and A. Subasi, (2022) “EEG-based emotion recognition using dual tree complex wavelet transform and random subspace ensemble classifier" Computer Methods in Biomechanics and Biomedical Engineering: 1–13.
[28] B.-D. Barkana, Y. Ozkan, and J. A. Badara, (2022) “Analysis of working memory from EEG signals under different emotional states" Biomedical Signal Processing and Control 71: 103249.
[29] S. Karim, Y. Zhang, S. Yin, A. A. Laghari, and A. A. Brohi, (2019) “Impact of compressed and down-scaledtraining images on vehicle detection in remote sensing imagery" Multimedia Tools and Applications 78: 32565–32583.
[30] S. Karim, Y. Zhang, A. A. Laghari, and A. Memon. “Image processing based proposed drone for detecting and controlling street crimes.” In: 2017 IEEE International Conference on Communication Technology
(ICCT). IEEE. 2017, 1–5.
[31] S. Karim, H. He, A. A. Laghari, A. H. Magsi, and R. A. Laghari, (2021) “Quality of service (QoS): measurements of image formats in social cloud computing" Multimedia Tools and Applications 80(23): 4507–4532.
[32] M. Norouzi, M. Hashemi, and Z. Pouri, (2021) “The Question of Global Society in Post-Corona Time: Towards a Paradigm Shift" International Journal of CommunityWell-Being 4(3): 339–343.
[33] N. Farsaeivahid, C. Grenier, S. Nazarian, and M. L. Wang, (2022) “A Rapid Label-Free Disposable Electrochemical Salivary Point-of-Care Sensor for SARS-CoV-2 Detection and Quantification" Sensors 23(1): 433.
[34] F. Salahshour, M.-M. Mehrabinejad, M. N. Toosi, M. Gity, H. Ghanaati, M. Shakiba, S. Nosrat Sheybani, H. Komaki, and S. Kolahi, (2021) “Clinical and chest CT features as a predictive tool for COVID-19 clinical progress: introducing a novel semi-quantitative scoring
system" European radiology 31: 5178–5188.
[35] R. Rezapour-Nasrabad, (2021) “Feasibility of providing Namaste managed care to the elderly with Alzheimer’s disease" Archivos Venezolanos de Farmacologia y Terapéutica 40(4): 455˘463.
[36] F. Salahshour, M.-M. Mehrabinejad, A. Zare Dehnavi, A. Alibakhshi, H. Dashti, M.-A. Ataee, and N. Ayoobi Yazdi, (2020) “Pancreatic neuroendocrine tumors (pNETs): the predictive value of MDCT characteristics in the differentiation of histopathological grades" Abdominal
Radiology 45: 3155–3162.
[37] B. Maas, E. Zabeh, and S. Arabshahi. “QuickTumor-Net: fast automatic multi-class segmentation of brain tumors”. In: 2021 10th International IEEE/EMBS Conference on Neural Engineering (NER). IEEE. 2021, 1–4.
[38] E. Ghafourian, F. Samadifam, H. Fadavian, P. Jerfi Canatalay, A. Tajally, and S. Channumsin, (2023) “An Ensemble Model for the Diagnosis of Brain Tumors through MRIs" Diagnostics 13(3): 561.
[39] A. Aggarwal, A. Srivastava, A. Agarwal, N. Chahal, D. Singh, A. A. Alnuaim, A. Alhadlaq, and H.-N. Lee, (2022) “Two-way feature extraction for speech emotion recognition using deep learning" Sensors 22(6): 2378.
[40] A. Mert and A. Akan, (2018) “Emotion recognition from EEG signals by using multivariate empirical mode decomposition" Pattern Analysis and Applications 21: 81–89.
[41] M. Chen, J. Han, L. Guo, J.Wang, and I. Patras. “Identifying valence and arousal levels via connectivity between EEG channels”. In: 2015 International Conference on Affective Computing and Intelligent Interaction (ACII). IEEE. 2015.
[42] S. Jirayucharoensak, S. Pan-Ngum, and P. Israsena, (2014) “EEG-based emotion recognition using deep learning network with principal component based covariate shift adaptation" The ScientificWorld Journal 2014:
[43] F. Bahari and A. Janghorbani. “EEG-based emotion recognition using recurrence plot analysis and k nearest neighbor classifier”. In: 2013 20th Iranian Conference on Biomedical Engineering (ICBME). IEEE. 2013.
[44] P. Arnau-González, M. Arevalillo-Herráez, and N. Ramzan, (2017) “Fusing highly dimensional energy and connectivity features to identify affective states from EEG signals" Neurocomputing 244: 81–89.
[45] Z. Liang, S. Oba, and S. Ishii, (2019) “An unsupervised EEG decoding system for human emotion recognition" Neural Networks 116: 257–268.
[46] A. Olamat, P. Ozel, and S. Atasever, (2022) “Deep learning methods for multi-channel EEG-based emotion recognition" International Journal of Neural Systems 32(05): 2250021.
[47] H. Cizmeci and C. Ozcan, (2023) “Enhanced deep capsule network for EEG-based emotion recognition" Signal, Image and Video Processing 17(2): 463–469.
[48] S. Koelstra, C. Muhl, M. Soleymani, J.-S. Lee, A. Yazdani, T. Ebrahimi, T. Pun, A. Nijholt, and I. Patras, (2011) “DEAP: A database for emotion analysis; using physiological signals" IEEE transactions on affective computing 3(1): 18–31.
[49] F. Sharbrough, (1991) “American Electroencephalographic Society guidelines for standard electrode position nomenclature" J clin Neurophysiol 8: 200–202.
[50] R. Oostenveld and P. Praamstra, (2001) “The five percent electrode system for high-resolution EEG and ERP measurements" Clinical neurophysiology 112(4): 713–719.
[51] D. Slayback, S. Abdali, J. Brooks,W. D. Hairston, and P. Groves. “Novel methods for eeg visualization and visualization”. In: 2018 IEEE International Symposium on Circuits and Systems (ISCAS). IEEE. 2018, 1–5.
[52] M. D. Zeiler and R. Fergus. “Visualizing and understanding convolutional networks”. In: Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part I. 13. Springer. 2014.
[53] S. Wager, S. Wang, and P. S. Liang, (2013) “Dropout training as adaptive regularization" Advances in neural information processing systems 26: 45–67.
[54] X. Glorot and Y. Bengio. “Understanding the difficulty of training deep feedforward neural networks”. In: Proceedings of the thirteenth international conference on artificial intelligence and statistics. 9. JMLR Workshop and Conference Proceedings. 2010, 249–256.
[55] D. Naser and G. Saha. “Recognition of emotions induced by music videos using DT-CWPT”. In: 2013 Indian Conference on Medical Informatics and Telemedicine (ICMIT). IEEE. 2013, 1–5