A Deep Learning Model Based On Multi-granularity Facial Features And LSTM Network For Driver Drowsiness Detection

Taiguo  Li; Chao  Li

doi:10.6180/jase.202407_27(7).0011

A Deep Learning Model Based On Multi-granularity Facial Features And LSTM Network For Driver Drowsiness Detection

Computer Science and Information Engineering

Taiguo Li¹This email address is being protected from spambots. You need JavaScript enabled to view it. and Chao Li²

¹School of Automation and Electrical Engineering, Lanzhou Jiaotong University, Lanzhou 730070, China

²Department I of Kinesitherapy, Shaanxi Kangfu Hospital, Xi’an 710065, China

Received: August 22, 2023
Accepted: October 9, 2023
Publication Date: November 5, 2023

Copyright The Author(s). This is an open access article distributed under the terms of the Creative Commons Attribution License (CC BY 4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are cited.

Download Citation: ||https://doi.org/10.6180/jase.202407_27(7).0011

Driver drowsiness can cause serious harm to drivers and other road participants. Exploring objective and efficient methods for detecting driver drowsiness has important application value for ensuring road safety. Considering the information complementary between local and global facial features for drowsiness detection, as well as the advantages of deep learning models in information mining, this paper proposes a deep learning model based on multi-granularity facial features and Long Short Term Memory (LSTM) network for driver drowsiness detection. To obtain local facial feature information, face detection and facial landmarks location are implemented based on Practical Facial Landmark Detector (PFLD). The local representation features of the eyes and mouth, as well as the head pose feature, are calculated from the coordinate information of facial landmarks. Furthermore, a global representation learning Vision Transformer (ViT) model that trained on the NTHU-DDD dataset to obtain higher-level semantic information. Due to drowsiness has an accumulative property, an LSTM network that takes the local and global multi-granularity representation features as input to further mine the drowsy clues in the temporal dimension. A large number of comparative experiments are conducted on the public NTHU-DDD dataset, and the results show that the proposed method outperformed other methods, achieving a detection accuracy of 93.15%. Experimental results show that the method can achieve much higher accuracy and can provide an alternative solution for the driver assistance system.

Keywords: Driver Drowsiness Detection, Deep Learning Model, Multi-granularity Representation Features, Vision Transformer, Long Short Term Memory Network

[1] M. Doudou, A. Bouabdallah, and V. Berge-Cherfaoui, (2020) “Driver drowsiness measurement technologies: Current research, market solutions, and challenges" International Journal of Intelligent Transportation Systems Research 18: 297–319. DOI: 10.1007/s13177-019-00199-w.
[2] Drowsy Driving NHTSA Reports. Accessed: March 2020. https: //www.nhtsa.gov/risky-driving/drowsydriving.
[3] S. Kaplan, M. A. Guvensan, A. G. Yavuz, and Y. Karalurt, (2015) “Driver behavior analysis for safe driving: A survey" IEEE Transactions on Intelligent Transportation Systems 16(6): 3017–3032. DOI: 10.1109/TITS.2015.2462084.
[4] Y. Albadawi, M. Takruri, and M. Awad, (2022) “A review of recent developments in driver drowsiness detection systems" Sensors 22(5): 2069. DOI: 10.3390/s22052069.
[5] T. Åkerstedt and M. Gillberg, (1990) “Subjective and objective sleepiness in the active individual" International journal of neuroscience 52(1-2): 29–37. DOI: 10.3109/00207459008994241.
[6] I. Stancin, M. Cifrek, and A. Jovic, (2021) “A review of EEG signal features and their application in driver drowsiness detection systems" Sensors 21(11): 3786. DOI: 10.3390/s21113786.
[7] S. Soares, S. Ferreira, and A. Couto, (2020) “Driving simulator experiments to study drowsiness: A systematic review" Traffic injury prevention 21(1): 29–37. DOI: 10.1080/15389588.2019.1706088.
[8] T. Li, T. Zhang, Y. Zhang, and L. Yang, (2022) “Driver fatigue detection method based on human pose information entropy" Journal of advanced transportation 2022: DOI: 10.1155/2022/7213841.
[9] K. Fujiwara, E. Abe, K. Kamata, C. Nakayama, Y. Suzuki, T. Yamakawa, T. Hiraoka, M. Kano, Y. Sumi, F. Masuda, et al., (2018) “Heart rate variability-based driver drowsiness detection and its validation with EEG" IEEE transactions on biomedical engineering 66(6): 1769–1778. DOI: 10.1109/TBME.2018.2879346.
[10] Y. Jiang, Y. Zhang, C. Lin, D. Wu, and C.-T. Lin, (2020) “EEG-based driver drowsiness estimation using an online multi-view and transfer TSK fuzzy system" IEEE Transactions on Intelligent Transportation Systems 22(3): 1752–1764. DOI: 10.1109/TITS.2020.2973673.
[11] K. T. Chui, K. F. Tsang, H. R. Chi, B. W. K. Ling, and C. K. Wu, (2016) “An accurate ECG-based transportation safety drowsiness detection scheme" IEEE Transactions on Industrial Informatics 12(4): 1438–1452. DOI: 10.1109/TII.2016.2573259.
[12] F. Wang, Q. Xu, and R. Fu, (2019) “Study on the effect of man-machine response mode to relieve driving fatigue based on EEG and EOG" Sensors 19(22): 4883. DOI: 10.3390/s19224883.
[13] C. Caponecchia and A. Williamson, (2018) “Drowsiness and driving performance on commuter trips" Journal of safety research 66: 179–186. DOI: 10.1016/j.jsr.2018.07.003.
[14] J. Wörle, B. Metz, M. B. Steinborn, L. Huestegge, and M. Baumann, (2021) “Differential effects of driver sleepiness and sleep inertia on driving behavior" Transportation research part F: traffic psychology and behaviour 82: 111–120. DOI: 10.1016/j.trf.2021.08.001.
[15] M. Shahverdy, M. Fathy, R. Berangi, and M. Sabokrou, (2020) “Driver behavior detection and classification using deep convolutional neural networks" Expert Systems with Applications 149: 113240. DOI: 10.1016/j.eswa.2020.113240.
[16] R. Huang, Y. Wang, Z. Li, Z. Lei, and Y. Xu, (2020) “RF-DCM: multi-granularity deep convolutional model based on feature recalibration and fusion for driver fatigue detection" IEEE Transactions on Intelligent Transportation Systems 23(1): 630–640. DOI: 10.1109/TITS. 2020.3017513.
[17] B. Mandal, L. Li, G. S. Wang, and J. Lin, (2016) “Towards detection of bus driver fatigue based on robust visual analysis of eye state" IEEE Transactions on Intelligent Transportation Systems 18(3): 545–557. DOI: 10.1109/TITS.2016.2582900.
[18] S. Bakheet and A. Al-Hamadi, (2021) “A framework for instantaneous driver drowsiness detection based on improved HOG features and naıve Bayesian classification" Brain Sciences 11(2): 240. DOI: 10.3390/brainsci11020240.
[19] S. Jamshidi, R. Azmi, M. Sharghi, and M. Soryani, (2021) “Hierarchical deep neural networks to detect driver drowsiness" Multimedia Tools and Applications 80: 16045–16058. DOI: 10.1007/s11042-021-10542-7.
[20] L. Zhang, F. Liu, and J. Tang, (2015) “Real-time system for driver fatigue detection by RGB-D camera" ACM Transactions on Intelligent Systems and Technology (TIST) 6(2): 1–17. DOI: 10.1145/2629482.
[21] S. Mehta, S. Dadhich, S. Gumber, and A. Jadhav Bhatt. “Real-time driver drowsiness detection system using eye aspect ratio and eye closure ratio”. In: Proceedings of international conference on sustainable computing in science, technology and management (SUSCOM), Amity University Rajasthan, Jaipur-India. 2019. DOI: 10.2139/ssrn.3356401.
[22] M. H. Baccour, F. Driewer, E. Kasneci, and W. Rosenstiel. “Camera-based eye blink detection algorithm for assessing driver drowsiness”. In: 2019 IEEE Intelligent Vehicles Symposium (IV). IEEE. 2019, 987–993. DOI: 10.1109/IVS.2019.8813871.
[23] M. Dreißig, M. H. Baccour, T. Schäck, and E. Kasneci. “Driver drowsiness classification based on eye blink and head movement features using the k-NN algorithm”. In: 2020 IEEE Symposium Series on Computational Intelligence (SSCI). IEEE. 2020, 889–896. DOI: 10.1109/SSCI47803.2020.9308133.
[24] S. Yin, L. Wang, M. Shafiq, L. Teng, A. A. Laghari, and M. F. Khan, (2023) “G2Grad-CAMRL: An Object Detection and Interpretation Model Based on Gradientweighted Class Activation Mapping and Reinforcement Learning in Remote Sensing Images" IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing: DOI: 10.1109/JSTARS.2023. 3241405.
[25] T. Zhu, C. Zhang, T. Wu, Z. Ouyang, H. Li, X. Na, J. Liang, and W. Li, (2022) “Research on a real-time driver fatigue detection algorithm based on facial video sequences" Applied Sciences 12(4): 2224. DOI: 10.3390/app12042224.
[26] B. K. Sava¸s and Y. Becerikli, (2020) “Real time driver fatigue detection system based on multi-task ConNN" Ieee Access 8: 12491–12498. DOI: 10.1109/ACCESS.2020.2963960.
[27] H. Yang, L. Liu, W. Min, X. Yang, and X. Xiong, (2020) “Driver yawning detection based on subtle facial action recognition" IEEE Transactions on Multimedia 23: 572–583. DOI: 10.1109/TMM.2020.2985536.
[28] M.-H. Sigari, M. Fathy, and M. Soryani, (2013) “A driver face monitoring system for fatigue and distraction detection" International journal of vehicular technology 2013: 1–11. DOI: 10.1155/2013/263983.
[29] M.-H. Sigari, M.-R. Pourshahabi, M. Soryani, and M. Fathy, (2014) “A review on driver face monitoring systems for fatigue and distraction detection" International Journal of Advanced Science and Technology 64: 73–100.
[30] M. Ye, W. Zhang, P. Cao, and K. Liu, (2021) “Driver fatigue detection based on residual channel attention network and head pose estimation" Applied Sciences 11(19): 9195. DOI: 10.3390/app11199195.
[31] T. Soukupova and J. Cech. “Eye blink detection using facial landmarks”. In: 21st computer vision winter workshop, Rimske Toplice, Slovenia. 2016, 2.
[32] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al., (2020) “An image is worth 16x16 words: Transformers for image recognition at scale" arXiv preprint arXiv:2010.11929:
[33] C.-H. Weng, Y.-H. Lai, and S.-H. Lai. “Driver drowsiness detection via a hierarchical temporal deep belief network”. In: Computer Vision–ACCV 2016 Workshops: ACCV 2016 International Workshops, Taipei, Taiwan, November 20-24, 2016, Revised Selected Papers, Part III 13. Springer. 2017, 117–133. DOI: 10.1007/978-3-319-54526-4_9.
[34] S. Park, F. Pan, S. Kang, and C. D. Yoo. “Driver drowsiness detection system based on feature representation learning using various deep networks”. In: Asian Conference on Computer Vision. Springer. 2016, 154–164. DOI: 10.1007/978-3-319-54526-4_12.
[35] L. Celona, L. Mammana, S. Bianco, and R. Schettini. “A multi-task CNN framework for driver face monitoring”. In: 2018 IEEE 8th International Conference on Consumer Electronics-Berlin (ICCE-Berlin). IEEE. 2018, 1–4. DOI: 10.1109/ICCE-Berlin.2018.8576244.
[36] M. Dua, Shakshi, R. Singla, S. Raj, and A. Jangra, (2021) “Deep CNN models-based ensemble approach to driver drowsiness detection" Neural Computing and Applications 33: 3155–3168. DOI: 10.1007/s00521-020-05209-7.
[37] G. Zhao, Y. He, H. Yang, and Y. Tao, (2022) “Research on fatigue detection based on visual features" IET Image Processing 16(4): 1044–1053. DOI: 10.1049/ipr2.12207.