Officical thesis title: Research and Development of Deep Learning Models for Emotion Recognition from EEG Signals under Resource Constraints
Full name: Duong Thi Mai Thuong
Major: Computer Science Code: 9 48 01 01
Supervisors: Assoc. Prof. Dr. Phung Trung Nghia
Summary of the new findings of the thesis
- Research objectives:
- Update the state of the art on the EEG_ER problem; study the key techniques to be implemented in data acquisition, EEG feature extraction, and emotion classification; review the most recent studies that apply deep learning to EEG_ER; and clearly identify the challenges to be overcome when building an EEG_ER system oriented toward AIoT (deployment on resource-constrained edge devices).
- Investigate the applicability of feature extraction techniques and various deep learning methods to EEG_ER; conduct verification experiments to determine, simultaneously, the most suitable baseline deep learning architecture and the appropriate feature input format so as to achieve the best performance for EEG_ER.
- Study the application of new techniques in combination with the baseline deep learning architecture identified in Objective 2, with the aim of constructing a lightweight deep learning model that can be deployed on resource-limited processing devices while still ensuring high recognition performance for EEG_ER.
- Investigate the use of appropriate transfer learning techniques to further improve recognition performance and speed up inference without increasing model size, and demonstrate effectiveness through both mathematical analysis and simulation
- Research subjects:
The EEG_ER problem; EEG signal feature extraction methods; deep learning methods; techniques for improving the performance of deep neural networks; transfer learning techniques.
- Research methods:
Theoretical research: Study the theory of emotion recognition (especially EEG_ER); study the theory of deep neural networks and techniques to improve their performance; study transfer learning techniques to identify effective, small-footprint deep learning models.
Simulation-based research: Use numerical simulation tools on Google Colab Pro to tune and evaluate the results derived from theory.
Scholarly communication: Conduct discussions and seminars, seek expert feedback, and publish research results in domestic and international conferences and journals.
- Scientific significance:
The thesis is among the few scientific works in Vietnam that study the search for efficient deep learning models for the EEG_ER problem, oriented toward deployment in practical AIoT applications. The thesis contributes scientific solutions for building lightweight deep learning models that operate effectively for EEG_ER. These models both ensure small size (capable of real-time deployment on common microcontrollers) and achieve high performance. The performance has been evaluated by simulation on the datasets DEAP, AMIGOS, and DREAMER (widely recognized by the international community). This architecture is achieved by two improvement measures. First is the application of Inception and Squeeze and Excitation techniques in combination with the traditional 1D-CNN deep network to form a deep learning model that can be deployed on computing devices with limited resources. Next is the application of the SKD transfer learning technique, which increases performance and accuracy without increasing model size.
- Practical significance:
The research results of the thesis provide a basis for pilot application and for improving recognition quality for EEG_ER problems in the AIoT orientation in social life. Typical examples of such applications include evaluating the effectiveness of advertising videos based on EEG_ER, assessing the behavioral capacity of drivers after alcohol consumption, and evaluating the level of autism in children. In particular, the thesis is very meaningful for the class of applications that support people with total loss of mobility to reintegrate into the community, such as detecting patients’ needs in daily activities, controlling electric wheelchairs, and controlling smart-home devices based on EEG signals.
- The thesis has achieved the following new results:
Building an efficient deep learning architecture for deploying the EEG_ER problem in practical HCI applications—on edge devices within AIoT networks where resources are limited—is an inevitable trend. However, the difficulty lies in finding a compact deep learning architecture that can be installed on devices constrained by storage and processing capabilities while still maintaining high recognition performance. To solve this problem, we first need to determine the baseline deep learning architecture. Next, certain technical measures must be applied to improve this architecture so as to meet the targets for model size and recognition effectiveness. Compared with the stated objectives, the thesis has accomplished the following main tasks:
– Updated the overview knowledge on the EEG_ER problem, with a deep focus on studying the key techniques to be implemented in the stages of EEG signal acquisition, feature extraction, and emotion classification; clearly identified the challenges to be overcome when building an EEG_ER system oriented toward AIoT. Part of this content has also been published in CB [1].
– Investigated the applicability of different deep learning techniques to the EEG_ER problem; updated the most recent studies that apply deep learning to address EEG_ER; and conducted verification experiments to determine that the 1D-CNN deep learning architecture together with FFT feature input is the most suitable for deploying EEG_ER under resource constraints. The research results were published in CB [2], CB [3], and CB [4].
– Studied the application of the two techniques “Inception” and “Squeeze and Excitation” to improve the traditional 1D-CNN deep learning architecture so that it is more suitable for EEG_ER applications under resource constraints. Proposed two deep learning models that are small in size but achieve high recognition performance. Demonstrated the effectiveness of these two models through simulation tools. The research results were published in CB [5] (the EEG_MCISNet model for binary classification) and CB [6] (the EEG_SICNet model for multi-class classification).
– Based on the obtained deep learning model EEG_SICNet, studied the application of the SKD transfer learning technique to increase recognition performance and speed up inference without increasing model size. In this, the scaled weight technique for the student loss helps the model focus more on important signals after FFT feature extraction, thereby improving performance and accuracy without increasing model size. The effectiveness of the resulting architecture (the EEG_SKDNet model) was demonstrated through simulation tools. The research results were published in CB [7].
Futher research directions
From the achieved results, the thesis orients several subsequent research and development directions as follows:
It can be seen that the EEG_ER problem is an attractive, highly promising research field that can be applied in many areas of real life. This is a complex problem but will be solved if we know how to apply research achievements in fields such as digital signal processing and artificial intelligence. Among them, the application of solutions to improve the traditional 1D-CNN deep learning architecture with FFT as the input feature (as presented in detail in the thesis) brings encouraging results when deploying applications under resource-constrained conditions. Therefore, in the view of the doctoral candidate, the thesis still has several development directions as follows:
– Study pilot deployment for several decision-support problems based on information from EEG signals, such as controlling electric wheelchairs and controlling smart homes.
– Study the application of new technical solutions to further improve the operational effectiveness of the deep learning architecture for the EEG_ER problem when deployed in practical AIoT applications.
– Especially for the EEG_ER problem, the content of the thesis has so far focused only on finding an efficient deep learning architecture. Besides that, data also plays an extremely important role. The lack of data as well as data heterogeneity and imbalance are major issues that need to be addressed. It is necessary to seek solutions to expand and enrich existing data, use semi-supervised techniques for automatic labeling, and develop standard test datasets in order to improve the accuracy and generalization capability of emotion recognition models.
