Combination of facial expressions and EEG for multimodal emotion recognition

Abstract

Emotion recognition is crucial in various fields, particularly in human-machine interac-
tion. Although previous research has focused predominantly on using facial expressions or elec-
troencephalogram (EEG) signals as the sole input for emotion recognition, integrating multiple data
types into a single system remains an underexplored area. This paper aims to contribute to this evolv-
ing field by exploring a multimodal emotion recognition approach that combines facial expressions
and EEG signals. For facial expression recognition, two deep learning models are developed: one
for detecting facial emotions and another for classifying them. The focus is on improving the effi-
ciency of the recognition model by leveraging modern machine learning techniques and minimizing
the number of parameters. In parallel, a small dataset is collected using the Emotiv FLEX 2 Saline
32-channel EEG headset, capturing four distinct emotional states. The processing of EEG signals
involves a complete workflow, from pre-processing to feature extraction, followed by mapping the
features into a format suitable for emotion recognition. The evaluation results demonstrate the effec-
tiveness of the proposed method. The facial expression recognition model achieves an accuracy of
92.5%, while the EEG-based recognition model achieves an impressive 98.7%. Furthermore, com-
bining the output of both models improves performance, as the integration of facial expressions and
EEG signals compensates for the limitations of using either data type individually.