MAE4DAR: Masked AutoEncoder for Driver Action Recognition

Abstract

Abstract: Driver Action Recognition (DAR) plays an important role in intelligent vehicle systems
and road safety applications. DAR is a subfield of Human Action Recognition (HAR) that identifies actions in videos. Besides the challenges inherited from HAR, DAR also faces severe occlusion
issues in confined environments such as vehicle cabins. In this paper, we introduce a Transformerbased framework for DAR, namely MAE4DAR, that leverages the representation learning capability
of VideoMAE V2 as a video encoder. The contribution of this work is twofold. First, we introduce a unified pipeline that handles both isolated and continuous recognition scenarios on the same
VideoMAE-based backbone. In isolated recognition setting, some experiments are conducted on two
datasets for driver action recognition that are UTCDriverAct and Driver Action Dataset (DAD). On
UTCDriverAct, MAE4DAR framework achieves strong performance, reaching perfect recognition
accuracy in four of the six driver action classes. In continuous setting, the proposed framework yields
a frame-wise accuracy of 92.7% on both views, with overlap scores of 0.72 and 0.69 for front-view
and rear-view data in UTCDriverAct dataset, respectively. Second, we propose multi-view fusion at
score level to combine prediction scores obtained from independently trained single-view models.
The experimental results indicate that multi-view fusion consistently improves recognition performance, achieving an overlap score of 0.75 and a frame-wise accuracy of 93.5%, outperforming both
single-view settings. Furthermore, we analyze the impact of multi-view fusion on both recognition
performance and computational cost, demonstrating that its performance gains justify the additional
inference overhead, which can be mitigated through parallel processing.
Keywords: Driver Action Recognition, Human Action Recognition, VideoMAE encoder,
Transformer-based Action Recognition.