ANTI-NOISE SPEECH RECOGNITION BASED ON INFORMATION FUSION

To improve the anti-noise capacity of single audio information-based continuous speech recognition approaches in a noisy environment, we propose an information fusion-based anti-noise audio-visual speech recognition (AAVSR) model. The proposed AAVSR model utilized the attention mechanism to learn the correspondences between the audio and video streams autonomously. Based on the learned correspondences, the features extracted from audio and video streams were fused to supplement the missing information of each individual modality. The fused complementary information improved information utilization and enhanced the robust recognition ability of the AAVSR model. Comprehensive simulation results on the LRS2 dataset demonstrate that AAVSR outperforms the other competing models in terms of the word error rate under a noisy environment with various signal-to-noise ratios.