Abstract: Aimed at the problem that the current deep learning-based methods cannot effectively fuse multi-modal features of traffic, a method for detecting anomaly traffic with residuals between multi-modal fusion and sequential feature is proposed. We segmented the network traffic in units of sessions and obtained multi-modal features of traffic records. The multi-modal attention was used to merge the multi-modal features, and Transformer was used to mine the temporal features of traffic records. The fusion feature and sequential feature of multi-modal were combined by residual connection to detect. Experimental results on CSE-CIC-IDS2018 dataset show that accuracy rates under two classifications and multiple classifications are 95.19% and 90.52%, respectively. Compared with the comparison method, it maintains the lowest false alarm rate when accuracy and precision are optimal.