Abstract: Non-intrusive load decomposition technology plays an important role in smart grid. In order to improve the perception ability of Transformer model to local information of active power sequence and improve the accuracy of non-intrusive load decomposition, the load decomposition is carried out by combining the classification model with the improved Transformer model. The local information and global information were extracted by the multi-encoder structure. The different features of the data were fused by two multi-head attention mechanisms in the decoder to enhance the representation ability of the neural network. The final decomposition result was obtained by fusing the regression result with the state probability. Comparative experiments on the public dataset UKDALE show that MAE is reduced by 14.20% at least, which verifies that the method is more accurate than the other methods.