Abstract: Aimed at the problem that some low-level and middle-level detail information is lost when the existing semantic segmentation algorithms extract high-level semantic information from feature map, an image semantic segmentation algorithm based on ASPP module and weighted fusion of feature map is proposed. With the help of the “residual idea”, the improved algorithm established the jump connection between the feature extraction network and the ASPP module, integrated the weighting of the feature map, increased the ability of the ASPP module to extract image semantic information, reduced the loss of detail information in the low-level and middle-level layers, and performed batch normalization after the corresponding convolution layer to alleviate the problem of gradient disappearance and improve the segmentation performance and operation efficiency of the model. The experimental results show that the reasonable fusion of feature maps makes the edge details of semantic segmentation map better preserved, and the segmentation accuracy of the improved algorithm is enhanced. The average intersection and union ratio is 6.8% higher than the original model, and the frequency weighted intersection and union ratio is 5.33%.