Abstract: Extracting the main branches is the key issue for visual picking of pepper. Aimed at the problem that the existing network model is difficult to segment the close-range main branches from complex natural scenes, a newly-designed attention module is added to the network, and a RGBD image pepper branch segmentation model based on spatial and channel attention is proposed. The color and depth image features were extracted by two encoders respectively. The design space attention module combined these two features. The spatial weight considered appearance and distance information at the same time, which could suppress the interference of distant branches. The channel attention module was constructed to pay attention to the main shape information of the branches and avoid the interference of the details of the branches. Experiments show that compared with the existing dual-encoder network, the segmentation accuracy and cross-union ratio of the proposed model are increased by 14.72% and 17.65%, respectively. The proposed model can segment satisfactory close-range trunk branches.