Abstract: In order to solve the problem that the decorative parts contained in ancient buildings are mostly attached to the building and smaller than the building body, which is difficult to detect and segment, a semantic segmentation method of ancient architecture images based on boundary detection and pixel expansion is proposed. The VGG network and HED detection technology were used to extract features of ancient building images and detect the boundary of image respectively so that the decorative parts which were difficult to detect in the ancient building were separated from the building body to generate boundary constraints. Within the boundary constrained area, the eight-connectivity feature was used to achieve the semantic segmentation of the ancient architecture images by expanding the hardly segment parts of ancient architecture. The effectiveness of this method was proved by the experiments based on ancient architecture image dataset and ADE20K dataset.