Abstract: At present, the field of surface defect detection lacks a deep learning method with low cost of label production and good detection effect. Therefore, a weakly supervised convolutional neural network using binary classification labels is proposed. This model used the framework of Segdec-Net, redesigned the convolutional layer structure of the segmentation sub-network, solved the problem of inaccurate segmentation of defect contours, and improved the classification effect. The classification sub-network was improved by structure simplification and dropout processing, which alleviated the overfitting problem and further improved the classification effect. The experimental results show that the average classification accuracy of this method reaches 96%, which is 22.7 percentage points higher than similar methods.