Abstract: With the supervised deep learning algorithms, it is prone to overfitting when restoring the blurred solar speckle images taken by Yunnan Observatories, and it is over-reliance on reference images. In order to solve the problems, a method for gradient energy-based grading of blurred datasets and weakly supervised image reconstruction for grading datasets is proposed. The method used the Scharr operator to calculate the gradient energy of the blurred image, and classified the blurred image according to the energy value, so that the blurred distribution of the images of the same level was basically the same. We used the degradation model to simulate the degradation of the graded unpaired data set, and then used the trained degradation model to construct new paired data set. We put the new paired data set into the reconstruction network for inverse degradation learning to realize image reconstruction. Experiments show that this method can not only prevent serious overfitting of the model, but also reduce the dependence on reference images, and the reconstructed images can meet the requirements of high-resolution reconstruction of solar speckle images.