Abstract: Colon endoscopy is the gold standard for detecting rectal polyps and rectal cancer. Among them, polyps are the main causative agent of colorectal cancer, but accurate diagnosis of polyps under colorectal endoscopy is extremely dependent on physicians with professional standards. For this reason, we propose an adaptive semantic calibration network (ASC-Net) based on real-time colon endoscopy. ASC-Net used a non-global self-calibration (NSC) module to develop long-range spatial dependency through contextual calibration operations to extract the most discriminative features. In addition, a context-guided semantic calibration (CSC) module was designed among the model to dynamically capture multi-scale polyps by collecting multi-scale contextual information. A multi-scale feature fusion (MFF) module was developed to learn more representative features and fuse them to refine the segmentation results. The proposed ASC-Net achieved Dice coefficient of 84.58% and AUC metrics of 98.80% on the Kasir-SEG dataset, which improved the Dice coefficient accuracy by 9.76, 4.41 and 1.74 percentage points compared with the U-Net, TransFuse and HarDNet algorithms, respectively.