Cervical cancer is a form of cancer that develops in the cells of the cervix, which links the uterus to the vagina. Cervical cancer can often be prevented by attending the cervical screening, which aims to find and treat changes to cells before they turn into cancer. Pap smear images are a more popular technique for screening cervical cancer. The nucleus and cytoplasm are present in the smear cell images. The spread of cancer is determined by the shape and structure of the nucleus. Thus, nucleus segmentation is an important step in cancer detection. However, overlapping, poor contrast, uneven staining, and other factors make cervical nucleus segmentation difficult. This paper proposes a new segmentation method for the cervical nucleus using digital image processing. In our proposed method, image processing techniques are employed to segment the nucleus. This method mainly compared the performance of using Red Green Blue (RGB) channels and Hue, Saturation, and Value (HSV). The best two channels from each two-colored image representation are compared, where the V channel is the best among H and S channels. On the other hand, G chancel is the best among the G and B channels. Therefore, the best two channels are compared, and the segmentation using the G channel is the best among all, with a sensitivity of 94.3%, specificity of 94.6% and precision of 88.3%. The main impact of this paper will be to assist doctors in diagnosing cervical cancer-based segmentation of the nucleus in Pap smear images.