Prostate cancer is one of the most common cancers in men, and the cases of this disease is increasing. Histopathological examination of prostate cancer is one of the main procedures for prostate cancer detection. The structural changes of the cytoplasm, stroma, lumen and nucleus in the glandular tissue will indicate the presence of cancerous or non-cancerous areas in the histopathology of prostate cancer. Therefore, a framework was developed to automatically segment and classify glandular tissue into cytoplasm, stroma, lumen, and nucleus, which can reduce the complexity of prostate cancer detection. The images underwent image enhancement using histogram equalization (HE) and Contrast Limited Adaptive Histogram Equalization (CLAHE). Then, in segmentation phase, K-means clustering (KMC) and multi-level thresholding (MT) methods were implemented to segment the enhanced image into cytoplasm and stroma, lumen and nuclei regions. A total of 8 feature vectors are extracted from each segmented image. All these features were introduced into the classification system namely K nearest neighbor (KNN) and decision tree (DT). The overall results showed that the performance of KNN is better than DT with an accuracy of 86.67%, sensitivity and specificity are both 100% (the features of the KMC category). With the features of MT category, KNN achieved 84.44% in term of accuracy, 100% sensitivity and 96.67% specificity. Here, it can also be concluded that the features of the KMC category are more suitable for the classifiers. In addition, leave-one-out cross-validation has been implemented, which can improve the performance of the two classifiers.
