An effective human metaphase chromosome analysis system can be used by doctors as a second opinion during diagnosis. Segmentation is necessary in developing this system to identify and distinguish between individual chromosomes. The main challenge in chromosome segmentation is the separation of overlapping chromosomes. Deep convolutional neural networks have been widely used for medical segmentation, especially with U-Net. This study investigated how Test time augmentation with a suitable number of U-Net layers can improve the design for this semantic segmentation problem. The proposed architecture was trained, validated and tested with 13,434 greyscale images with 88 × 88 pixels of overlapping chromosome pairs. With the implementation of the proposed method, the training result became more accurate without any mislabelling and additional pre-processing became unnecessary. An improved segmentation accuracy of 99.68% was obtained, which was higher than the 99.22% obtained using the method of Hu et al.