The construction site is one of the most hazardous working places with numerous sources of hazards and potentially high risk to deals with. Typically, the main sources of hazards are unsafe condition and unsafe act by the human. Eventually, both sources related to human errors which potentially have an effect on the performance of the site. To reveals the underlying, this study aims to determine the influential factors that governed the success of safety performance of a construction site. The study involved a self-administered three-part questionnaire among the construction personnel. Part A measures the workers' attitude and safety performance while Part B on communication skills. The respondents randomly selected from construction sites range from school building and infrastructure renovation. The sample size was 113. From the survey, it was found that the most influential factor was the worker's attitude and followed closely by communication skills. Suggestion and recommendations on equipment design and improved work practices and procedures to improve efficiency and productivity of construction workers were proposed. Management was urged to get their workers better informed about safety matters.