In the research of the human motion analysis, the characteristic movements of the human upper body are intensively investigated for many applications such as sign language recognition, robot control and gait analysis. The human upper body consists of many body parts such as both arms including fingers, facials and head movements. Previously, many researches proposed various sensors to record arm movements and the acquired data are used to train the computer understand the behavioral motion of arms movements by using various algorithmic approaches. However, the current challenge is to increase the knowledge level of the computational systems to recognize gestural information containing in arm movements. The objective of this paper is to construct and derive the arm movement’s model based on the conceptual of ontology. The gestural information is investigated from characteristic features of arm movements. The knowledge of the computational systems about gestural information is developed by describing the characteristic features of arm movements in the form of the ontological framework. The ontological framework is defined as a structure containing characteristic features placed in mathematical order and has the relationship among them. Based on the mathematical model as proposed in this paper, the ontology framework could be used to describe knowledge of the arm gesture and could recognize it with a higher accuracy.