Visual-gestural control

Gestural control is a relevant aspect in multimedia systems. With the current state-of-the-art human movement tracking technology, it is possible to represent most of degrees of freedom of a (part of the) human body in real-time. This allows for a growing number of applications, from advanced human-computer interfaces in multimedia systems to new kinds of interactive multimedia systems (e.g., [175,247]).

The classification of gestural information patterns in human-machine interaction is an important research topic. A short overview of existing literature is given in Appendix D, mainly referred to robotics and cognitive/psychological studies. The recent research on virtual environments has given a strong acceleration to the research in this field --- see for example [247]. From another point of view, it is also important to analyze human activities which involve significant gesture communication. A significant field is related to the study of the languages developed over centuries for dance and music. This field is important also from the point of view of the study of the involved emotional communication [252,11]. The main areas regard

(i)

dance gesture languages and classifications of movement patterns [55,297,51,336,340];

(ii)

languages for music conducting [232];

(iii)

the study of the movement involved in music instruments performance [9]; and

(iv)

the recent research efforts on ``Hyper-Instruments'' at MIT and in several other research centers [234,196,360] to simulate existing musical instruments and to find out new routes of communicating movement and emotional contents in art performance.

Different taxonomies of gesture can be defined with respect to different aspects of gesture communication. The following are some main examples, explored in WP 2.3:

1. Types of movement detected:
   • gross-grain movements (typically full-body);
   • medium-grain movements (arms, legs; e.g., dance patterns);
   • fine-grain movements (e.g., hand gestures).
2. Types of sensoring devices:
   • inertial sensors (inclinometers, accelerometers, gyrometers and gyroscopes);
   • exoskeletons or compliant manipulators, grasped by users or linked to joints of his/her body. Such devices are controlled by a tunable compliance matrix;
   • position, distance sensors (e.g., Costel, MacReflex, V-scope), e.g. based on infrared and ultrasound sensor technology, which should be able to detect in real-time the three-dimensional position of a number of markers, typically worn by the user;
   • devices for gross-movement detection. e.g. a non-linear matrix of active infrared sensors together a sensorized floor, covering a single-user 3D area. Examples of gross movements include rhythmic (part-of-the) body patterns, energy of the movement (which/how many sensors activated in the time unit).
3. Semantics:

   The important aspect is here the meaning associated to different gestures. The use of metaphors and analogies with other modalities is a fundamental aspect to devise evoking associations to gestures.

Esprit Project 8579/MIAMI (Schomaker et al., '95)
Thu May 18 16:00:17 MET DST 1995