Next: Multimodal Mobile Robot
Up: Scenarios & Dreams
Previous: Scenarios & Dreams
A possible application scenario which can be relevant for entertainment,
education, and clinical evaluation, is one in which the user is immersed in
a multimodal experience but not a conventional virtual reality that he
alone can perceive. Rather, we can think of an audio-visual environment
which can be communicated to other humans, either other actors
participating in the same event or external spectators of the action. For
example, we can think of a sound/music/light/image/speech synthesis system
which is driven/tuned by the movements of the actors using specific
metaphors for reaching, grasping, turning, pushing, navigating, etc.
Specific scenarios could regard:
- multimodal musical instruments
- multimodal orchestras
- multimodal choreography
- multimodal kinetic painting
- multimodal teaching by showing
Multimodality technology requires an instrumented environment, instrumented
dresses, and instrumented tools which allow to capture natural human
movements. Learning problems have two sides and at least some of them can
be translated to a dimensionality-reduction paradigm:
- on the human side, like when learning to drive a car, there is the
need to perceive the complex but smooth and continuous associations
between movements and actions, avoiding cognitive bottlenecks which may
arise with discontinuous and/or symbolic feedback;
- on the computer side, there is the dimensionality-reduction problem
of extracting "principal components" from a high-dimensional space of
redundant degrees of freedom and this may or may not imply the
recognition of gestures.
In any case, we need learning because in multimodal systems there is not,
in general, a simple one-to-one translation of signals and events as in the
VR systems.
Esprit Project 8579/MIAMI (Schomaker et al., '95)
Thu May 18 16:00:17 MET DST 1995