Next: Complexity of information
Up: Synchronization
Previous: Synchronization
As stated in 2.3.1 , audiovisual integration depends on
the acoustical and visual signals interaction and complexity. The following
taxonomy captures the essential aspects of the interaction and complexity.
We are not providing here numerical data characterizing the integration
processes as they are highly dependent on the particular signals used.
Fundamental to the underlying integration and cross-modal effects are the
following properties:
- Synchronized objects
- This category refers to audiovisual objects
whose all relevant spatial and/or temporal properties as measured by
single senses, are overlapping. Synchrony of representations may refer to
many different aspects: spatial location, temporal changes, motion. The
synchronized representation leads to a fast, strong and 'natural'
enhancement of the representation which might be perhaps best described
that single-sensory object descriptions give rise to the creation of an
'enhanced' multisensory object. This enhancement lead in turn to many
specific cross-modal effects, one of them is for example enhancement of
intelligibility of audio-visual speech perception [208].
- Nonsynchronized objects
- This term refers to the situation when at
least some of the spatial and/or temporal properties of objects, as
measured by the single senses, are in conflict. Here one can consider as
a reasonable hypothesis, that because of its goal of building of a
consistent representation the sensory integration system will try to put
a lot of effort to integrate the conflicting data, even at a cost of
sacrificing the precision from a single sense. Only if this process of
building of an integrated representation fails, the data will be
interpreted as coming from many objects. One of the most prominent
effects of this kind is ventriloquism [321], where by a clever
visual stimulation, ventriloquist uses its own voice but creates strong
illusion of a talking puppet. This is achieved by perfect integration of
visual and acoustical stimuli. However, it seems that the integration
effect in the spatial domain is much more pronounced than in the temporal
domain.
Next: Complexity of information
Up: Synchronization
Previous: Synchronization
Esprit Project 8579/MIAMI (Schomaker et al., '95)
Thu May 18 16:00:17 MET DST 1995