next up previous contents
Next: Complexity of information Up: Synchronization Previous: Synchronization

Object synchronization

As stated in 2.3.1 , audiovisual integration depends on the acoustical and visual signals interaction and complexity. The following taxonomy captures the essential aspects of the interaction and complexity. We are not providing here numerical data characterizing the integration processes as they are highly dependent on the particular signals used. Fundamental to the underlying integration and cross-modal effects are the following properties:

Synchronized objects
This category refers to audiovisual objects whose all relevant spatial and/or temporal properties as measured by single senses, are overlapping. Synchrony of representations may refer to many different aspects: spatial location, temporal changes, motion. The synchronized representation leads to a fast, strong and 'natural' enhancement of the representation which might be perhaps best described that single-sensory object descriptions give rise to the creation of an 'enhanced' multisensory object. This enhancement lead in turn to many specific cross-modal effects, one of them is for example enhancement of intelligibility of audio-visual speech perception [208].
Nonsynchronized objects
This term refers to the situation when at least some of the spatial and/or temporal properties of objects, as measured by the single senses, are in conflict. Here one can consider as a reasonable hypothesis, that because of its goal of building of a consistent representation the sensory integration system will try to put a lot of effort to integrate the conflicting data, even at a cost of sacrificing the precision from a single sense. Only if this process of building of an integrated representation fails, the data will be interpreted as coming from many objects. One of the most prominent effects of this kind is ventriloquism [321], where by a clever visual stimulation, ventriloquist uses its own voice but creates strong illusion of a talking puppet. This is achieved by perfect integration of visual and acoustical stimuli. However, it seems that the integration effect in the spatial domain is much more pronounced than in the temporal domain.



next up previous contents
Next: Complexity of information Up: Synchronization Previous: Synchronization



Esprit Project 8579/MIAMI (Schomaker et al., '95)
Thu May 18 16:00:17 MET DST 1995