Multimodal perception is not specific to speech, nor to audition/vision. Information transfer and intermodal integration are classical problems widely studied in psychology and neuropsychology [136,137,358,135,321]. Therefore, interactions between various modalities have been studied in literature:
From literature (see [137], for a review), three main classes of intersensory fusion models can be considered:
In the first case, information is categorized in symbols which provide the interaction with a support. Classification thus precedes fusion: Such models can be called as late integration [348]. In order for the transfer between modalities to be done at a symbolic level, the subject is supposed to have learned the symbols. Since infants and animals are capable of visual and tactile interaction, the hypothesis of intermodal transfer through language is obsolete for visual-tactile perceptions [136].
The second category assumes that one modality dominates the other for a given task. For instance, it is considered that vision is dominant for spatial localization [281] and that audition is dominant for judgments dealing with the temporal organization of stimuli [137]. The non-dominant (or submissive?) modality is then recoded into the dominant one. It is only after this recoding process that the interaction (transfer, fusion) take place. According to Hatwell [136,137], transfer from vision to the tactile modality must be explained by a model which falls into this category:
``Spatial perceptions coming from the dominant modality (vision) remain in their own code, whereas those coming from the non-dominant modality (tactile) are, as best as possible, recoded into the dominant modality and adjusted to it''.
[136, page 340,]
In the third case, it is considered that the physical signal is used by the sensory system in order to retrieve information about the source responsible for its generation. Sensations are thus transcoded into an amodal space before to be integrated. This amodal space might be isomorphic to the physical properties of the source, as assumed in the Theory of Direct Perception (cf. Fowler [108]). It might also be made of internal dimensions that are independent from the modality, such as intensity, duration, etc. It is hence assumed that there is no direct transfer between modalities, but that each sensation is projected upon a same amodal space where interactions then take place.
Esprit Project 8579/MIAMI (Schomaker et al., '95)