(Multi-) Modality

According to [67], modality is defined as follows:

`` Perception via one of the three perception-channels. You can distinguish the three modalities: visual, auditive, and tactile (physiology of senses).''

visual:

concerned with, used in seeing (comp. against optical)

auditive:

related to the sense of hearing (comp. against acoustical)

tactile:

experienced by the sense of touch

haptic:

most authors are using `` tactile'' and `` haptic'' as synonyms. However, in [120] tactile as perception-modality is distinguished from haptic as output manner.

optics/optical:

optics is the theory on light as well as the infrared and ultraviolet radiation. Thus the attribute `` optical'' refers to physical quantities and laws rather than to physiological ones. (comp. against visual)

acoustics/acoustical:

acoustics is the theory on vibrations and oscillations in elastic mediums, especially of sound, its generation, spreading, and its reception. Thus the attribute `` acoustical'' refers to physical rather than to physiological quantities and laws. (comp. against auditive)

  
Table 1.1 : Different senses and their corresponding modalities (taken from: [311])

In our opinion, the sense of smell and the sense of taste are not very interesting for our concerns (see 2.1.1 for a deeper discussion). However, the sense of balance seems to become more and more interesting with respect to virtual reality environments (see 4.4 ). Presently it is already used in flight simulators for example.

Whenever more than two of these modalities are involved, we will speak of multimodality. To be more precise, in some cases we will also use the term bimodal (or bimodality) to denote the usage of exactly two different modalities. In this sense, every human-computer interaction has to be considered as multimodal, because the user looks at the monitor, types in some commands or moves the mouse (or some other device) and clicks at certain positions, hears the reaction (beeps, key clicks, etc.) and so on.

Therefore, in our understanding of multimodality is restricted to those interactions which comprise more than one modality on either the input (i.e., perception) or the output (i.e., control) side of the loop and the use of more than one device on either side. Thus, the combination of, e.g., visual, auditive, and tactile feedback which is experienced by typing on a keyboard is explicitly excluded, whereas the combination of visual and auditive output produced by the monitor and a loudspeaker when an error occurred is a 'real' multimodal (or --- in this case --- bimodal) event.

Esprit Project 8579/MIAMI (Schomaker et al., '95)
Thu May 18 16:00:17 MET DST 1995