Perception

In this chapter we will deal with perception, i.e., the process of transforming sensorial information to higher-level representations which can be used in associative processes (memory access) and cognitive processes such as reasoning. In the perceptual process, two basic components can be identified as described earlier (figure 1.1 , right part). First there is the human sensory system with its typical characteristics and constraints, providing the ``human input channel'' (HIC) processing functions. Second, there are those components of a computer system which provide a multidimensional signal for the human user: ``computer output media'' (COM).

According to our basic model introduced in section 1 , we use the term perception to describe the communication from a machine to a human. First, we will show the different kinds of human input channels through which perception becomes possible and give a short overview on these channels at the neurophysiological level ( 2.1 ). The second part of this section ( 2.2 ) deals with devices and methods used by computers to address these human senses. Finally, in 2.3 we will present our results of bi- and multimodal machine-to-man communication.

• Human Input Channels
  • Input modalities
  • Vision
  • Hearing
  • Somatic senses
• Computer Output Media
  • Output modalities
  • Devices and methods for visual output
  • Devices and methods for acoustical output
    • Speech output
    • Non-speech audio output
    • Sound spatialization
    • Hardware platforms
  • Devices and methods for tactile/haptic output
• Bi- and Multimodal Perception
  • Visual-acoustical perception
  • Visual-speech perception
    • The intrinsic bimodality of speech communication
    • Intelligibility of visible speech
    • The need for coherence between facial gestures and speech sounds
      • Spatial coherence
      • Temporal coherence
      • Source coherence
    • The specific nature of speech coherence between acoustics and optics
    • The bimodality of speech
      • The synergetic bimodality of speech
      • The specific bimodality of speech

Esprit Project 8579/MIAMI (Schomaker et al., '95)
Thu May 18 16:00:17 MET DST 1995