A Taxonomy of Multimodal Interaction in the
Human Information Processing System

A Report of the ESPRIT PROJECT 8579
-- WP 1 --

February, 1995

Written by
L. Schomaker, J. Nijtmans (NICI)
A. Camurri, F. Lavagetto, P. Morasso (DIST)
C. Benoît, T. Guiard-Marigny, B. Le Goff,
J. Robert-Ribes, A. Adjoudani (ICP)
I. Defée (RIIT)
S. Münch (UKA)
K. Hartung, J. Blauert (RUB)

Abstract:

• Contents
• Introduction
  • Definitions of Basic Terms
    • The basic model for human-computer interaction
    • Levels of observation
    • (Multi-) Modality
    • Multimodal vs. multimedia vs. virtual reality system
    • Communication channels
      • The Psychophysical Law
      • Uncertainty and discrimination
  • Additional Notes and Caveats
    • An extra information loop?
    • Notes with respect to intention
    • Topics which had to be excluded from this report
  • Structure of the Document
• Perception
  • Human Input Channels
    • Input modalities
    • Vision
    • Hearing
    • Somatic senses
  • Computer Output Media
    • Output modalities
    • Devices and methods for visual output
    • Devices and methods for acoustical output
      • Speech output
      • Non-speech audio output
      • Sound spatialization
      • Hardware platforms
    • Devices and methods for tactile/haptic output
  • Bi- and Multimodal Perception
    • Visual-acoustical perception
    • Visual-speech perception
      • The intrinsic bimodality of speech communication
      • Intelligibility of visible speech
      • The need for coherence between facial gestures and speech sounds
      • The specific nature of speech coherence between acoustics and optics
      • The bimodality of speech
• Control and Manipulation
  • Human Output Channels
    • Cybernetics: Closed-loop control
    • Open-loop models
    • Coordinative structure models
    • Relevance of these theories for multimodal interaction
  • Computer Input Modalities
    • Keyboards
    • Mice
    • Pens
      • Identifying the ``inking'' condition
      • Integrated display and digitizer
      • Signal processing issues
      • Pen ergonomics
      • Other signals
    • Cameras
    • Microphones
    • 3D input devices
    • Other input devices
    • Generalized input devices
  • Event Handling Architectures in CIM
    • Within-application event loops: GEM, X11
    • Event-routine binding: Motif, Tcl/Tk
  • Bi- and Multimodal Control
    • Visual-gestural control
    • Handwriting-visual control
    • Handwriting-speech control
      • Automatic recognition and artificial synthesis
      • Handwriting Recognition/Speech Recognition: Improved text entry
      • Handwriting Recognition/Speech Synthesis: Handwriting recognizer feedback by synthesized speech
      • Speech Recognition/Handwriting Synthesis: The user dictates a ``written'' letter
      • Handwriting Synthesis/Speech Synthesis: Improved communication through bimodality
      • Recording and replay of speech and handwriting
    • Visual-motoric control
• Interaction
  • Architectures and Interaction Models for Multimodal Systems
    • A design space for multimodal systems
    • A framework for the design space of interfaces
    • Architecture of a multimodal dialogue interface
    • Software structure of UIMS
    • The cognitive coprocessor architecture
    • Architectural qualities and principles
    • Interactions in Virtual Environments
    • Man-machine communication
    • Ecological interface design
    • Intelligent user interfaces
    • Wizard of Oz technique for multimodal systems
  • Input/Output Coupling
  • Synchronization
    • Object synchronization
    • Complexity of information
  • Virtual Reality?
  • Analysis of Interaction
• Cognition
  • Cognition in Humans
    • Symbolic, subsymbolic, and analogical
    • High-level representations: Basic issues and requirements
    • Human learning and adaptation
    • Hybrid interactive systems
  • (Intelligent) Agents and Multimedia
    • Application Scenarios
• Scenarios & Dreams
  • The Multimodal Orchestra
  • Multimodal Mobile Robot Control
• An Introduction to Binaural Technology
  • The Ears-and-Head Array: Physics of Binaural Hearing
    • Binaural recording and authentic reproduction
    • Binaural measurement and evaluation
    • Binaural simulation and displays
  • The Subcortical Auditory System: Psychophysics of Binaural Hearing
    • Spatial hearing
    • Binaural psychoacoustic descriptors
    • Binaural signal enhancement
  • The Cortex: Psychology of Binaural Hearing
  • Acknowledgement
• Audio-Visual Speech Synthesis
  • Visual Speech Synthesis from Acoustics
    • Articulatory description
    • Articulatory synthesis
  • Audio-Visual Speech Synthesis from Text
    • Animation of synthetic faces
    • Audio-visual speech synthesis
• Audio-Visual Speech Recognition
  • Integration Models of Audio-Visual Speech by Humans
    • General principles for integration
    • Five models of audio-visual integration in speech perception
    • Conclusion
    • Taxonomy of the integration models
  • Audio-Visual Speech Recognition by Machines
    • Audio-visual speech perception by humans
    • Automatic visual speech recognition
    • Automatic audio-visual speech recognition
    • Current results obtained at ICP
      • Experimental paradigm
      • Technique used
      • Preliminary results
      • Results with degraded acoustic training
    • Forecast for future works
• Gesture Taxonomies
  • Hand Gestures Taxonomy
• Two-dimensional Movement in Time: Handwriting, Drawing, and Pen Gestures
  • The Pen-based CIM/HOC as a Means of Human-Computer Interaction
  • Textual Data Input
    • Conversion to ASCII (handwriting recognition)
      • Free Text Entry
      • Boxed Forms
      • Virtual keyboard
    • Graphical text storage
  • Command Entry
    • Widget selection (discrete selection)
    • Drag-and-drop operations (continuous control)
    • Pen gestures
      • Position-independent gestures
      • Position-dependent context gestures
    • Continuous control
  • Handwriting and Pen Gestures Computer Output Media (COM)
  • Graphical Pattern Input
    • Free-style drawings
    • Flow charts and schematics
    • Miscellaneous symbolic input
  • Known Bimodal Experiments in Handwriting
    • Speech command recognition and pen input
    • Handwriting recognition and speech synthesis
• References
• About this document ...

• Download a PostScript or Gzipped PostScript version of this document.

Esprit Project 8579/MIAMI (Schomaker et al., '95)
Thu May 18 16:00:17 MET DST 1995