Multimodal Integration for Advanced Multimedia Interfaces

ESPRIT III, Basic Research Project 8579

Current topic: has finished

In memoriam Christian Benoit

Project Description

Project Title:
Multimodal Integration for Advanced Multimedia Interfaces
Acronym:
MIAMI
Number:
8579
Work Area:
Multimedia, advanced multimodal human-computer interfaces
Coordinator:
Nijmegen Institute of Cognition and Information
Nijmegen University, P.O.Box 9104
NL- 6500 HE NIJMEGEN
Coordinator Country:
NL
Partners
Ruhr-Universität Bochum D
Universität Karlsruhe (IPR) D
Université Stendhal Grenoble F
Universita di Genova I
University of Technology Tampere SF
Contact Point:
dr. Lambert Schomaker
Telephone:
+31 50 3617908
E-Mail:
schomaker@computer.org
Keywords:
multimedia, human-computer interaction, speech synthesis, sound spatialisation, speech recognition, pen gestures, teleoperating, virtual and real object manipulation, music, handwriting recognition, virtual talking faces
Start Date:
March 1, 1994
Duration:
36 months
Status:
Finished
Abstract:
The goal of MIAMI is to develop approaches for integrating multimedia data on the basis of knowledge of the Human Information Processing system. Multimodality aspects in accessing, representing and producing multimedia information by visual, acoustical and tactile/gestural systems are studied, as the basis for future multimedia systems.

AIMS

The project aims at developing approaches for building integrated representations of multimedia data by modelling certain basic properties of the Human Information Processing system (HIP). In MIAMI, we want to study full multimodality aspects in accessing, representing and producing multimedia information by visual, acoustical and tactile/gestural systems. The main aspect studied is how to build integrated representations of different modalities as they occur at many levels of processing. This integration is useful, both in the disambiguation of the human input to the system and in the disambiguation of the system output.

APPROACH AND METHODS

MIAMI aims to follow the human information processing by providing and analyzing data on selected aspects of human multimodal data processing, and by devising an architecture for revealing and exploiting its integrative capabilities. To avoid bottlenecks, a systematic, bottom-up approach combining accumulated knowledge, psychophysical experiments and deductive thinking is applied to developing new algorithms, their experimental verification and practical applications. The application side is stressed by developing two typical scenarios for multimedia information acquisition, representation, and use. The first scenario is related to symbolic information retrieval and develops an "Information City" metaphor. The second scenario is related to the interaction and manipulation in information space as exemplified by teleoperation. The scenarios cover fundamental aspects of multimedia data handling and are close to practical applications.

POTENTIAL

Multimedia is the technology of the future and this project is of primary importance for next generation information processing systems. These systems will be evolving from the present "computers" to integrated information assistants, managers and communicators. This project is focused on a central issue of the integrated handling of information acquired by different modalities and presented in a multimedia context. The ideas, models, and algorithms produced in this project are most likely to exert a profound influence on a whole range of future information technology products.

MORE INFORMATION


(J. Nijtmans/Oct 25 1995, LS/Dec 2001)