Handwriting Synthesis/Speech Synthesis: Improved communication through bimodality

The combined presentation of linguistic information in the form of speech and handwriting has long been known to be effective by movie directors. Typically, a letter is shown on the screen, and an off-screen voice reads it aloud. The ambiguity in the handwriting style is reduced by the spoken words, and vice versa.

[COM]

HOC

Any form of earlier text entry

CIM

Any text-oriented modality

COM

(a) Synthesized handwriting in the form of pixel ink, or a handwriting trajectory production animation is presented to the user
(b) synthesized speech, pronouncing the same content as the handwriting is presented to the user

HIC

The user reads the handwriting from the screen, and listens to the synthesized speech

Issues here are the optional synchronization of handwriting and speech presentation. Modes are: (i) completely asynchronous, the handwriting can be displayed almost immediately, and is continually present on screen while the speech production takes its time; (ii) synchronous, incremental, where the handwritten words follow the spoken words; and (ii) synchronous, by highlighting the 'current word' in the handwritten ink on screen. These paradigms are relatively well known for the case of speech synthesis and machine-printed fonts on the screen.

Esprit Project 8579/MIAMI (Schomaker et al., '95)
Thu May 18 16:00:17 MET DST 1995