next up previous contents
Next: Speech Recognition/Handwriting Synthesis: Up: Handwriting-speech control Previous: Handwriting Recognition/Speech Recognition:

Handwriting Recognition/Speech Synthesis: Handwriting recognizer feedback by synthesized speech

[COM]
HOC
(a) hand/arm musculature: position and compliance control
(b) speech musculature, vocal chords, respiratory system: vocal sound production
CIM
(a) XY digitizer, handwriting recognition algorithm
(b) Microphone, speech recognition algorithm
COM
Voice-like sound is being produced by the computer, representing recognized text or the state of the recognition agent
HIC
The user gets:
(a) Immediate feedback on speech and handwriting by the intrinsic feedback loop (see Figure 1.1 )
(b) Auditory feedback

At NICI, a number of experiments have been performed with an experimentation platform called PenBlock running under Unix. In the case of isolated characters (handprint), a spoken spelled letter can be produced, after each character that is entered. If the character is not recognized with sufficient likelihood, an ``uh?'' sound is produced. The meaning of this is picked up quickly by the subjects, and makes more explicit the fact that an active deciding 'agent' is trying to assess the incoming handwriting products, instead of an infallible machine-like tool. However, the usability of feedback in the form of spoken recognized letters must be studied in more detail. A more neutral ``click'' sound, after entering each character was considered more acceptable by a small number of informal subjects. Variations on this theme can be designed, giving only speech feedback in case of doubt by the recognition agent.

Other experiments have been performed on a PC, in a small project dubbed ``PenTalk'' together with Tulip Computers. In this project, the user wrote isolated cursive words (Dutch) which were pronounced after on-line recognition by a speech synthesizer program. Preliminary experiments indicated that the use of speech feedback is limited, especially if the delay is more than about half a second after writing the word. In later versions of the program, speech synthesis synthesised speech feedback was deferred to a drag-and-drop type function. The philosophy was that if it could not be immediate, speech feedback was better placed under active user control. The user decides when and what must be spoken. This has the advantage that the speech synthesis algorithm can pronounce sentences and paragraphs, effectively coding the prosody of speech, which is not possible in isolated words. Based on these preliminary findings it is concluded that speech as an immediate feedback in handwriting recognition is probably most useful as a way of multimodally informing the user on the state of the recognition process or the quality of the handwriting input. Auditory feedback in any case has the advantage that the user does not have to look away from the pen-tip, i.e., where the action takes place. Other forms of (non-vocal) auditory feedback may be considered in .



next up previous contents
Next: Speech Recognition/Handwriting Synthesis: Up: Handwriting-speech control Previous: Handwriting Recognition/Speech Recognition:



Esprit Project 8579/MIAMI (Schomaker et al., '95)
Thu May 18 16:00:17 MET DST 1995