The assistive multimodal system for hands-free PC control based on Russian speech recognition and head tracking system

ICANDO:

Intellectual Computer AssistaNt for Disabled Operators

Aim of the system:

ICANDO assistive multimodal system is intended for hands‑free PC control for users with disabilities of their hands or arms. The interaction between a user and a machine is performed by voice and head (nose) movements. It gives the opportunity for disabled people to carry out a work with PC that improves the life comfort of disabled people as well as independence of their living from other persons.

촠 촨 x

Required hardware:

As hardware for hands-free PC control the miniature web-camera Logitech QuickCam for Notebooks Pro is used. This camera provides video signal in 640x480x30fps and audio signal, obtained from microphone built in the camera with 16 KHz and satisfactory SNR. Any other web-camera can be applied in the system. In noisy environments the good-quality microphone is required (for instance, microphone Sony DR-50).

Architecture:

In ICANDO system two natural input modalities are used: speech and head movements. As both modalities are active ones, then their input into the system must be controlled continuously (non-stop) by the computer. Each of the modalities transmits own semantic information: head (nose) position indicates the coordinates of some marker (cursor) in the current time moment, and speech signal transmits the information about meaning of the action, which must be performed with an object selected by cursor (or irrespectively to the cursor position).

It was determined experimentally that the most suitable point on face for tracking is the tip of nose. It is the center of face and when we make any gestures by head (turn to the right, left, up or down) the position of tip of nose is moved to this direction and it can indicate the position of mouse cursor on screen.

The system can recognize the voice commands of a user in three languages: English, Russian and French. The list of voice commands for ICANDO contains over 20 voice commands for PC control.

Left	Start
Right	New
Left down	Open
Left up	Save
Double click	Close
Scroll down	Copy
Scroll up	Cut
Enter	Paste
Escape	Print
Delete	Next
Shut down	Previous
Calibration	Select all

Demonstration of the system:

1. The video fragment of work of the assistive multimodal system. The scenario: finding and printing the information about current weather in the web-portal www.rambler.ru

[avi] 17.0 Mb, DivX format

2. The video was shown by Russian TV channel ("First channel") in the news program “Vremja” (“Time”) at prime-time. It is the story about a man without hands, who is able to work with the computer by the ICANDO system [link]

[avi

] 40.0 Mb, DivX format

Presentations:

1. The poster at EUSIPCO-2008 Conference (Switzerland) [pdf]

2. The presentation at Interspeech-2006 Conference (USA) [ppt]

3. The pictures of Loco Mummy Contest 2006 nomination (Belgium) [jpg] [jpg]

4. The poster at SIMILAR Industrial day meeting 2006 (Brussels) [jpg]

5. The presentation at SPECOM’2006 conference (Russia) [ppt]

6. The presentation at EUSIPCO’2005 conference (Turkey) [ppt]

Scientific papers on the ICANDO system:

1. A. Karpov, A. Ronzhin, I. Kipyatkova. An Assistive Bi-Modal User Interface Integrating Multi-Channel Speech Recognition and Computer Vision. In Proc. 14th International Conference on Human-Computer Interaction HCI International-2011, Springer-Verlag Berlin Heidelberg, LNCS 6762, Orlando, FL, USA, 2011, pp. 454-463.

2. A. Karpov, S. Carbini, A. Ronzhin, J.E. Viallet, chapter “Two Similar Different Speech and Gestures Multimodal Interfaces” in the book "Multimodal User Interfaces: From Signals to Interaction", D. Tzovaras (Ed.), Springer, 325 p., 2008.