Decoding Silent Speech Commands from Articulatory Movements Through Soft Magnetic Skin and Machine Learning

Silent speech recognition is a promising technique to restore spoken communication for individuals with voice disorders and to facilitate intuitive communications when the acoustic signal is unreliable, inappropriate, or undesired. However, the current methodology for silent speech faces several challenges, including bulkiness, obtrusiveness, limited portability, and susceptibility to interferences. In this work, we present a wireless, unobtrusive, and robust silent speech interface for capturing and decoding speech-relevant movements of the temporomandibular joint. Our solution employs a single ultrasoft magnetic skin placed behind the ear for wireless and more socially acceptable silent speech recognition, greatly alleviate concerns associate with existing interfaces based on face-worn sensors, including large number of sensors, highly visible interfaces on the face, and obtrusive interconnections between sensors and testing unit. Through optimizations in material composition, sensor structure, and sensing location, we present a lightweight wireless sensor that can conform to the skin surface to capture subtle movements induced by speech. With machine learning-based signal processing techniques, good speech recognition accuracy is achieved (92.7% accuracy for phonemes, 85.6% for word pairs from the same viseme group, and 96.7% for sentences/phrases). Moreover, the reported silent speech interface demonstrates robustness against noises from both ambient environment and user’s daily motions. Finally, we illustrate the great potential in assistive technology and human-machine interactions through two proof-of-concept demonstrations –silent speech enabled smartphone assistant and drone control.