Generating intelligible audio speech from visual speech

Thomas Le Cornu, Ben P. Milner

Research output: Contribution to journalArticlepeer-review

36 Citations (Scopus)
24 Downloads (Pure)


This work is concerned with generating intelligible audio speech from a video of a person talking. Regression and classification methods are proposed first to estimate static spectral envelope features from active appearance model (AAM) visual features. Two further methods are then developed to incorporate temporal information into the prediction - a feature-level method using multiple frames and a model-level method based on recurrent neural networks. Speech excitation information is not available from the visual signal, so methods to artificially generate aperiodicity and fundamental frequency are developed. These are combined within the STRAIGHT vocoder to produce a speech signal. The various systems are optimised through objective tests before applying subjective intelligibility tests that determine a word accuracy of 85% from a set of human listeners on the GRID audio-visual speech database. This compares favourably with a previous regression-based system that serves as a baseline which achieved a word accuracy of 33%.
Original languageEnglish
Pages (from-to)1447-1457
Number of pages11
JournalIEEE Transactions on Audio, Speech, and Language Processing
Issue number9
Early online date15 Jun 2017
Publication statusPublished - Sep 2017

Cite this