Visually derived Wiener filters for speech enhancement

I. Almajai, Ben Milner

Research output: Contribution to journalArticlepeer-review

66 Citations (Scopus)


The aim of this work is to examine whether visual speech information can be used to enhance audio speech that has been contaminated by noise. First, an analysis of audio and visual speech features is made, which identifies the pair with highest audio-visual correlation. The study also reveals that higher audio-visual correlation exists within individual phoneme sounds rather than globally across all speech. This correlation is exploited in the proposal of a visually derived Wiener filter that obtains clean speech and noise power spectrum statistics from visual speech features. Clean speech statistics are estimated from visual features using a maximum a posteriori framework that is integrated within the states of a network of hidden Markov models to provide phoneme localization. Noise statistics are obtained through a novel audio-visual voice activity detector which utilizes visual speech features to make robust speech/nonspeech classifications. The effectiveness of the visually derived Wiener filter is evaluated subjectively and objectively and is compared with three different audio-only enhancement methods over a range of signal-to-noise ratios.
Original languageEnglish
Pages (from-to)1642-1651
Number of pages10
JournalIEEE Transactions on Audio, Speech, and Language Processing
Issue number6
Publication statusPublished - Aug 2011

Cite this