Audio-to-Visual Speech Conversion using Deep Neural Networks

Sarah Taylor; Akihiro Kato; Ben Milner; Iain Matthews

doi:10.21437/Interspeech.2016-483

Audio-to-Visual Speech Conversion using Deep Neural Networks

Sarah Taylor, Akihiro Kato, Ben Milner, Iain Matthews

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

25 Citations (Scopus)

112 Downloads (Pure)

Abstract

We study the problem of mapping from acoustic to visual speech with the goal of generating accurate, perceptually natural speech animation automatically from an audio speech signal.
We present a sliding window deep neural network that learns a mapping from a window of acoustic features to a window of visual features from a large audio-visual speech dataset. Overlapping
visual predictions are averaged to generate continuous, smoothly varying speech animation. We outperform a baseline HMM inversion approach in both objective and subjective evaluations
and perform a thorough analysis of our results.

Original language	English
Title of host publication	Proceedings of the Interspeech Conference 2016
Publisher	International Speech Communication Association
Pages	1482-1486
Number of pages	5
DOIs	https://doi.org/10.21437/Interspeech.2016-483
Publication status	Published - Sept 2016
Event	Interspeech 2016 - San Francisco, United States Duration: 8 Sept 2016 → 12 Sept 2016

Conference

Conference	Interspeech 2016
Country/Territory	United States
City	San Francisco
Period	8/09/16 → 12/09/16

Access to Document

10.21437/Interspeech.2016-483

Accepted manuscriptAccepted author manuscript, 650 KB

Ben Milner
- B.Milneruea.acuk
- School of Computing Sciences - Senior Lecturer
- Data Science and AI - Member
- Interactive Graphics and Audio - Member
- Smart Emerging Technologies - Member
Person: Research Group Member, Academic, Teaching & Research

Cite this

@inproceedings{dd9029f4cc2c4dc3b69563a52bd13348,

title = "Audio-to-Visual Speech Conversion using Deep Neural Networks",

abstract = "We study the problem of mapping from acoustic to visual speech with the goal of generating accurate, perceptually natural speech animation automatically from an audio speech signal.We present a sliding window deep neural network that learns a mapping from a window of acoustic features to a window of visual features from a large audio-visual speech dataset. Overlappingvisual predictions are averaged to generate continuous, smoothly varying speech animation. We outperform a baseline HMM inversion approach in both objective and subjective evaluationsand perform a thorough analysis of our results.",

author = "Sarah Taylor and Akihiro Kato and Ben Milner and Iain Matthews",

year = "2016",

month = sep,

doi = "10.21437/Interspeech.2016-483",

language = "English",

pages = "1482--1486",

booktitle = "Proceedings of the Interspeech Conference 2016",

publisher = "International Speech Communication Association",

note = "Interspeech 2016 ; Conference date: 08-09-2016 Through 12-09-2016",

}

TY - GEN

T1 - Audio-to-Visual Speech Conversion using Deep Neural Networks

AU - Taylor, Sarah

AU - Kato, Akihiro

AU - Milner, Ben

AU - Matthews, Iain

PY - 2016/9

Y1 - 2016/9

N2 - We study the problem of mapping from acoustic to visual speech with the goal of generating accurate, perceptually natural speech animation automatically from an audio speech signal.We present a sliding window deep neural network that learns a mapping from a window of acoustic features to a window of visual features from a large audio-visual speech dataset. Overlappingvisual predictions are averaged to generate continuous, smoothly varying speech animation. We outperform a baseline HMM inversion approach in both objective and subjective evaluationsand perform a thorough analysis of our results.

AB - We study the problem of mapping from acoustic to visual speech with the goal of generating accurate, perceptually natural speech animation automatically from an audio speech signal.We present a sliding window deep neural network that learns a mapping from a window of acoustic features to a window of visual features from a large audio-visual speech dataset. Overlappingvisual predictions are averaged to generate continuous, smoothly varying speech animation. We outperform a baseline HMM inversion approach in both objective and subjective evaluationsand perform a thorough analysis of our results.

U2 - 10.21437/Interspeech.2016-483

DO - 10.21437/Interspeech.2016-483

M3 - Conference contribution

SP - 1482

EP - 1486

BT - Proceedings of the Interspeech Conference 2016

PB - International Speech Communication Association

T2 - Interspeech 2016

Y2 - 8 September 2016 through 12 September 2016

ER -

Audio-to-Visual Speech Conversion using Deep Neural Networks

Abstract

Conference

Access to Document

Profiles

Ben Milner

Cite this