A Mouth Full of Words: Visually Consistent Acoustic Redubbing

Sarah Taylor; Barry-John Theobald; Iain Matthews

doi:10.1109/ICASSP.2015.7178903

A Mouth Full of Words: Visually Consistent Acoustic Redubbing

Sarah Taylor, Barry-John Theobald, Iain Matthews

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

6 Citations (Scopus)

115 Downloads (Pure)

Abstract

This paper introduces a method for automatic redubbing of video that exploits the many-to-many mapping of phoneme sequences to lip movements modelled as dynamic visemes [1]. For a given utterance, the corresponding dynamic viseme sequence is sampled to construct a graph of possible phoneme sequences that synchronize with the video. When composed with a pronunciation dictionary and language model, this produces a vast number of word sequences that are in sync with the original video, literally putting plausible words into the mouth of the speaker. We demonstrate that traditional, one-to-many, static visemes lack flexibility for this application as they produce significantly fewer word sequences. This work explores the natural ambiguity in visual speech and offers insight for automatic speech recognition and the importance of language modeling.

Original language	English
Title of host publication	2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)
Publisher	The Institute of Electrical and Electronics Engineers (IEEE)
Pages	4904-4908
ISBN (Electronic)	978-1-4673-6997-8
DOIs	https://doi.org/10.1109/ICASSP.2015.7178903
Publication status	Published - 6 Aug 2015
Event	International Conference on Acoustics, Speech and Signal Processing - Brisbane, Australia Duration: 19 Apr 2015 → 24 Apr 2015

Conference

Conference	International Conference on Acoustics, Speech and Signal Processing
Country/Territory	Australia
City	Brisbane
Period	19/04/15 → 24/04/15

Keywords

Audio-visual speech
dynamic visemes
acoustic redubbing

Access to Document

10.1109/ICASSP.2015.7178903

preprintAccepted author manuscript, 1.68 MB

Cite this

@inproceedings{e64ca4291e854e769884fa1f5f04cd03,

title = "A Mouth Full of Words: Visually Consistent Acoustic Redubbing",

abstract = "This paper introduces a method for automatic redubbing of video that exploits the many-to-many mapping of phoneme sequences to lip movements modelled as dynamic visemes [1]. For a given utterance, the corresponding dynamic viseme sequence is sampled to construct a graph of possible phoneme sequences that synchronize with the video. When composed with a pronunciation dictionary and language model, this produces a vast number of word sequences that are in sync with the original video, literally putting plausible words into the mouth of the speaker. We demonstrate that traditional, one-to-many, static visemes lack flexibility for this application as they produce significantly fewer word sequences. This work explores the natural ambiguity in visual speech and offers insight for automatic speech recognition and the importance of language modeling.",

keywords = "Audio-visual speech, dynamic visemes, acoustic redubbing",

author = "Sarah Taylor and Barry-John Theobald and Iain Matthews",

note = "Print ISSN: 1520-6149 Electronic ISSN: 2379-190X; International Conference on Acoustics, Speech and Signal Processing ; Conference date: 19-04-2015 Through 24-04-2015",

year = "2015",

month = aug,

day = "6",

doi = "10.1109/ICASSP.2015.7178903",

language = "English",

pages = "4904--4908",

booktitle = "2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)",

publisher = "The Institute of Electrical and Electronics Engineers (IEEE)",

address = "United States",

}

Taylor, S, Theobald, B-J & Matthews, I 2015, A Mouth Full of Words: Visually Consistent Acoustic Redubbing. in 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). The Institute of Electrical and Electronics Engineers (IEEE), pp. 4904-4908, International Conference on Acoustics, Speech and Signal Processing, Brisbane, Australia, 19/04/15. https://doi.org/10.1109/ICASSP.2015.7178903

A Mouth Full of Words: Visually Consistent Acoustic Redubbing. / Taylor, Sarah; Theobald, Barry-John; Matthews, Iain.

2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). The Institute of Electrical and Electronics Engineers (IEEE), 2015. p. 4904-4908.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - A Mouth Full of Words: Visually Consistent Acoustic Redubbing

AU - Taylor, Sarah

AU - Theobald, Barry-John

AU - Matthews, Iain

N1 - Print ISSN: 1520-6149 Electronic ISSN: 2379-190X

PY - 2015/8/6

Y1 - 2015/8/6

N2 - This paper introduces a method for automatic redubbing of video that exploits the many-to-many mapping of phoneme sequences to lip movements modelled as dynamic visemes [1]. For a given utterance, the corresponding dynamic viseme sequence is sampled to construct a graph of possible phoneme sequences that synchronize with the video. When composed with a pronunciation dictionary and language model, this produces a vast number of word sequences that are in sync with the original video, literally putting plausible words into the mouth of the speaker. We demonstrate that traditional, one-to-many, static visemes lack flexibility for this application as they produce significantly fewer word sequences. This work explores the natural ambiguity in visual speech and offers insight for automatic speech recognition and the importance of language modeling.

AB - This paper introduces a method for automatic redubbing of video that exploits the many-to-many mapping of phoneme sequences to lip movements modelled as dynamic visemes [1]. For a given utterance, the corresponding dynamic viseme sequence is sampled to construct a graph of possible phoneme sequences that synchronize with the video. When composed with a pronunciation dictionary and language model, this produces a vast number of word sequences that are in sync with the original video, literally putting plausible words into the mouth of the speaker. We demonstrate that traditional, one-to-many, static visemes lack flexibility for this application as they produce significantly fewer word sequences. This work explores the natural ambiguity in visual speech and offers insight for automatic speech recognition and the importance of language modeling.

KW - Audio-visual speech

KW - dynamic visemes

KW - acoustic redubbing

U2 - 10.1109/ICASSP.2015.7178903

DO - 10.1109/ICASSP.2015.7178903

M3 - Conference contribution

SP - 4904

EP - 4908

BT - 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

PB - The Institute of Electrical and Electronics Engineers (IEEE)

T2 - International Conference on Acoustics, Speech and Signal Processing

Y2 - 19 April 2015 through 24 April 2015

ER -