The UEA Digital Humans entry to the GENEA Challenge 2023

Jonathan Windle; Iain Matthews; Ben Milner; Sarah Taylor

doi:10.1145/3577190.3616116

The UEA Digital Humans entry to the GENEA Challenge 2023

Jonathan Windle (Lead Author), Iain Matthews, Ben Milner, Sarah Taylor

Research output: Contribution to conference › Paper › peer-review

5 Citations (Scopus)

Abstract

This paper describes our entry to the GENEA (Generation and Evaluation of Non-verbal Behaviour for Embodied Agents) Challenge 2023. This year's challenge focuses on generating gestures in a dyadic setting - predicting a main-agent's motion from the speech of both the main-agent and an interlocutor. We adapt a Transformer-XL architecture for this task by adding a cross-attention module that integrates the interlocutor's speech with that of the main-agent. Our model is conditioned on speech audio (encoded using PASE+), text (encoded using FastText) and a speaker identity label, and is able to generate smooth and speech appropriate gestures for a given identity. We consider the GENEA Challenge user study results and present a discussion of our model strengths and where improvements can be made.

Original language	English
Pages	802-810
Number of pages	9
DOIs	https://doi.org/10.1145/3577190.3616116
Publication status	Published - 9 Oct 2023
Event	INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION - Duration: 9 Oct 2023 → 13 Oct 2023

Conference

Conference	INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION
Period	9/10/23 → 13/10/23

Keywords

3D pose prediction
Cross-Attention
Self-Attention
Speech-to-gesture
Transformer-XL
gesture generation

Access to Document

10.1145/3577190.3616116

Cite this

@conference{5af0d50624de4775a42e3fd1d6f4347d,

title = "The UEA Digital Humans entry to the GENEA Challenge 2023",

abstract = "This paper describes our entry to the GENEA (Generation and Evaluation of Non-verbal Behaviour for Embodied Agents) Challenge 2023. This year's challenge focuses on generating gestures in a dyadic setting - predicting a main-agent's motion from the speech of both the main-agent and an interlocutor. We adapt a Transformer-XL architecture for this task by adding a cross-attention module that integrates the interlocutor's speech with that of the main-agent. Our model is conditioned on speech audio (encoded using PASE+), text (encoded using FastText) and a speaker identity label, and is able to generate smooth and speech appropriate gestures for a given identity. We consider the GENEA Challenge user study results and present a discussion of our model strengths and where improvements can be made.",

keywords = "3D pose prediction, Cross-Attention, Self-Attention, Speech-to-gesture, Transformer-XL, gesture generation",

author = "Jonathan Windle and Iain Matthews and Ben Milner and Sarah Taylor",

year = "2023",

month = oct,

day = "9",

doi = "10.1145/3577190.3616116",

language = "English",

pages = "802--810",

note = "INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION ; Conference date: 09-10-2023 Through 13-10-2023",

}

TY - CONF

T1 - The UEA Digital Humans entry to the GENEA Challenge 2023

AU - Windle, Jonathan

AU - Matthews, Iain

AU - Milner, Ben

AU - Taylor, Sarah

PY - 2023/10/9

Y1 - 2023/10/9

N2 - This paper describes our entry to the GENEA (Generation and Evaluation of Non-verbal Behaviour for Embodied Agents) Challenge 2023. This year's challenge focuses on generating gestures in a dyadic setting - predicting a main-agent's motion from the speech of both the main-agent and an interlocutor. We adapt a Transformer-XL architecture for this task by adding a cross-attention module that integrates the interlocutor's speech with that of the main-agent. Our model is conditioned on speech audio (encoded using PASE+), text (encoded using FastText) and a speaker identity label, and is able to generate smooth and speech appropriate gestures for a given identity. We consider the GENEA Challenge user study results and present a discussion of our model strengths and where improvements can be made.

AB - This paper describes our entry to the GENEA (Generation and Evaluation of Non-verbal Behaviour for Embodied Agents) Challenge 2023. This year's challenge focuses on generating gestures in a dyadic setting - predicting a main-agent's motion from the speech of both the main-agent and an interlocutor. We adapt a Transformer-XL architecture for this task by adding a cross-attention module that integrates the interlocutor's speech with that of the main-agent. Our model is conditioned on speech audio (encoded using PASE+), text (encoded using FastText) and a speaker identity label, and is able to generate smooth and speech appropriate gestures for a given identity. We consider the GENEA Challenge user study results and present a discussion of our model strengths and where improvements can be made.

KW - 3D pose prediction

KW - Cross-Attention

KW - Self-Attention

KW - Speech-to-gesture

KW - Transformer-XL

KW - gesture generation

UR - http://www.scopus.com/inward/record.url?scp=85170504213&partnerID=8YFLogxK

U2 - 10.1145/3577190.3616116

DO - 10.1145/3577190.3616116

M3 - Paper

SP - 802

EP - 810

T2 - INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION

Y2 - 9 October 2023 through 13 October 2023

ER -

The UEA Digital Humans entry to the GENEA Challenge 2023

Abstract

Conference

Keywords

Access to Document

Other files and links

Cite this