The Relative Importance of Language, Gaze, and Gesture in Deictic Reference

When people communicate, they use a combination of modalities—speech, gesture, and eye gaze—to engage and transmit information to an addressee. Spatial deictic communication is a paradigmatic case, with spatial demonstratives (this/that) frequently co-occurring with eye gaze and pointing gestures to draw the attention of an addressee to an object location (e.g., this cup, that chair). Yet the effectiveness of these individual modalities in guiding attention has not been established. In two experiments, we manipulated pointing, gazing, and spatial demonstratives to establish their relative and combined effectiveness in directing attention to a specific referent. Participants saw an image (Experiments 1 and 2) or a short video clip (Experiment 2) with a person (agent) sitting behind a table, describing (this, that), gazing, and/or pointing at the items placed proximally or distally relative to the agent. All three modalities individually affected which of the two objects participants thought the person in the picture was referring to. However, pointing was the dominant cue to referent choice, with demonstratives on their own acting as a relatively weak spatial deictic cue. Overall, the effect of spatial demonstratives (Experiments 1 and 2) and gaze (Experiment 2) on attention to a referent was enhanced when coupled with pointing, both when targeting the distal and proximal positions. The results help to illuminate why spatial deictic communication is usually multimodal, with individual modalities contributing different communicative functions in the act of spatial deixis.