Reduced gaze following has been previously associated with lower language scores in children with autism spectrum disorder (ASD). Here we use eye-tracking to investigate whether attention distribution during a word learning task requiring gaze following associates with later developmental and clinical outcomes, in a population of infants at familial risk for ASD. Fifteen-month-old infants (n = 124; n = 101 with familial risk) watched an actress repeatedly gaze towards and label one of two objects present in front of her. We show that infants who later developed ASD followed gaze as frequently as typically developing peers but tended to spend less time engaged with either object. Moreover, more time spent on faces and less on objects was associated with lower concurrent and later verbal and non-verbal abilities, in the whole group and in the high-risk group only. None of the groups showed evidence for word learning. Thus, reduced engagement with objects rather than poor gaze following, associates with poorer language and also more broadly with developmental level in this population.