Objects can be described in terms of low-level (e.g., boundaries) and high-level properties (e.g., object semantics). While recent behavioral findings suggest that the influence of semantic relatedness between objects on attentional allocation can be independent of task-relevance, the underlying neural substrate of semantic influences on attention remains ill-defined. Here, we employ behavioral and functional magnetic resonance imaging (fMRI) measures to uncover the mechanism by which semantic information increases visual processing efficiency. We demonstrate that the strength of the semantic relatedness signal decoded from the left inferior frontal gyrus (IFG): (i) influences attention, producing behavioral semantic benefits; (ii) biases spatial attention maps in the intraparietal sulcus (IPS), subsequently modulating early visual cortex (EVC) activity; (iii) directly predicts the magnitude of behavioral semantic benefit. Together, these results identify a specific mechanism driving task-independent semantic influences on attention.