We can learn about the affective content of the environment by observing the behavior of others; their responses to stimuli tend to be appropriate to the context. To investigate the impact of observing such appropriate, compared with inappropriate, behaviors, we developed a novel behavioral task where participants observed different faces reacting to emotional scenes. We found that affective categorization of a scene was facilitated when it was presented alongside an appropriate facial expression (Experiment 1). Further, we observed that several brain areas in the right hemisphere—the putamen, insula, orbitofrontal cortex, and dorsal anterior cingulate cortex—were more activate when viewing faces that were previously observed emoting inappropriately (Experiment 2). We contend that these areas form a network that codes for the retrieval of affective conflict information generated by observing individuals producing inappropriate emotions.