This study examined the contributions of the different components of the working memory (WM) model to a range of mathematical skills in children, using measures of WM function that did not involve numerical stimuli. A sample of 148 children (78 Year 3, mean age 8 years and 1 month, and 70 Year 5 pupils, mean age 9 years and 10 months) completed WM measures and age‐appropriate mathematics tests designed to assess four mathematical skills defined by the National Curriculum for England. Visuo‐spatial sketchpad and central executive, but not phonological loop, scores predicted unique variance in children’s curriculum‐based mathematical attainment but the relative contributions of each component did not vary much across the different skills. Subsequently, the mathematics data were re‐analysed using cluster analysis and new performance‐related mathematics factors were derived. All three components of WM predicted unique variance in these performance‐related skills, but revealed a markedly distinct pattern of associations across the two age groups. In particular, the data indicated a stronger role for the visuo‐spatial sketchpad in the younger children’s mathematics performance. We discuss our findings in terms of the importance of WM in the development of early mathematical ability.