Reconstructing the three-dimensional (3D) size and shape distribution of randomly oriented grains using only images of cross sections remains an important challenge. Even for ellipsoids, a solution is only possible when they are solids of revolution, and may still be numerically unstable. Here we show that crystallographic orientation data, for example from electron back-scatter diffraction (EBSD), provides enough additional information to obtain moments of the 3D grain distribution, provided grain shapes can be assumed to align with crystal axes. We show that this moment method can give an average 3D grain size and shape (with error estimate) which is rigorous for ellipsoids and a good approximation for cuboidal grains, indicating that it may be a useful technique for polycrystalline materials in general. High throughput image analysis and EBSD now make the necessary sample sizes practical. We illustrate by applying the method to a basaltic rock specimen.