Data-driven Artificial Intelligence systems are playing an increasingly significant role in the advancement of precision agriculture. Currently, precision sprayers lack fully automated methods to evaluate effectiveness of their operation, e.g. whether spray has landed on target weeds. In this paper, using an agricultural spot spraying system images were collected from an RGB camera to locate spray deposits on weeds or lettuces. We present an interpretable deep learning pipeline to identify spray deposits on lettuces and weeds without using existing methods such as tracers or water sensitive papers. We implement a novel stratification and sampling methodology to improve results from a baseline. Using a binary classification head after transfer learning networks, spray deposits are identified with over 90% Area Under the Receiver Operating Characteristic (AUROC). This work offers a data-driven approach for an automated evaluation methodology for the effectiveness of precision sprayers.