We explore a metric-learning approach to create representations of sorghum images grown in a field setting. We train a convolutional neural network to embed images so that images from the same variety have similar features and images of different varieties have different features. We that these features are good at discriminating unseen cultivars, can be used to predict standard phenotypes (height and leaf length and width), and can be used to predict presence or absence of genetic mutations. We evaluate these results using TERRA-REF data from field-scale trials of hundreds of varieties of sorghum. This demonstrates an end-to-end solution for creating useful image phenotypes in