Object detection algorithms have heavily relied on deep learning techniques to estimate the count of grapes as the resulting quality of grapes is directly correlated to its yield. With temporal analytics, early actions and logistical organizing can be performed to maintain the quality of grapes. However, the issue with using these object detection algorithms is that they are limited to counting only the visible grapes, thus omitting hidden grapes, and in turn affecting the true estimate of grape yield. Many grapes are occluded because of either the compactness of the bunch cluster, or due to canopy interference. Therefore, models need to be able to estimate the unseen berries in order to give a near true yield estimate. An end-to-end framework is proposed in which the grape clusters are first segmented using a deep learning model, after which the extracted candidate regions of grape clusters are fed to a CNN regression model that can estimate the count of berries by incorporating a correction factor. A new dataset is also proposed which consists imagery of grape clusters, along with their ground truth values of grape count and weight. The proposed framework will also be tested using three open-source datasets and will encourage future research in determining which features of grapes can be leveraged to correct grape counting algorithms and produce higher accurate results.