Traditional grading and sorting of fruits is a time-consuming process that demands skilled labor. So, the use of computer vision and machine learning methods provides feasible, cost-effective, and time-effective solutions to this specific issue. In this study, an efficient model is introduced for automatic fruit grade and quality classification. After image acquisition, augmentation is performed by utilizing zoom, height shift, width shift, rotation, and horizontal flip techniques. Image augmentation techniques improve the robustness of the proposed model by forming different and new examples to train the datasets. The proposed model performs a more precise classification if the dataset is sufficient and rich. In this model, a convolutional neural network (CNN) along with a long short-term memory (LSTM) network is employed for fruit grade and quality classification. In the CNN-LSTM network, a novel optimizer named MAdam is introduced to control the momentum and learning rate of every parameter for improving the model’s generalization ability and convergence rate. The MAdam optimizer adaptively adjusts the learning rate for each parameter, which, in turn, improves the stability of training and prevents overshooting. The numerical analysis reveals that this model attains superior accuracies of 99.24% and 98.99% on the pomegranate fruit dataset and union dataset, respectively. © The Institution of Engineers (India) 2024.