For some nations, including India, agriculture is the main source of income. Crop/plant infections are important factors in reduced production quantity and quality, which leads to economic loss. Therefore, it is essential to detect crop diseases as soon as possible. The symptoms of plant diseases can be seen in several parts of a plant. Plant leaves, however, are frequently used to diagnose infections. Therefore, in this paper, we focus on using the deep learning model to automatically detect leaf disease. The pre-processing, feature extraction, feature selection, and classification phases constituents of the presented model. The images of the paddy leaf are first converted into an RGB color model, and the noise in the green band is subsequently removed using a Gaussian filter. The green band is then used to extract the texture and color features. Important features are selected using a combination of machine learning and optimization algorithms after feature extraction. Here, support vector machine-recursive feature elimination (SV-RFE) and an adaptive red fox algorithm (ARFA) are used to initially select the features. Then, the common features are selected. The selected features are given to the modified bi-long short-term memory (MBi-LSTM) classifier to classify an image as Blast disease, Bacterial Leaf Blight disease, Tungro, or normal image. The experimental results shows, the proposed method achieves the better accuracy of 97.16% which is high compared to state-of-the-art-techniques.