Cesarean Section (C-Section) is an operation whereby a baby is brought to the world through a cut made on the abdomen and uterus. The factors that lead to C-Section includes premature rupture of the membranes, having an abnormal fetal heart rate, having previous C-Sections, or previous surgeries on the uterus, human immunodeficiency virus, having undergone cervical cerclage, or having sustained accidental injuries. If not predicted in the early stages, the mother and especially the baby faces many risks. Timely intervention and monitoring are made possible by early forcasting the mode of delivery. Although Deep Learning (DL) models aid in medical decision making, practitioners continue to have reservations about the accuracy and transparency in identifying important contributing factors. Our study presents an explainable DL system that utilizes a health indicators dataset comprising data gathered in India between 2019 and 2021 to forecast C-Section deliveries in the early stages. In order to solve class imbalance issues, the dataset is preprocessed utilizing encoding of categorical features, Proximity Weighted Synthetic oversampling, and a point biserial correlation coefficient approach to identify the highly correlated 65 features. A number of measures are used to assess the proposed Gated Highway Multi-layer-perceptron (GHiM) model, including accuracy, F1-score, recall, precision, MCC, loss, and execution time. With its accuracy of 0.866, F1-score of 0.863, and MCC of 0.732, the GHiM model outperformed models such as GRU, MLP, and HighwayNet by 2.5-3.5% Due to the complex structure of DL models, they are prone to over-fitting which is mitigated by training the GHiM using stratified 10-fold cross validation. Additionally, the most important characteristics in the categorization of C-Sections and normal births are determined by applying SHapley Additive exPlanation. This framework demonstrates that the GHiM model offers good accuracy and interpretability, and can enhance clinical decision support systems for C-Section delivery forecasting.