The growing worldwide population has led to an increase in traffic congestion. The problem of traffic congestion has brought numerous issues, including increased fuel consumption, higher air pollution, and substantial financial expenditures from delays that have a detrimental impact on daily commutes and enterprises. Long-term traffic jams can increase the stress levels of commuters and the likelihood of car accidents. Determining workable solutions for controlling and reducing traffic congestion has now become essential. This study uses convolutional neural networks (CNNs) combined with Incremental Extreme Learning Machine (IELM) to address the problem of traffic congestion. To implement this technique for traffic congestion, the classification process is carried out in high, medium, or low traffic. This dual technique provides an extensive understanding of traffic circumstances by determining the presence of traffic and its severity. Two datasets, i.e., Mendeley’s public dataset and a customized version of Mendeley’s dataset, were used to train the proposed model. The multi-layered structure of CNN effectively recognizes many aspects of input traffic images and generates a feature vector. This feature vector is then passed to IELM for classification. The fully connected layer of CNN is replaced with IELM to avoid the lengthy backpropagation process. This allows the proposed model to quickly adjust to new data and change with traffic circumstances without requiring extensive retraining. As a result, it reduces the training period and makes the accuracy of the proposed model to 99.18%. A detailed comparison of the proposed approach is done with different machine-learning approaches combined with CNN. Experimental results show the effectiveness of the proposed approach over the state-of-the-art approaches.