Deep feature voting: a semantic-driven and local context-aware approach for image classification

In the context of addressing new image classification tasks with insufficient training samples via pre-trained deep learning models, the methods based on the Bag-of-Deep-Visual-Words (BoDVW) model have achieved higher classification accuracy across various image classification tasks compared to directly using the new classification layer of the pre-trained model for classification. These methods perform a sequence of operations on the input image - deep feature extraction, feature encoding, and feature pooling - to obtain an image representation vector, which is then fed into classifiers for classification. However, they ignore two crucial aspects: the high-level semantic characteristics of deep features and their local context within the feature space, which limits the image classification performance. To address this issue, we propose a new image classification method with a unique workflow. Specifically, our method identifies low-entropy local regions in the feature space by constructing multiple decision trees, using the set of labelled deep features built from training images. For a given image, the voting vector of each deep feature from the image is calculated based on the category label distributions of the low-entropy local regions where it is located. This vector reflects the degree of support that the feature provides for the hypothesis that it belongs to each category. The voting vectors of all features are aggregated according to image regions of different sizes and positions to obtain the representation vector of the image. The representation vectors of testing images are input into Support Vector Machines (SVMs) trained using those of training images to predict their categories. Experimental results on six public datasets show that our method achieves higher classification accuracy by 0.07% to 3.6% (averaging at 0.8%) compared to two BoDVW methods, and by 0.1% to 10.69% (averaging at 2.69%) compared to directly using the new classification layer of the pre-trained model for classification. These results demonstrate the effectiveness of considering the high-level semantic characteristics of deep features and their local context within the feature space for image classification. Importantly, the unique workflow of our method opens up new potential avenues for improving classification performance. These include increasing the number of local regions where deep features primarily originate from one or a few image categories, improving the accuracy of low-entropy local region identification, and developing an end-to-end deep learning model based on this workflow. While maintaining classification accuracy comparable to recent works, our method offers notable potential for the advancement of the image classification field.