A Malware Detection Method of Code Texture Visualization Based on an Improved Faster RCNN Combining Transfer Learning

Today, with the continuous promotion and development of IoT and 5G technology, Cyberspace has become an important pillar of economic and social development, and also a foundational domain of national security. Cyberspace security is attracting more and more attention. Therefore, detecting malware and its variants is of great significance to Cyberspace. However, the increasing sophistication of malicious variants, such as encryption, polymorphism and obfuscation, makes it more difficult to identified malware effectively. In this article, a malware detection method of code texture visualization based on an improved Faster RCNN (Region-Convolutional Neural Networks) combining transfer learning is proposed. We utilize visualization technology to map malicious code into corresponding images with typical texture features, and realize the classification of malware. Firstly, in order to quickly acquire and locate the representative texture of malware, we adopt CNN to extract the global and deeper features of malicious code images. Then with RPN (Region Proposal Network) we generate the target image frame, which is used to locate the core texture of malware file (.text file), to realize the accurate positioning of malicious features. Secondly, we preprocess and train Faster RCNN model with ImageNet set, and then transfer the model to the malware classification model to accelerate the convergence of the first model and promote generation performance. Thirdly, we construct an improved objective function in which a novel multi-label of classification proportion is added to solve the problem that the texture change of ".text" section and other sections in malicious code image is not obvious after transfer learning. We collect code samples of six malware families from Kaggle platform, and compared the experimental results before and after transfer. The results show that the novel method can accelerate the convergence of loss function, and obtain higher accuracy (92.8%), lower FPR (6.8%) and better P-R (precision-recall) curve.