FMUZZ: A Novel Greybox Fuzzing Approach based on Mutation Strategy Optimization with Byte Scheduling

Mutation-based greybox fuzzing is an efficient and widely used software testing technique, and its performance heavily depends on the mutation strategy. Existing solutions guide the seed mutation by using program-adaptive mutation strategies or constraint solving techniques. However, they disregard the characteristic that the execution information of seeds with similar behavior contains general strategies for solving specific constraints. In this paper, we propose the FMUZZ, a lightweight fuzzing approach based on mutation strategy optimization. FMUZZ first clusters the seeds based on their execution information into different seed groups and then learns the byte mutation scheduling strategies applicable to different program paths to improve efficiency in generating seeds that satisfy specific branch constraints. Meanwhile, FMUZZ removes the redundant seeds during the learning process by using the customized multi-objective optimization algorithm, thereby improving the efficiency of learning byte mutation scheduling strategies for different program paths. We test the effectiveness of FMUZZ on 9 real-world programs with the comparison of 3 state-of-the-art mutation-based fuzzers. Extensive experimental results show that compared to the benchmark fuzzers, FMUZZ achieves 8.9% higher branch coverage and outperforms 35.3% in discovering unique crashes on average.