Inspecting steel surfaces is important to ensure steel quality. Numerous defect-detection methods have been developed for steel surfaces. However, they are primarily used for local defects, and their accuracy in detecting distributed defects is unsatisfactory because such defects are difficult to locate and have complex texture characteristics. To solve these issues, an improved random forest algorithm with optimal multi-feature-set fusion (OMFF-RF algorithm) is proposed for distributed defect recognition in this paper. The OMFF-RF algorithm includes the following three aspects. First, a histogram of oriented gradient (HOG) feature-set and a gray-level co-occurrence matrix (GLCM) feature-set are extracted and fused to describe local and global texture characteristics, respectively. Second, given the small number of samples of distributed defect images and the high dimensionality of the extracted feature-sets, a random forest algorithm is introduced to perform defect classification. Third, the feature-sets vary greatly in performance and dimensionality. To improve the fusion efficiency, OMFF-RF merges the HOG feature-set and the GLCM feature-set through a multi-feature-set fusion factor, which changes the number of decision trees that correspond to each feature-set in the RF algorithm. The OMFF factor is found by optimizing the fitting curve of the classification accuracy of the test set using a stepping multi-feature-set fusion factor. In experiments, the effectiveness of the proposed OMFF-RF was verified using 5 types of distributed defects collected from an actual steel production line. OMFF-RF achieved a recognition accuracy of 91%, a result superior to support vector machine (SVM) and conventional RF algorithms.