Detection for Approximately Duplicate Records Based on Fuzzy Comprehensive Evaluation

To solve the problem of attribute weight determination in the approximately duplicate records, we put forward a method based on fuzzy comprehensive evaluation to get attribute weight in data set. We first perform an analysis of the composition factors of attribute. Then we carry out an evaluation of their rank. Finally, we make a determination of the attribute weight using the fuzzy comprehensive evaluation method, on the basis of which the approximately duplicate records are detected. Theoretical analysis and experimental results show that the method can objectively determine all attributes weight, and effectively detect the approximately duplicate records in massive data set.