Research on outlier detection in CTD conductivity data based on cubic spline fitting

被引:0
|
作者
Yu, Long [1 ,2 ,3 ,4 ,5 ]
Sun, Jia [2 ,3 ,4 ,5 ]
Guo, Yanliang [2 ,3 ,4 ,5 ]
Zhang, Baohua [5 ]
Yang, Guangbing [2 ,3 ,4 ,5 ]
Chen, Liang [2 ,3 ,4 ,5 ]
Ju, Xia [2 ,3 ,4 ,5 ]
Yang, Fanlin [1 ]
Xiong, Xuejun [2 ,3 ,4 ,5 ]
Lv, Xianqing [6 ,7 ]
机构
[1] Shandong Univ Sci & Technol, Coll Geodesy & Geomat, Qingdao, Peoples R China
[2] Minist Nat Resources, Inst Oceanog 1, Qingdao, Peoples R China
[3] Minist Nat Resources, Key Lab Marine Sci & Numer Modeling, Qingdao, Peoples R China
[4] Pilot Natl Lab Marine Sci & Technol, Lab Reg Oceanog & Numer Modeling, Qingdao, Peoples R China
[5] Shandong Key Lab Marine Sci & Numer Modeling, Qingdao, Peoples R China
[6] Ocean Univ China, Frontier Sci Ctr Deep Ocean Multispheres & Earth S, Qingdao, Peoples R China
[7] Ocean Univ China, Phys Oceanog Lab, Qingdao, Peoples R China
基金
中国国家自然科学基金;
关键词
CTD (conductivity-temperature-depth); outlier detection; cubic spline fitting; independent points; optimal number of iterations; OCEAN; PARAMETERS;
D O I
10.3389/fmars.2022.1030980
中图分类号
X [环境科学、安全科学];
学科分类号
08 ; 0830 ;
摘要
Outlier detection is the key to the quality control of marine survey data. For the detection of outliers in Conductivity-Temperature-Depth (CTD) data, previous methods, such as the Wild Edit method and the Median Filter Combined with Maximum Deviation method, mostly set a threshold based on statistics. Values greater than the threshold are treated as outliers, but there is no clear specification for the selection of threshold, thus multiple attempts are required. The process is time-consuming and inefficient, and the results have high false negative and positive rates. In response to this problem, we proposed an outlier detection method in CTD conductivity data, based on a physical constraint, the continuity of seawater. The method constructs a cubic spline fitting function based on the independent points scheme and the cubic spline interpolation to fit the conductivity data. The maximum fitting residual points will be flagged as outliers. The fitting stops when the optimal number of iterations is reached, which is automatically obtained by the minimum value of the sequence of maximum fitting residuals. Verification of the accuracy and stability of the method by means of examples proves that it has a lower false negative rate (17.88%) and false positive rate (0.24%) than other methods. Indeed, rates for the Wild Edit method are 56.96% and 2.19%, while for the Median Filter Combined with Maximum Deviation method rates are 23.28% and 0.31%. The Cubic Spline Fitting method is simple to operate, the result is clear and definite, better solved the problem of conductivity outliers detection.
引用
收藏
页数:11
相关论文
共 50 条
  • [31] Outlier detection and robust plane fitting for building roof extraction from LiDAR data
    Dey, Emon Kumar
    Awrangjeb, Mohammad
    Stantic, Bela
    INTERNATIONAL JOURNAL OF REMOTE SENSING, 2020, 41 (16) : 6325 - 6354
  • [32] Research on smoothing support vector regression based on cubic spline interpolation
    Ren, Bin
    Cheng, LiangLun
    INTERNATIONAL CONFERENCE ON IMAGE PROCESSING AND PATTERN RECOGNITION IN INDUSTRIAL ENGINEERING, 2010, 7820
  • [33] Data processing method of noise logging based on cubic spline interpolation
    Hao, Wu
    Rui, Deng
    Song, Linghu
    Yang Ruixiang
    Zhang Jinhai
    Cao Juan
    APPLIED MATHEMATICS AND NONLINEAR SCIENCES, 2021, 6 (01) : 93 - 102
  • [34] Interpolation method for lidar data visualization based on cubic spline function
    Chen, Hao
    Hua, Dengxin
    Zhang, Yikun
    Yan, Qing
    Li, Shichun
    Yi Qi Yi Biao Xue Bao/Chinese Journal of Scientific Instrument, 2013, 34 (04): : 831 - 837
  • [35] Distance-based Outlier Detection in Data Streams
    Tran, Luan
    Fan, Liyue
    Shahabi, Cyrus
    PROCEEDINGS OF THE VLDB ENDOWMENT, 2016, 9 (12): : 1089 - 1100
  • [36] Distance-based outlier detection on uncertain data
    Yu, Hao
    Wang, Bin
    Xiao, Gang
    Yang, Xiaochun
    Jisuanji Yanjiu yu Fazhan/Computer Research and Development, 2010, 47 (03): : 474 - 484
  • [37] SVDD-based outlier detection on uncertain data
    Bo Liu
    Yanshan Xiao
    Longbing Cao
    Zhifeng Hao
    Feiqi Deng
    Knowledge and Information Systems, 2013, 34 : 597 - 618
  • [38] Big Data Outlier Detection Algorithm Based on Grid
    Guo Wei-Wei
    Liu Feng
    2018 11TH INTERNATIONAL CONFERENCE ON INTELLIGENT COMPUTATION TECHNOLOGY AND AUTOMATION (ICICTA 2018), 2018, : 274 - 277
  • [39] SVDD-based outlier detection on uncertain data
    Liu, Bo
    Xiao, Yanshan
    Cao, Longbing
    Hao, Zhifeng
    Deng, Feiqi
    KNOWLEDGE AND INFORMATION SYSTEMS, 2013, 34 (03) : 597 - 618
  • [40] Three-dimensional reconstruction of light field based on cubic B-spline curve fitting
    Feng, Wei
    Fan, Jiahao
    Liu, Qianqian
    Wang, Henghui
    Wang, Xuanze
    OPTICS COMMUNICATIONS, 2024, 552