Research on outlier detection in CTD conductivity data based on cubic spline fitting

被引:0
|
作者
Yu, Long [1 ,2 ,3 ,4 ,5 ]
Sun, Jia [2 ,3 ,4 ,5 ]
Guo, Yanliang [2 ,3 ,4 ,5 ]
Zhang, Baohua [5 ]
Yang, Guangbing [2 ,3 ,4 ,5 ]
Chen, Liang [2 ,3 ,4 ,5 ]
Ju, Xia [2 ,3 ,4 ,5 ]
Yang, Fanlin [1 ]
Xiong, Xuejun [2 ,3 ,4 ,5 ]
Lv, Xianqing [6 ,7 ]
机构
[1] Shandong Univ Sci & Technol, Coll Geodesy & Geomat, Qingdao, Peoples R China
[2] Minist Nat Resources, Inst Oceanog 1, Qingdao, Peoples R China
[3] Minist Nat Resources, Key Lab Marine Sci & Numer Modeling, Qingdao, Peoples R China
[4] Pilot Natl Lab Marine Sci & Technol, Lab Reg Oceanog & Numer Modeling, Qingdao, Peoples R China
[5] Shandong Key Lab Marine Sci & Numer Modeling, Qingdao, Peoples R China
[6] Ocean Univ China, Frontier Sci Ctr Deep Ocean Multispheres & Earth S, Qingdao, Peoples R China
[7] Ocean Univ China, Phys Oceanog Lab, Qingdao, Peoples R China
基金
中国国家自然科学基金;
关键词
CTD (conductivity-temperature-depth); outlier detection; cubic spline fitting; independent points; optimal number of iterations; OCEAN; PARAMETERS;
D O I
10.3389/fmars.2022.1030980
中图分类号
X [环境科学、安全科学];
学科分类号
08 ; 0830 ;
摘要
Outlier detection is the key to the quality control of marine survey data. For the detection of outliers in Conductivity-Temperature-Depth (CTD) data, previous methods, such as the Wild Edit method and the Median Filter Combined with Maximum Deviation method, mostly set a threshold based on statistics. Values greater than the threshold are treated as outliers, but there is no clear specification for the selection of threshold, thus multiple attempts are required. The process is time-consuming and inefficient, and the results have high false negative and positive rates. In response to this problem, we proposed an outlier detection method in CTD conductivity data, based on a physical constraint, the continuity of seawater. The method constructs a cubic spline fitting function based on the independent points scheme and the cubic spline interpolation to fit the conductivity data. The maximum fitting residual points will be flagged as outliers. The fitting stops when the optimal number of iterations is reached, which is automatically obtained by the minimum value of the sequence of maximum fitting residuals. Verification of the accuracy and stability of the method by means of examples proves that it has a lower false negative rate (17.88%) and false positive rate (0.24%) than other methods. Indeed, rates for the Wild Edit method are 56.96% and 2.19%, while for the Median Filter Combined with Maximum Deviation method rates are 23.28% and 0.31%. The Cubic Spline Fitting method is simple to operate, the result is clear and definite, better solved the problem of conductivity outliers detection.
引用
收藏
页数:11
相关论文
共 50 条
  • [21] Displacement Response Reconstruction of Slender Flexible Structures Based on Cubic Spline Fitting Method
    Qing-hua Han
    Ye-xuan Ma
    Xin-xin Feng
    Wan-hai Xu
    China Ocean Engineering, 2019, 33 : 226 - 236
  • [22] Robust Fitting of a Wrapped Normal Model to Multivariate Circular Data and Outlier Detection
    Greco, Luca
    Saraceno, Giovanni
    Agostinelli, Claudio
    STATS, 2021, 4 (02): : 454 - 471
  • [23] Simple method for outlier detection in fitting experimental data under interval error
    Zhilin, S. I.
    CHEMOMETRICS AND INTELLIGENT LABORATORY SYSTEMS, 2007, 88 (01) : 60 - 68
  • [24] Research on outlier detection for high dimensional data stream
    Yu, Liping
    Li, Yunfei
    Jia, Juncheng
    PROCEEDINGS OF THE 2016 INTERNATIONAL CONFERENCE ON ARTIFICIAL INTELLIGENCE AND ENGINEERING APPLICATIONS, 2016, 63 : 395 - 398
  • [25] Research on Outlier Detection Methods for Dam Monitoring Data Based on Post-Data Classification
    Mao, Yanpian
    Li, Jiachen
    Qi, Zhiyong
    Yuan, Jin
    Xu, Xiaorong
    Jin, Xinxin
    Du, Xuhuang
    BUILDINGS, 2024, 14 (09)
  • [26] Abnormal Detection Research Based on Outlier Mining
    Zhang, Lei
    Liu, Changjiang
    Chen, Yong
    Lao, Shaowen
    2018 11TH INTERNATIONAL CONFERENCE ON INTELLIGENT COMPUTATION TECHNOLOGY AND AUTOMATION (ICICTA 2018), 2018, : 5 - 7
  • [27] Medical CT Image Preprocessing Based on Edge Detection and Spline Fitting
    Chi, Jing
    Wu, Xiaoming
    Zhang, Caiming
    2008 IEEE INTERNATIONAL SYMPOSIUM ON IT IN MEDICINE AND EDUCATION, VOLS 1 AND 2, PROCEEDINGS, 2008, : 709 - +
  • [28] Method of Boundary Detection Based on B-spline Curve Fitting
    Lu Shaofang
    Wang Kun
    Li Yufang
    Chen Yuncong
    2011 30TH CHINESE CONTROL CONFERENCE (CCC), 2011, : 3248 - 3253
  • [29] Three-Dimensional Curve Fitting Based on Cubic B-Spline Interpolation Curve
    Wan, Yan
    Yin, Suna
    2014 7TH INTERNATIONAL CONGRESS ON IMAGE AND SIGNAL PROCESSING (CISP 2014), 2014, : 765 - 770
  • [30] Data Fitting About BRDF Based on B-spline Function
    Wang, Zhan
    Yu, Jun
    Su, Bida
    Xie, Xiaodan
    PROCEEDINGS OF THE FOURTH INTERNATIONAL CONFERENCE OF MODELLING AND SIMULATION (ICMS2011), VOL 1, 2011, : 399 - 402