Research on outlier detection in CTD conductivity data based on cubic spline fitting

被引:0
|
作者
Yu, Long [1 ,2 ,3 ,4 ,5 ]
Sun, Jia [2 ,3 ,4 ,5 ]
Guo, Yanliang [2 ,3 ,4 ,5 ]
Zhang, Baohua [5 ]
Yang, Guangbing [2 ,3 ,4 ,5 ]
Chen, Liang [2 ,3 ,4 ,5 ]
Ju, Xia [2 ,3 ,4 ,5 ]
Yang, Fanlin [1 ]
Xiong, Xuejun [2 ,3 ,4 ,5 ]
Lv, Xianqing [6 ,7 ]
机构
[1] Shandong Univ Sci & Technol, Coll Geodesy & Geomat, Qingdao, Peoples R China
[2] Minist Nat Resources, Inst Oceanog 1, Qingdao, Peoples R China
[3] Minist Nat Resources, Key Lab Marine Sci & Numer Modeling, Qingdao, Peoples R China
[4] Pilot Natl Lab Marine Sci & Technol, Lab Reg Oceanog & Numer Modeling, Qingdao, Peoples R China
[5] Shandong Key Lab Marine Sci & Numer Modeling, Qingdao, Peoples R China
[6] Ocean Univ China, Frontier Sci Ctr Deep Ocean Multispheres & Earth S, Qingdao, Peoples R China
[7] Ocean Univ China, Phys Oceanog Lab, Qingdao, Peoples R China
基金
中国国家自然科学基金;
关键词
CTD (conductivity-temperature-depth); outlier detection; cubic spline fitting; independent points; optimal number of iterations; OCEAN; PARAMETERS;
D O I
10.3389/fmars.2022.1030980
中图分类号
X [环境科学、安全科学];
学科分类号
08 ; 0830 ;
摘要
Outlier detection is the key to the quality control of marine survey data. For the detection of outliers in Conductivity-Temperature-Depth (CTD) data, previous methods, such as the Wild Edit method and the Median Filter Combined with Maximum Deviation method, mostly set a threshold based on statistics. Values greater than the threshold are treated as outliers, but there is no clear specification for the selection of threshold, thus multiple attempts are required. The process is time-consuming and inefficient, and the results have high false negative and positive rates. In response to this problem, we proposed an outlier detection method in CTD conductivity data, based on a physical constraint, the continuity of seawater. The method constructs a cubic spline fitting function based on the independent points scheme and the cubic spline interpolation to fit the conductivity data. The maximum fitting residual points will be flagged as outliers. The fitting stops when the optimal number of iterations is reached, which is automatically obtained by the minimum value of the sequence of maximum fitting residuals. Verification of the accuracy and stability of the method by means of examples proves that it has a lower false negative rate (17.88%) and false positive rate (0.24%) than other methods. Indeed, rates for the Wild Edit method are 56.96% and 2.19%, while for the Median Filter Combined with Maximum Deviation method rates are 23.28% and 0.31%. The Cubic Spline Fitting method is simple to operate, the result is clear and definite, better solved the problem of conductivity outliers detection.
引用
收藏
页数:11
相关论文
共 50 条
  • [1] FITTING BIOMECHANICAL DATA WITH CUBIC SPLINE FUNCTIONS
    ZERNICKE, RF
    CALDWELL, G
    ROBERTS, EM
    RESEARCH QUARTERLY, 1976, 47 (01): : 9 - 19
  • [2] Genetic algorithms based cubic spline functions fitting
    Chen, Xiaoping
    Yu, Shenglin
    Liu, Wenbo
    Shu Ju Cai Ji Yu Chu Li/Journal of Data Acquisition & Processing, 2000, 15 (02): : 138 - 141
  • [3] Research of Cutter Track Elimination Algorithm for CNC Cam Grinding Based on Cubic Spline Curve Fitting and Spline Interpolation
    Huang, W. S.
    Mao, G. Y.
    Zhang, J. S.
    FUNCTIONAL MANUFACTURING TECHNOLOGIES AND CEEUSRO I, 2010, 426-427 : 550 - 554
  • [4] Research on Weak Chirp Signal Detection Based on Cubic Spline Interpolation
    Duan, Yu
    Jing, Peiliang
    Shi, Guowei
    Du, Jian
    Gui, Chunmei
    2019 11TH INTERNATIONAL CONFERENCE ON INTELLIGENT HUMAN-MACHINE SYSTEMS AND CYBERNETICS (IHMSC 2019), VOL 1, 2019, : 37 - 40
  • [5] EDGE-DETECTION BY REGULARIZED CUBIC B-SPLINE FITTING
    CHEN, G
    YANG, YHH
    IEEE TRANSACTIONS ON SYSTEMS MAN AND CYBERNETICS, 1995, 25 (04): : 636 - 643
  • [6] Research on outlier detection of data based on machine learning
    Wang, Chunyang
    PROCEEDINGS OF ACM TURING AWARD CELEBRATION CONFERENCE, ACM TURC 2021, 2021, : 200 - 203
  • [7] RESEARCH AND APPLICATION OF CURVE FITTING BASED ON CUBIC B-SPLINE WITH ADAPTIVE SHAPE PARAMETERS
    Wang, Xiaoli
    Ma, Xiaohu
    Yong, Shihua
    INTERNATIONAL SYMPOSIUM ON COMPUTER SCIENCE & TECHNOLOGY, PROCEEDINGS, 2009, : 244 - 247
  • [8] Roof edge detection using regularized cubic B-spline fitting
    Liang, KH
    Tjahjadi, T
    Yang, YH
    PATTERN RECOGNITION, 1997, 30 (05) : 719 - 728
  • [9] Fitting and Smoothing Data Using Algebraic Hyperbolic Cubic Hermite Spline Interpolation
    Oraiche, Mohammed
    Lamnii, Abdellah
    Louzar, Mohamed
    Madark, Mhamed
    ENGINEERING LETTERS, 2022, 30 (02)
  • [10] Analysis of the Error on Fitting Highway Alignment Based on the Cubic Spline Function
    Zhang Hang
    Huang Yun
    Liu Fucheng
    Chen Xiaofang
    ICICTA: 2009 SECOND INTERNATIONAL CONFERENCE ON INTELLIGENT COMPUTATION TECHNOLOGY AND AUTOMATION, VOL III, PROCEEDINGS, 2009, : 487 - +