Survey on Testing of Deep Neural Networks

被引：0
|
作者：
Wang Z. [1 ]
Yan M. [1 ]
Liu S. [1 ]
Chen J.-J. [1 ]
Zhang D.-D. [1 ]
Wu Z. [2 ]
Chen X. [3 ]
机构：
[1] College of Intelligence and Computing, Tianjin University, Tianjin
[2] International Engineering Institute, Tianjin University, Tianjin
[3] School of Information Science and Technology, Nantong University, Nantong
来源：
Ruan Jian Xue Bao/Journal of Software | 2020年 / 31卷 / 05期
基金：
中国国家自然科学基金;
关键词：
Deep neural network; Test case generation; Test coverage;
D O I：
10.13328/j.cnki.jos.005951
中图分类号：
学科分类号：
摘要：
With the rapid development of deep neural networks, the emerging of big data as well as the advancement of computational power, Deep Neural Network (DNN) has been widely applied in various safety-critical domains such as autonomous driving, automatic face recognition, and aircraft collision avoidance systems. Traditional software systems are implemented by developers with carefully designed programming logics and tested with test cases which are designed based on specific coverage criteria. Unlike traditional software development, DNN defines a data-driven programming paradigm, i.e., developers only design the structure of networks and the inner logic is reflected by weights which are learned during training. Traditional software testing methods cannot be applied to DNN directly. Driven by the emerging demand, more and more research works have focused on testing of DNN, including proposing new testing evaluation criteria, generation of test cases, etc. This study provides a thorough survey on testing DNN, which summarizes 92 works from related fields. These works are systematically reviewed from three perspectives, i.e., DNN testing metrics, test input generation, and test oracle. Existing achievements are introduced in terms of image processing, speech processing, and natural language processing. The datasets and tools used in DNN testing are surveyed and finally the thoughts on potential future research directions are summarized on DNN testing, which, hopefully, will provide references for researchers interested in the related directions. © Copyright 2020, Institute of Software, the Chinese Academy of Sciences. All rights reserved.
引用
收藏
页码：1255 / 1275
页数：20
相关论文
共 98 条
  • [41] Wicker M, Huang X, Kwiatkowska M., Feature-guided black-box safety testing of deep neural networks, Proc. of the Int'l Conf. on Tools and Algorithms for the Construction and Analysis of Systems, pp. 408-426, (2018)
  • [42] Liang H, Pei X, Jia X, Et al., Fuzzing: State of the art, IEEE Trans. on Reliability, 67, 3, pp. 1199-1218, (2018)
  • [43] Jia Y, Harman M., An analysis and survey of the development of mutation testing, IEEE Trans. on Software Engineering, 37, 5, pp. 649-678, (2010)
  • [44] Ma L, Zhang F, Sun J, Et al., Deepmutation: Mutation testing of deep learning systems, Proc. of the 2018 IEEE 29th Int'l Symp. on Software Reliability Engineering (ISSRE), pp. 100-111, (2018)
  • [45] Majumdar R, Sen K., Hybrid concolic testing, Proc. of the 29th Int'l Conf. on Software Engineering (ICSE 2007), pp. 416-426, (2007)
  • [46] Li ZN, Ma XX, Xu C, Et al., Boosting operational DNN testing efficiency through conditioning, Proc. of the 27th ACM Joint Meeting on European Software Engineering Conf. and Symp. on the Foundations of Software Engineering, pp. 499-509, (2019)
  • [47] Cao X, Gong NZ., Mitigating evasion attacks to deep neural networks via region-based classification, Proc. of the 33rd Annual Computer Security Applications Conf. ACM, pp. 278-287, (2017)
  • [48] Moosavi-Dezfooli SM, Fawzi A, Frossard P., Deepfool: A simple and accurate method to fool deep neural networks, Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition, pp. 2574-2582, (2016)
  • [49] Barr ET, Harman M, McMinn P, Et al., The oracle problem in software testing: A survey, IEEE Trans. on Software Engineering, 41, 5, pp. 507-525, (2014)
  • [50] Dong GW, Xu BW, Chen L, Et al., Survey of metamorphic testing, Journal of Frontiers of Computer Science and Technology, 3, 2, pp. 130-143, (2009)
12345678910