Survey on Testing of Deep Neural Networks

被引：0
|
作者：
Wang Z. [1 ]
Yan M. [1 ]
Liu S. [1 ]
Chen J.-J. [1 ]
Zhang D.-D. [1 ]
Wu Z. [2 ]
Chen X. [3 ]
机构：
[1] College of Intelligence and Computing, Tianjin University, Tianjin
[2] International Engineering Institute, Tianjin University, Tianjin
[3] School of Information Science and Technology, Nantong University, Nantong
来源：
Ruan Jian Xue Bao/Journal of Software | 2020年 / 31卷 / 05期
基金：
中国国家自然科学基金;
关键词：
Deep neural network; Test case generation; Test coverage;
D O I：
10.13328/j.cnki.jos.005951
中图分类号：
学科分类号：
摘要：
With the rapid development of deep neural networks, the emerging of big data as well as the advancement of computational power, Deep Neural Network (DNN) has been widely applied in various safety-critical domains such as autonomous driving, automatic face recognition, and aircraft collision avoidance systems. Traditional software systems are implemented by developers with carefully designed programming logics and tested with test cases which are designed based on specific coverage criteria. Unlike traditional software development, DNN defines a data-driven programming paradigm, i.e., developers only design the structure of networks and the inner logic is reflected by weights which are learned during training. Traditional software testing methods cannot be applied to DNN directly. Driven by the emerging demand, more and more research works have focused on testing of DNN, including proposing new testing evaluation criteria, generation of test cases, etc. This study provides a thorough survey on testing DNN, which summarizes 92 works from related fields. These works are systematically reviewed from three perspectives, i.e., DNN testing metrics, test input generation, and test oracle. Existing achievements are introduced in terms of image processing, speech processing, and natural language processing. The datasets and tools used in DNN testing are surveyed and finally the thoughts on potential future research directions are summarized on DNN testing, which, hopefully, will provide references for researchers interested in the related directions. © Copyright 2020, Institute of Software, the Chinese Academy of Sciences. All rights reserved.
引用
收藏
页码：1255 / 1275
页数：20
相关论文
共 98 条
  • [1] Hinton GE, Salakhutdinov R., Reducing the dimensionality of data with neural networks, Science, 313, 5786, pp. 504-507, (2006)
  • [2] Huang X, Kroening D, Kwiatkowska M, Et al., Safety and trustworthiness of deep neural networks: A survey, (2018)
  • [3] Zhang JM, Harman M, Ma L, Liu Y., Machine learning testing: Survey, landscapes and horizons, (2019)
  • [4] Xiang W, Musau P, Wild AA, Et al., Verification for machine learning, autonomy, and neural networks survey, (2018)
  • [5] Pei K, Cao Y, Yang J, Et al., Deepxplore: Automated whitebox testing of deep learning systems, Proc. of the 26th Symp. on Operating Systems Principles, pp. 1-18, (2017)
  • [6] Sun Y, Huang X, Kroening D., Testing deep neural networks, (2018)
  • [7] Ma L, Juefei-Xu F, Zhang FY, Et al., DeepGauge: Multi-granularity testing criteria for deep learning systems, Proc. of the Automated Software Engineering, pp. 120-131, (2018)
  • [8] Ma L, Juefei-Xu F, Xue M, Et al., DeepCT: Tomographic combinatorial testing for deep learning systems, Proc. of the 2019 IEEE 26th Int'l Conf. on Software Analysis, Evolution and Reengineering (SANER), pp. 614-618, (2019)
  • [9] Wang D, Wang Z, Fang C, Et al., DeepPath: Path-driven testing criteria for deep neural networks, Proc. of the 2019 IEEE Int'l Conf. On Artificial Intelligence Testing (AITest), pp. 119-120, (2019)
  • [10] Tian Y, Zhong Z, Ordonez V, Et al., Testing deep neural network based image classifiers, (2019)
12345678910