Learning to Reweight Examples for Robust Deep Learning

被引:0
|
作者
Ren, Mengye [1 ,2 ]
Zeng, Wenyuan [1 ,2 ]
Yang, Bin [1 ,2 ]
Urtasun, Raquel [1 ,2 ]
机构
[1] Adv Technol Grp, Toronto, ON, Canada
[2] Univ Toronto, Dept Comp Sci, Toronto, ON, Canada
来源
INTERNATIONAL CONFERENCE ON MACHINE LEARNING, VOL 80 | 2018年 / 80卷
关键词
D O I
暂无
中图分类号
TP18 [人工智能理论];
学科分类号
081104 ; 0812 ; 0835 ; 1405 ;
摘要
Deep neural networks have been shown to be very powerful modeling tools for many supervised learning tasks involving complex input patterns. However, they can also easily overfit to training set biases and label noises. In addition to various regularizers, example reweighting algorithms are popular solutions to these problems, but they require careful tuning of additional hyperparameters, such as example mining schedules and regularization hyperparameters. In contrast to past reweighting methods, which typically consist of functions of the cost value of each example, in this work we propose a novel meta-learning algorithm that learns to assign weights to training examples based on their gradient directions. To determine the example weights, our method performs a meta gradient descent step on the current mini-batch example weights (which are initialized from zero) to minimize the loss on a clean unbiased validation set. Our proposed method can be easily implemented on any type of deep network, does not require any additional hyperparameter tuning, and achieves impressive performance on class imbalance and corrupted label problems where only a small amount of clean validation data is available.
引用
收藏
页数:10
相关论文
共 50 条
  • [1] Towards Adversarially Robust Text Classifiers by Learning to Reweight Clean Examples
    Xu, Jianhan
    Zhang, Cenyuan
    Zheng, Xiaoqing
    Li, Linyang
    Hsieh, Cho-Jui
    Chang, Kai-Wei
    Huang, Xuanjing
    FINDINGS OF THE ASSOCIATION FOR COMPUTATIONAL LINGUISTICS (ACL 2022), 2022, : 1694 - 1707
  • [2] Learning to Reweight with Deep Interactions
    Fan, Yang
    Xia, Yingce
    Wu, Lijun
    Xie, Shufang
    Liu, Weiqing
    Bian, Jiang
    Qin, Tao
    Li, Xiang-Yang
    THIRTY-FIFTH AAAI CONFERENCE ON ARTIFICIAL INTELLIGENCE, THIRTY-THIRD CONFERENCE ON INNOVATIVE APPLICATIONS OF ARTIFICIAL INTELLIGENCE AND THE ELEVENTH SYMPOSIUM ON EDUCATIONAL ADVANCES IN ARTIFICIAL INTELLIGENCE, 2021, 35 : 7385 - 7393
  • [3] Learning to reweight examples in multi-label classification
    Zhong, Yongjian
    Du, Bo
    Xu, Chang
    NEURAL NETWORKS, 2021, 142 (142) : 428 - 436
  • [4] Learning to Reweight for Graph Neural Network
    Chen, Zhengyu
    Xiao, Teng
    Kuang, Kun
    Lv, Zheqi
    Zhang, Min
    Yang, Jinluan
    Lu, Chengqiang
    Yang, Hongxia
    Wu, Fei
    THIRTY-EIGHTH AAAI CONFERENCE ON ARTIFICIAL INTELLIGENCE, VOL 38 NO 8, 2024, : 8320 - 8328
  • [5] Learning to Reweight Terms with Distributed Representations
    Zheng, Guoqing
    Callan, Jamie
    SIGIR 2015: PROCEEDINGS OF THE 38TH INTERNATIONAL ACM SIGIR CONFERENCE ON RESEARCH AND DEVELOPMENT IN INFORMATION RETRIEVAL, 2015, : 575 - 584
  • [6] The Problem of the Adversarial Examples in Deep Learning
    Zhang S.-S.
    Zuo X.
    Liu J.-W.
    Jisuanji Xuebao/Chinese Journal of Computers, 2019, 42 (08): : 1886 - 1904
  • [7] Analysing Adversarial Examples for Deep Learning
    Jung, Jason
    Akhtar, Naveed
    Hassan, Ghulam
    VISAPP: PROCEEDINGS OF THE 16TH INTERNATIONAL JOINT CONFERENCE ON COMPUTER VISION, IMAGING AND COMPUTER GRAPHICS THEORY AND APPLICATIONS - VOL. 5: VISAPP, 2021, : 585 - 592
  • [8] Learning to Reweight Imaginary Transitions for Model-Based Reinforcement Learning
    Huang, Wenzhen
    Yin, Qiyue
    Zhang, Junge
    Huang, Kaiqi
    THIRTY-FIFTH AAAI CONFERENCE ON ARTIFICIAL INTELLIGENCE, THIRTY-THIRD CONFERENCE ON INNOVATIVE APPLICATIONS OF ARTIFICIAL INTELLIGENCE AND THE ELEVENTH SYMPOSIUM ON EDUCATIONAL ADVANCES IN ARTIFICIAL INTELLIGENCE, 2021, 35 : 7848 - 7856
  • [9] Robust boosting for learning from few examples
    Wolf, L
    Martin, I
    2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Vol 1, Proceedings, 2005, : 359 - 364
  • [10] Competition on robust deep learning
    Dong, Yinpeng
    Liu, Chang
    Xiang, Wenzhao
    Su, Hang
    Zhu, Jun
    NATIONAL SCIENCE REVIEW, 2023, 10 (06)