Optimizing low-rank adaptation with decomposed matrices and adaptive rank allocation

被引：0

作者：

Zhang, Dacao ^{[1
]}

Yang, Fan ^{[1
]}

Zhang, Kun ^{[1
]}

Li, Xin ^{[2
]}

Wei, Si ^{[2
]}

Hong, Richang ^{[1
]}

Wang, Meng ^{[1
]}

机构：

[1] Hefei Univ Technol, Sch Comp Sci Informat Engn, Hefei 230601, Peoples R China

[2] iFLYTEK Co Ltd, Artificial Intelligence Res Inst, Hefei 230088, Peoples R China

来源：

FRONTIERS OF COMPUTER SCIENCE | 2025年 / 19卷 / 05期

基金：

中国国家自然科学基金;

关键词：

D O I：

10.1007/s11704-024-40317-w

中图分类号：

TP [自动化技术、计算机技术];

学科分类号：

0812 ;

摘要：

In this paper, we argue that the same rank setting of LoRA will inhibit its potential. In response, we proposed two novel PEFT strategies to improve the capability of LoRA in single-task and multi-task scenarios. We also conducted extensive experiments to show the effectiveness of our proposed method.

引用

页数：3

共 50 条

[1] Adaptive Low-Rank Approximation of Collocation Matrices
M. Bebendorf
S. Rjasanow
Computing, 2003, 70 : 1 - 24
[2] Adaptive low-rank approximation of collocation matrices
Bebendorf, M
Rjasanow, S
COMPUTING, 2003, 70 (01) : 1 - 24
[3] Low-rank and sparse matrices fitting algorithm for low-rank representation
Zhao, Jianxi
Zhao, Lina
COMPUTERS & MATHEMATICS WITH APPLICATIONS, 2020, 79 (02) : 407 - 425
[4] STATISTICAL RANK SELECTION FOR INCOMPLETE LOW-RANK MATRICES
Zhang, Rui
Shapiro, Alexander
Xie, Yao
2019 IEEE INTERNATIONAL CONFERENCE ON ACOUSTICS, SPEECH AND SIGNAL PROCESSING (ICASSP), 2019, : 2912 - 2916
[5] REDUCED BASIS METHODS: FROM LOW-RANK MATRICES TO LOW-RANK TENSORS
Ballani, Jonas
Kressner, Daniel
SIAM JOURNAL ON SCIENTIFIC COMPUTING, 2016, 38 (04): : A2045 - A2067
[6] Low-rank adaptive filters
Strobach, P
IEEE TRANSACTIONS ON SIGNAL PROCESSING, 1996, 44 (12) : 2932 - 2947
[7] Iterative Concave Rank Approximation for Recovering Low-Rank Matrices
Malek-Mohammadi, Mohammadreza
Babaie-Zadeh, Massoud
Skoglund, Mikael
IEEE TRANSACTIONS ON SIGNAL PROCESSING, 2014, 62 (20) : 5213 - 5226
[8] Matrices with Hierarchical Low-Rank Structures
Ballani, Jonas
Kressner, Daniel
EXPLOITING HIDDEN STRUCTURE IN MATRIX COMPUTATIONS: ALGORITHMS AND APPLICATIONS, 2016, 2173 : 161 - 209
[9] Correlation Clustering with Low-Rank Matrices
Veldt, Nate
Wirt, Anthony
Gleic, David F.
PROCEEDINGS OF THE 26TH INTERNATIONAL CONFERENCE ON WORLD WIDE WEB (WWW'17), 2017, : 1025 - 1034
[10] Decentralized sketching of low-rank matrices
Srinivasa, Rakshith S.
Lee, Kiryung
Junge, Marius
Romberg, Justin
ADVANCES IN NEURAL INFORMATION PROCESSING SYSTEMS 32 (NIPS 2019), 2019, 32

← 1 2 3 4 5 →