Multi-Agent Double Deep Q-Learning for Fairness in Multiple-Access Underlay Cognitive Radio Networks

被引:0
|
作者
Ali, Zain [1 ]
Rezki, Zouheir [1 ]
Sadjadpour, Hamid [1 ]
机构
[1] Electrical and Computer Engineering Department, Baskin School of Engineering, University of California at Santa Cruz, Santa Cruz,CA,95064, United States
来源
IEEE Transactions on Machine Learning in Communications and Networking | 2024年 / 2卷
关键词
Cognitive systems - Game theory - Information management - Iterative methods - Multi agent systems - Optimal systems - Radio interference - Radio systems - Radio transmission - Reinforcement learning - Resource allocation - Spectrum efficiency;
D O I
10.1109/TMLCN.2024.3391216
中图分类号
学科分类号
摘要
Underlay Cognitive Radio (CR) systems were introduced to resolve the issue of spectrum scarcity in wireless communication. In CR systems, an unlicensed Secondary Transmitter (ST) shares the channel with a licensed Primary Transmitter (PT). Spectral efficiency of the CR systems can be further increased if multiple STs share the same channel. In underlay CR systems, the STs are required to keep interference at a low level to avoid outage at the primary system. The restriction on interference in underlay CR prevents some STs from transmitting while other STs may achieve high data rates, thus making the underlay CR network unfair. In this work, we consider the problem of achieving fairness in the rates of the STs. The considered optimization problem is non-convex in nature. The conventional iteration-based optimizers are time-consuming and may not converge when the considered problem is non-convex. To deal with the problem, we propose a deep-Q reinforcement learning (DQ-RL) framework that employs two separate deep neural networks for the computation and estimation of the Q-values which provides a fast solution and is robust to channel dynamic. The proposed technique achieves near optimal values of fairness while offering primary outage probability of less than 4%. Further, increasing the number of STs results in a linear increase in the computational complexity of the proposed framework. A comparison of several variants of the proposed scheme with the optimal solution is also presented. Finally, we present a novel cumulative reward framework and discuss how the combined-reward approach improves the performance of the communication system. ©2024 The Authors.
引用
收藏
页码:580 / 595
相关论文
共 50 条
  • [21] Untangling Braids with Multi-Agent Q-Learning
    Khan, Abdullah
    Vernitski, Alexei
    Lisitsa, Alexei
    2021 23RD INTERNATIONAL SYMPOSIUM ON SYMBOLIC AND NUMERIC ALGORITHMS FOR SCIENTIFIC COMPUTING (SYNASC 2021), 2021, : 135 - 139
  • [22] Q-learning with FCMAC in multi-agent cooperation
    Hwang, Kao-Shing
    Chen, Yu-Jen
    Lin, Tzung-Feng
    ADVANCES IN NEURAL NETWORKS - ISNN 2006, PT 1, 2006, 3971 : 599 - 606
  • [23] Cooperative Multi-Agent Learning and Coordination for Cognitive Radio Networks
    Zame, William
    Xu, Jie
    van der Schaar, Mihaela
    IEEE JOURNAL ON SELECTED AREAS IN COMMUNICATIONS, 2014, 32 (03) : 464 - 477
  • [24] Multi-agent Q-Learning of Channel Selection in Multi-user Cognitive Radio Systems: A Two by Two Case
    Li, Husheng
    2009 IEEE INTERNATIONAL CONFERENCE ON SYSTEMS, MAN AND CYBERNETICS (SMC 2009), VOLS 1-9, 2009, : 1893 - 1898
  • [25] A novel multi-agent Q-learning algorithm in cooperative multi-agent system
    Ou, HT
    Zhang, WD
    Zhang, WY
    Xu, XM
    PROCEEDINGS OF THE 3RD WORLD CONGRESS ON INTELLIGENT CONTROL AND AUTOMATION, VOLS 1-5, 2000, : 272 - 276
  • [26] Multi-Agent Exploration for Faster and Reliable Deep Q-Learning Convergence in Reinforcement Learning
    Majumdar, Abhijit
    Benavidez, Patrick
    Jamshidi, Mo
    2018 WORLD AUTOMATION CONGRESS (WAC), 2018, : 222 - 227
  • [27] Deep Q-Learning Market Makers in a Multi-Agent Simulated Stock Market
    Fernandez Vicente, Oscar
    Fernandez Rebollo, Fernando
    Garcia Polo, Francisco Javier
    ICAIF 2021: THE SECOND ACM INTERNATIONAL CONFERENCE ON AI IN FINANCE, 2021,
  • [28] Pricing in agent economies using multi-agent Q-learning
    Tesauro, G
    Kephart, JO
    AUTONOMOUS AGENTS AND MULTI-AGENT SYSTEMS, 2002, 5 (03) : 289 - 304
  • [29] Cooperation and Coordination Regimes by Deep Q-Learning in Multi-agent Task Executions
    Miyashita, Yuki
    Sugawara, Toshiharu
    ARTIFICIAL NEURAL NETWORKS AND MACHINE LEARNING - ICANN 2019: THEORETICAL NEURAL COMPUTATION, PT I, 2019, 11727 : 541 - 554
  • [30] Pricing in Agent Economies Using Multi-Agent Q-Learning
    Gerald Tesauro
    Jeffrey O. Kephart
    Autonomous Agents and Multi-Agent Systems, 2002, 5 : 289 - 304