Source Separation in Joint Communication and Radar Systems Based on Unsupervised Variational Autoencoder

Source separation of a mixed signal in the time-frequency domain is critical for joint communication and radar (JCR) systems to achieve the required performance, especially at a low signal-to-noise ratio (SNR). In this paper, we propose the use of a generative model, such as the unsupervised variational autoencoder (VAE), to separate sensing and data communication signals. We first analyse the VAE system using different mask techniques; then, the best technique is selected for comparison with popular blind source separation (BSS) algorithms. We verify the performance of the proposed VAE by adopting different metrics such as the signal-to-distortion ratio (SDR), source-to-interference ratio (SIR), and sources-to-artifacts ratio (SAR). Simulation results show that the proposed VAE outperforms the BSS techniques at low SNR for the case of a mixed signal in the time-frequency domain and at low and high SNR for a mixed signal in the time domain. It enables the JCR system in the challenging first scenario to obtain SDR gains of 11.1 dB and 6 dB at 0 dB SNR for recovering the sensing and data communication signals respectively. Finally, we analyse the robustness of the JCR system in detecting an interference signal operating in the same frequency band, where the simulation result indicates an accuracy of 91% based on the proposed steps.