Spin liquid versus long-range magnetic order in the frustrated body-centered-tetragonal lattice

We show how spin-liquid (SL) states can be stabilized in a realistic three-dimensional model as a result of frustration. SU(n)-symmetric generalization of the Heisenberg model for quantum spin S operators is used to investigate the frustrated body-centered tetragonal (BCT) lattice with antiferromagnetic interlayer coupling J(1) and intralayer first and second-neighbor couplings J(2) and J(3). By using complementary representations of the spin operators, we study the phase diagram characterizing the ground state of this system. For small n, we find that the most stable solutions correspond to four different families of long-range magnetic orders that are governed by J(1), J(2), and J(3). First, some possible instabilities of these phases are identified for n = 2, in large S expansions, up to the linear spin-wave corrections. Then, using a fermionic representation of the SU(n) spin operators for S = 1/2, we find that purely magnetic orders occur for n <= 3 while SL solutions are stabilized for n >= 10. The SL solution governed by J(1) breaks the lattice translation symmetry. The modulated SL is associated with a commensurate ordering wave vector (1,1,1). For 4 <= n <= 9, we show how the competition between J(1), J(2), and J(3) can turn the magnetically ordered ground state into a SL state. Finally, we discuss the relevance of this scenario for correlated systems with BCT crystal structure.