Nematic spin liquid phase in a frustrated spin-1 system on the square lattice

Frustration in quantum spin systems promote a variety of novel quantum phases. An important example is the frustrated spin-1 model on the square lattice with the nearest-neighbor bilinear (J(1)) and biquadratic (K-1) interactions. We provide strong evidence for a nematic spin liquid phase in a range of K-1/J(1) near the SU(3)-symmetric point (J(1) = K-1), based on the linear flavor-wave theory and extensive density matrix renormalization group calculation. This phase displays no spin dipolar or quadrupolar order, preserves translational symmetry but spontaneously breaks C-4 lattice rotational symmetry, and possesses fluctuations peaked at the wave vector (pi, 2 pi/3). The spin excitation gap drops rapidly with system size and appears to be gapless, and the nematic order is attributed to the dominant (pi, 2 pi/3) fluctuations. Our results provide a novel mechanism for electronic nematic order and, more generally, open up a new avenue to explore frustration-induced exotic ground states.