Quantum fluctuations in the van der Waals material NiPS3

We present the magnetic excitation spectrum of the quantum magnet NiPS3 near the zig-zag ground state of a minimal honeycomb spin Hamiltonian that includes bilinear and biquadratic spin interactions. Our analysis, using a multiboson generalized spin wave theory suited for spin S = 1 systems, revealed two normal modes at the linear level. The one at lower energy corresponds to a single magnon mode, consistent with results from spectroscopy experiments. Without single-ion anisotropy, this mode features a Goldstone mode at the corner of the Brillouin zone. When single ion anisotropy is introduced, the zig-zag phase's global U(1) invariance is broken, resulting in a gap. The higher energy mode corresponds to two-magnon fluctuations, which appear at the harmonic level in the generalized spin wave theory. This mode forms a gapped flat band due to bilinear spin interactions and becomes dispersive when biquadratic interactions are considered. The higher energy dispersion is related to quadrupolar fluctuations, which are feasible in magnets where the order parameter fluctuates in the SU(3) space. The spectrum analysis yielded quantum corrections to the order parameter and detected instabilities in the NiPS3 dipolar phases. Identifying the highest energy branch in experiments could provide insight into hidden nematic orders in NiPS3 and other van der Waals magnets.