Planetary turbulent flows are observed to self-organize into large scale structures such as zonal jets and coherent vortices. In this work, the eddy-mean flow dynamics underlying the formation of both zonal and nonzonal coherent structures in a barotropic turbulent flow is investigated within the statistical framework of stochastic structural stability theory (S3T). Previous studies have shown that the coherent structures emerge due to the instability of the homogeneous turbulent flow in the statistical dynamical S3T system and that the statistical predictions of S3T are reflected in direct numerical simulations. In this work, the dynamics underlying the structure forming S3T instability are studied. It is shown that, for weak planetary vorticity gradient beta, both zonal jets and non-zonal large-scale structures form from upgradient momentum fluxes due to shearing of the eddies by the emerging flow. For large beta, the dynamics of the S3T instability differs for zonal and non-zonal flows. Shearing of the eddies by the mean flow continues to be the mechanism for the emergence of zonal jets while non-zonal large-scale flows emerge from resonant and near-resonant triad interactions between the large-scale flow and the stochastically forced eddies.