Dynamical analysis of a predator-prey system with prey vigilance and hunting cooperation in predators

In this work, we propose a predator-prey system with a Holling type II functional response and study its dynamics when the prey exhibits vigilance behavior to avoid predation and predators exhibit cooperative hunting. We provide conditions for existence and the local and global stability of equilibria. We carry out detailed bifurcation analysis and find the system to experience Hopf, saddle-node, and transcritical bifurcations. Our results show that increased prey vigilance can stabilize the system, but when vigilance levels are too high, it causes a decrease in the population density of prey and leads to extinction. When hunting cooperation is intensive, it can destabilize the system, and can also induce bi-stability phenomenon. Furthermore, it can reduce the population density of both prey and predators and also change the stability of a coexistence state. We provide numerical experiments to validate our theoretical results and discuss ecological implications. © 2024 the Author(s), licensee AIMS Press. This is an open access article distributed under the terms of the Creative Commons Attribution License.