Purpose The pilot ratio is a critical parameter of the hydraulic system containing counterbalance valves, which has an important influence on the dynamic characteristics and energy consumption. Generally, these two aspects are contradictory. That is, a high pilot ratio is easy to cause the poor dynamic characteristics of the system, while a low pilot ratio leads to excessive energy consumption. Currently, there is still no effective method for determining the pilot ratio and analyzing the unstable dynamical behavior of the hydraulic system containing counterbalance valves. Methods This paper establishes a dynamics model for the valve-controlled hydraulic cylinder system containing counterbalance valves and investigates the bifurcation in detail. Firstly, the mathematical model, in which the pilot chamber and the spring chamber of counterbalance valves are considered, is established for the segment erector of a tunnel boring machine. Regarding load mass as the bifurcation parameter, the central manifold theorem and normal form method are then introduced to analyze Hopf bifurcation. Simultaneously, the interval of periodic solutions is further determined by the 0-1 test method. Due to the limitation of the quantitative result, a novel homotopy-based method is finally proposed for solving the periodic solution at the above interval. Results In the case of a high pilot ratio (k(x) = 8), the supercritical Hopf bifurcation occurs when the load mass exceeds 2352.5 kg. Moreover, the periodic solution exists at the interval (2352.5 kg, 3583 kg). Compared with the numerical solution, the homotopy series solution gives an analytical expression and has a good approximation for the periodic solution. Conclusions The qualitative results of bifurcation behaviors can be regarded as the basis for determining the pilot ratio of counterbalance valves. And the proposed method is adapted to analyze the unstable dynamical behavior introduced by counterbalance valves in a typical hydraulic circuit.