With the advances of ship noise reduction technology, the working frequency of the passive sonar must be reduced in order to detect a target. For the conventional array, it requires a large array aperture, comparable to the wavelength, in order to achieve an acceptable angular resolution. Arrays of small physical size with high angular resolution are thus attractive for low-frequency direction-of-arrival estimation of underwater sound source. In this paper, we consider a 3 x 3 uniform rectangular array which consists of vector sensors with inter-sensor spacing much smaller than the wavelength. A broadband super-directive beamforming method is proposed for this vector sensor array, which extracts multi-pole modes of different orders from the spatial differentials of the sound field. By normalizing the amplitudes of the multi-pole modes, frequency invariant mode functions can be obtained, which are used to build the desired beam pattern, despite the Rayleigh limit on the achievable angular resolution. Vector sensors are used to replace the pressure difference operation, thus to achieve a desirable beam pattern, the order of spatial differential will be reduced. In other words, for the same array configuration, using the vector sensors provides higher directivity than using the pressure sensor. To concentrate on the sources, and to minimize all hindrances from around circumference, a suitable beam pattern is constructed as an example to analysis. To verify the algorithm, a prototype is built and tested in a water tank. Comparisons are carried out between the actually synthesized beam patterns and the theoretical ones. The experimental results show good agreement with the theoretical results, and that the directivity increases with the multi-pole mode order increasing, at the expense of lower robustness. The performances for different values of k a are also investigated, where k is the wave number and a denotes the inter-sensor spacing. Simulation results show that when the inter-sensor spacing is no more than one-sixth of the incident wave length, the error introduced by the approximations for muti-pole mode extraction can be neglected. It should be noted that this result of the inter-sensor spacing still applicable when considering array gain, showing that the array is insensitive to uncorrelated noise while preserving a relatively high array gain. Finally, the influence of the underwater acoustic waveguide on the array performance is analyzed. Simulations and experimental tests show that due to the small array aperture, the waveguide effects on the array performance are limited.
