In this article, the index of imprimitivity of an irreducible nonnegative matrix in the famous PerronFrobenius theorem is studied within a more general framework, both in a more general tensor setting and in a more natural spectral symmetry perspective. A k-th order tensor has symmetric spectrum if the set of eigenvalues is symmetric under a group action with the group being a subgroup of the multiplicative group of k-th roots of unity. A sufficient condition, in terms of linear equations over the quotient ring, for a tensor possessing symmetric spectrum is given, which becomes also necessary when the tensor is nonnegative, symmetric and weakly irreducible, or an irreducible nonnegative matrix. Moreover, it is shown that for a weakly irreducible nonnegative tensor, the spectral symmetries are the same when either counting or ignoring multiplicities of the eigenvalues. In particular, the spectral symmetry(index of imprimitivity) of an irreducible nonnegative Sylvester matrix is completely resolved via characterizations with the indices of its positive entries. It is shown that the spectrum of an irreducible nonnegative Sylvester matrix can only be 1-symmetric or 2-symmetric, and the exact situations are fully described. With this at hand, the spectral symmetry of a nonnegative two-dimensional symmetric tensor with arbitrary order is also completely characterized.