The transformation properties of a Kalb-Ramond field are those of a gauge potential. However, it is not clear what is the group structure to which these transformations are associated. In this Letter, we complete a program started in previous articles in order to clarify this question. Using the spectral theorem, we improve and generalize previous approaches and find the possible group structures underneath the 2-form gauge potential as extensions of Lie groups, when its representations are assumed to act into any tensor (or spinor) space with inner product. We also obtain a fundamental representation where a two-form field turns out to be a connection on a flat Euclidean basis manifold, with a corresponding canonical curvature. However, we show that these objects are not associated to space-time tensors and, in particular, that a standard Yang-Mills action is not relativistically invariant, except (as expected) in the Abelian case. This is our main result, from the physical point of view. (C) 2003 Published by Elsevier B.V.