Signed graphs are prevalent data structures containing both positive and negative links. Recently, the fundamental network analysis task on signed graphs, namely link sign prediction, has received careful attention. Existing methods learn two target node representations independently, and the sign between these two nodes is predicted based on similarity. However, such a paradigm is node-centric that cannot distinguish node pairs with distinct contexts, thus lowering the prediction performance. Learning pair-centric representation is therefore a rewarding way to be aware of differences between pairs. There is no study yet on how to build such an appropriate representation that can effectively infer the sign between the target node pair. In this paper, we provide a new perspective to conduct link sign prediction within the paradigm of subgraph classification and propose a novel Subgraph-based link Sign Prediction (SSP) model. Technically, SSP uses importance-based sampling to extract an informative subgraph around each target node pair. For each subgraph, an innovative node labeling scheme is designed to encode its structural and signed information for representation learning. To further utilize the subgraph representation for imbalanced sign classification, SSP employs self-pruning contrastive learning to gain balanced representations. Extensive experiments on real-world datasets demonstrate that SSP consistently and significantly outperforms all the state-of-the-art baselines.