Pavement crack detection is a practical and challenging task that has the ability to significantly reduce the burden of manual building and road maintenance in intelligent transportation systems. Existing methods mainly focus on addressing common crack diseases and are poor in generalizing to other conditions of crack detection due to diverse environmental factors (e.g., illumination), topology complexity, and intensity in-homogeneity. Moreover, the samples suffer from the severe foreground-background imbalance and the model is easily prone to overfitting on trained anomalies, resulting in unsatisfactory performance. To tackle the aforementioned challenges and achieve high-quality pavement crack detection, we propose an innovative approach termed boundary-aware axial attention network (BAAN), which is composed of multiple position-guided axial attention (PAA) modules in a hierarchical encoder-decoder architecture. Specifically, it learns efficient contextual information via decomposed multidimensional position-guided attention to capture more precise spatial structures, and the proposed boundary regularization module (BRM) mines more discriminative foreground-background relationships to regularize the ambiguous details between diverse spatial regions. Moreover, we propose a novel boundary refinement loss (BRL) to alleviate the challenges associated with regional losses (e.g., pixel-wise cross-entropy loss) in the context of heavily imbalanced crack detection problems. The proposed BAAN is evaluated on four crack datasets and experimental results indicate that the BAAN consistently outperforms the state-of-the-art methods with fewer computational requirements.