SAMPolyBuild: Adapting the Segment Anything Model for polygonal building extraction

Extracting polygonal buildings from high-resolution remote sensing images is a critical task for large-scale mapping, 3D city modeling, and various geographic information system applications. Traditional methods are often restricted in accurately delineating boundaries and exhibit limited generalizability, which can affect their real-world applicability. The Segment Anything Model (SAM), a promptable segmentation model trained on an unprecedentedly large dataset, demonstrates remarkable generalization ability across various scenarios. In this context, we present SAMPolyBuild, an innovative framework that adapts SAM for polygonal building extraction, allowing for both automatic and prompt-based extraction. To fulfill the requirement for object location prompts in SAM, we developed the Auto Bbox Prompter, which is trained to detect building bounding boxes directly from the image encoder features of the SAM. The boundary precision of the SAM mask results was insufficient for vector polygon extraction, especially when challenged by blurry edges and tree occlusions. Therefore, we extended the SAM decoder with additional parameters to enable multitask learning to predict masks and generate Gaussian vertex and boundary maps simultaneously. Furthermore, we developed a mask- guided vertex connection algorithm to generate the final polygon. Extensive evaluation on the WHU-Mix vector dataset and SpaceNet datasets demonstrate that our method achieves a new state-of-the-art in terms of accuracy and generalizability, significantly improving average precision (AP), average recall (AR), intersection over union (IoU), boundary F1, and vertex F1 metrics. Moreover, by combining the automatic and prompt modes of our framework, we found that 91.2% of the building polygons predicted by SAMPolyBuild on out- of-domain data closely match the quality of manually delineated polygons. The source code is available at https://github.com/wchh-2000/SAMPolyBuild.