The ongoing coronavirus disease 2019 (COVID-19) epidemic has made a huge impact on health, economies, and societies all over the world. Although reverse transcription-polymerase chain reaction (RT-PCR)-based nucleic acid detection has been primarily used in the diagnosis of COVID-19, it is time-consuming with limited application scenarios and must be operated by qualified personnel. Antibody test, particularly point-of-care antibody testing, is a suitable complement to nucleic acid test as it provides rapid, portable, and cost-effective detection of infections. In this study, a Rapid Antibody Test Kit was developed based on fluorescence immunochromatography for the sensitive, accurate, and automated detection of immunoglobulin M (IgM) and IgG antibodies against severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) in human serum, plasma, and whole blood samples within 10 min. The sensitivity, specificity, precision, and stability of the test kit were of good performance. No cross-activity and no interference was observed. In the multiple-center parallel study, 223 samples from hospitalized patients were used to evaluate the clinical specificity of the test. Both SARS-CoV-2 IgM and IgG achieved a clinical specificity of 98.21%. The clinical sensitivities of SARS-CoV-2 IgM and IgG were 79.54% and 87.45%, respectively, among 733 reverse transcription-polymerase chain reaction (RT-PCR) confirmed SARS-CoV-2 samples. For the combined IgM and IgG assays, the sensitivity and specificity were 89.22% and 96.86%, respectively. Our results demonstrate that the combined use of IgM and IgG could serve as a more suitable alternative detection method for patients with COVID-19, and the developed kit is of great public health significance for the prevention and control of the COVID-19 pandemic.