Fruits of Syzygium jambos (L.) are recognized as a “food”, exhibiting significant antidiabetic activities. However, the α-glucosidase inhibition of the components from Syzygium jambos (L.) have not yet been investigated. In this study, a total of 14 compounds were isolated from Syzygium jambos (L.) Alston, eight of which showed significant inhibitory effects on α-glucosidase, with IC50 values in the range of 0.011–0.665 mM. Notably, compounds 1–3 (IC50: 0.013, 0.011 and 0.030 mM, respectively) exhibited much stronger activity than acarbose (IC50: 2.329 ± 0.109 mM). The enzyme kinetics study indicated that compound 1 was an uncompetitive inhibitor, and compounds 2–8 were mixed-type inhibitors. Moreover, the interactions between compounds and α-glucosidase were investigated by molecular docking, which further revealed that the number of olefin double bonds and 2-COOH of heptadeca-phenols had a notable effect on the α-glucosidase inhibitory activity. This study demonstrated that Syzygium jambos (L.) fruit might serve as a functional food for the prevention of diabetes mellitus.