Objective. Despite substantial advancements in Brain-Computer Interface (BCI), inherent limitations such as extensive training time and high sensitivity to noise largely hinder their rapid development. To address such issues, this paper proposes a novel extreme learning machine (ELM) based self-attention (E-SAT) mechanism to enhance subject-specific classification performances. Approach. Specifically, for E-SAT, ELM is employed both to improve self-attention module generalization ability for feature extraction and to optimize the model's parameter initialization process. Meanwhile, the extracted features are also classified using ELM, and the end-to-end ELM based setup is used to evaluate E-SAT performance on different motor imagery (MI) EEG signals. Main results. Extensive experiments with different datasets, such as BCI Competition III Datasets IV-a, IV-b and BCI Competition IV Datasets 1, 2a, 2b, 3 are conducted to verify the effectiveness of the proposed E-SAT strategy. Results show that E-SAT outperforms several state-of-the-art and existing methods in subject-specific classification on all the datasets. An average classification accuracy of 99.8%, 99.1%, 98.9%, 75.8%, 90.8%, and 95.4% respectively is achieved for each datasets which demonstrate an improvement of 5%-6% compared to the existing methods. In addition, Kruskal Wallis test is performed to demonstrate the statistical significance of E-SAT and the results indicate significant difference with a 95% confidence level. Significance. The experimental results not only show outstanding performance of E-SAT in feature extraction, but also demonstrate that it helps achieve the best results among nine other robust classifiers. In addition, results in this study also demonstrate that E-SAT achieves exceptional performance in both binary and multi-class classification tasks, as well as for noisy and non-noisy datasets.