The Big Five Factors Model (BFFM) is the most widely accepted personality theory used by psychologists today. The theory states that personality can be described with five core factors which are Conscientiousness, Agreeableness, Emotional Stability, Openness to Experience, and Extraversion. In this work, we measure the five factors using handwriting analysis instead of answering a long questionnaire of personality test. Handwriting analysis is a study that merely needs a writing sample to assess personality traits of the writer. It started manually by interpreting the extracted features such as size of writing, slant, and space between words into personality traits based on graphological rules. In this work, we proposed an automated BFFM system called Averaging of SMOTE multi-label SVM-CNN (AvgMlSC). AvgMlSC constructs synthetic samples to handle imbalanced data using Synthetic Minority Oversampling Technique (SMOTE). It averages two learning-based classifiers i.e. Multi-label Support Vector Machine and Multi-label Convolutional Neural Network based on offline handwriting recognition to produce one optimal predictive model. The model was trained using 1066 handwriting samples written in English, French, Chinese, Arabic, and Spanish. The results reveal that our proposed model outperformed the overall performance of five traditional models i.e. Logistic Regression (LR), Naive Bayes (NB), K-Neighbors (KN), Support Vector Machine (SVM), and Convolutional Neural Network (CNN) with 93% predictive accuracy, 0.94 AUC, and 90% F-Score.