Auto-Tagging for Massive Online Selection Tests: Machine Learning to the Rescue

Difficulty Level of a question is relative to that of other questions in a test and also to the test takers; hence manually assigning Difficulty Level tags may not be accurate. There is a need to infer them from historical data pertaining to the performance of students in a test. e-Yantra Robotics Competition (eYRC) is an annual competition having around 5000 teams (20,000 students) registering in the latest edition of the competition, eYRC-2015. All four team members take a test simultaneously and each individual gets questions which are different but have a similar Difficulty Level. A Question Bank containing 1800 unique questions from 3 subjects-Aptitude, Electronics, and C-Programming-is used to generate question sets each having 30 questions. It is a challenge to ensure that each set contains questions of similar Difficulty Levels tagged manually as Easy, Medium or Hard. In this paper, we discuss a learning algorithm called Weighted Clustering that can automatically tag questions by analyzing the performance of students. We used this algorithm to analyze the performance data in eYRC2014 for 614 questions from the Question Bank; we found that Manual Tagging accuracy was 44%. We retagged questions with Suggested Tags resulting from our analysis and used them again in eYRC-2015. When we applied the algorithm to the performance data in eYRC-2015, we found that the accuracy of tagging had significantly improved to 67%.