Exploring the Effectiveness of Data-Driven Learning Materials for Promoting Student Performance in Introductory Programming

Cryptic and ambiguous error messages provided by programming environments are crucial barriers to beginners when learning to program. This study designed a coding manual to explain common Python programming errors to students using a data-driven approach, following guidelines suggested by prior studies. A quasi-experiment with two groups of middle school students was conducted to examine the effectiveness of the coding manual in reducing student errors, increasing confidence, and promoting learning performance. By analyzing 6015 erroneous student programs collected by the automated assessment tool (AAT) Mulberry, we found that the experimental group who used the coding manual during learning did not make fewer language specification errors (LSEs) in general. However, for LSEs indicating the actual errors in the program, the coding manual showed significant effects in reducing error frequencies. We also found that the coding manual failed to increase students' confidence in programming and promote learning performance. Possible causes of the ineffectiveness may include high cognitive loads during programming, the productivity of learning from the debugging process, and the incompleteness of the explanations and examples in the coding manual. We recommend that computer programming instructors use AATs or similar tools to collect learning data and identify students' common errors, directly explain identified common LSEs in class, and explicitly teach debugging methods and strategies. Future research should focus on students' self-regulation during programming, better methods of explaining common errors to novices, and the long-term effects of using a coding manual on students' learning in introductory programming.