Experiments with Auto-generated Socratic Dialogue for Source Code Understanding

Intelligent Tutoring Systems have been proven to generate excellent learning outcomes in many domains such as physics, mathematics and computer programming. However, they have seen relatively little use in training and school classrooms due to the time and cost of designing and authoring. We developed an authoring tool for dialogue-based intelligent tutoring system for programming called Auto-author to reduce the time and cost. The tool allows teachers to create fully functional Socratic tutoring dialogue for learning programming from Java code. First, we conducted a controlled experiment on 45 introductory to programming students to assess auto-authored tutoring dialogues' learning outcomes. The result shows that the auto-authored dialogues improved students' programming knowledge by 43% in terms of learning gain. Secondly, we conducted a survey of auto-authored tutoring dialogues by introductory to programming course instructors to evaluate the dialogues' quality. The result shows that the instructors rated the questions as agree or strongly agree. However, the instructors suggested that more improvement is required to help students develop a robust understanding of programming concepts.