Evaluating the Impact and Usability of an AI-Driven Feedback System for Learning Design

Despite the momentum that Artificial Intelligence (AI) is gaining in education, its role and impact on teachers' learning design practices are still underexplored. This paper reports an experimental study (N = 38) taking place in a teacher training where an AI-driven feedback system aided teachers in the creation of learning designs. The study analyses the impact that using the AI feedback had on the quality of designs that teachers created, and the usability evaluation of the system. We noticed statistically significant differences between the designs created by the randomly assigned teachers in the experimental (using AI) and control group (without AI), suggesting that AI algorithms specialized to perform specific tasks related to the learning design could help teachers to better meet their design goals. While teachers graded the usability of the feedback system as above average, they also found it easy to use and its functions well integrated. In open-ended questions, teachers expressed doubts about their trust in AI systems and the impact that they may have in school communities, suggesting that future work should explore not only the long-term impact that using AI can have on teachers' design practices, but also on their perceptions and understanding of the technology.