I have now completed the development of an LLM-augmented, emotionally intelligent pedagogical AI conversational agent (called AIvaluate) and tested it with students. I have written up the study and submitted it for peer-review at (Springer) Educational Technology & Research Development (ETRD), a Q1 journal for educational technology. Stay tuned for the full study once peer-reviewed.
Abstract
Performance-based assessments, such as oral presentations and viva voce exams, are widely recognised for their pedagogical benefits but can induce significant student anxiety, negatively impacting performance and motivation. The integration of AI and educational technology offers a promising avenue for creating more emotionally sensitive and supportive assessment environments. This study investigates the use of AIvaluate, an emotionally intelligent LLM-augmented pedagogical AI conversational agent, designed to reduce student anxiety during oral performance-based assessments. A counterbalanced mixed-methods study design was employed, involving 35 pre-university students who participated in both face-to-face and virtual AIvaluate performance-cased assessments. Data was collected through real-time emotional state reporting during both assessment formats, System Usability Scale (SUS) questionnaires and post-task open-response surveys and follow-up interviews captured user preferences and experience. Quantitative and qualitative analyses were conducted to compare anxiety levels, systems usability, and user experiences of both face-to-face and virtual AI assessment formats. Students reported significantly lower anxiety levels during AIvaluate assessments compared to face-to-face sessions (p = 0.028). The overall SUS score for AIvaluate indicated “good” usability, surpassing the face-to-face format. Thematic analysis revealed key advantages of AIvaluate, including reduced pressure, flexible response options, and ease of use. However, technical challenges, and a lack of human interaction, were noted as limitations. Equally, face-to-face assessments were praised for their dynamic engagement and real-time feedback but were associated with higher anxiety levels. AIvaluate represents a promising tool for reducing student anxiety in performance-based assessments, whilst maintaining high levels of usability and acceptance. Future research could explore the value of hybrid assessment models that combine AI conversational agent-based and face-to-face assessment formats. These could address diverse student needs and capitalise on the strengths of both approaches to foster more inclusive and effective assessment environments. There is also a need to focus on addressing technical limitations and the long-term impact of AI conversational agent-driven assessments on student performance and well-being.
Keywords
Artificial intelligence, Conversational agent, Chatbot, Assessment, Education, Emotional Intelligence, Performance based assessment, Student anxiety, Student well-being, Generative AI, LLM
