Presenting ISILA Insights at ICSLE 2025: RAG vs. Vanilla LLaMA for Actionable Early Intervention in Learning Analytics

ISILA Member Ramy Elmoazen has presented a study titled “Assessing AI-Driven Learning Analytics Recommendations: Comparative Analysis of RAG and vanilla LLaMA” at the International Conference on Smart Learning Environments (ICSLE 2025).

This work directly advances the ISILA project’s core mission of improving the quality and sustainability of learning through early intervention methods based on learning analytics, by evaluating how advanced AI techniques can generate more effective, context-aware recommendations for timely educational support.

The study compares AI-driven learning analytics recommendations generated by the open-source LLaMA model with and without RAG. Using a structured prompt design, the study assessed 30 questions related to ten topics with a focus on data collection, measurement, and optimization. Model responses were assessed using six evaluation criteria.

Results indicate RAG’s superior performance in four dimensions: Justification, Usefulness, Coherence, and Implementability, while the vanilla LLaMA showed superior performance in Equity. Both configurations achieved comparable scores in Accuracy with more explanatory depth and contextual adaptation in RAG. These findings highlight RAG strengths in context-specific, practical recommendations but identify its equity limitations—insights that can inform the design of fairer, more sustainable AI-driven early interventions within the ISILA framework.”