LLM-Based Academic Assessment Generation
Fine-tuned language models generating learning-outcome-aligned academic assignments with semantic evaluation, LLM-as-judge review, and human-in-the-loop workflows. Accepted poster at ICoDT2 December 2025.
Challenge
Educators spend significant time creating assignments that align with course learning outcomes (CLOs). Manual assessment generation is time-consuming, and ensuring consistent alignment with educational objectives requires expertise and repetition. The research explores:
- Whether LLMs can generate pedagogically sound assignments
- How to align generated content with specific learning outcomes
- Evaluation methods beyond subjective human judgment
- Integration of automated generation with human oversight
- Fine-tuning strategies for educational domain adaptation
Technical Approach
Model Selection & Fine-Tuning
Experimented with Llama family and Mistral models. Applied LoRA (Low-Rank Adaptation) and QLoRA (Quantized LoRA) for parameter-efficient fine-tuning on educational assessment datasets. This approach adapts foundation models to education-specific generation without full model retraining.
CLO-Aligned Generation
Structured generation to explicitly reference course learning outcomes. Prompting strategies and fine-tuning examples ensure generated assignments test specific competencies rather than arbitrary knowledge recall. The system takes learning outcomes as input and produces aligned assessment items.
Multi-Level Evaluation
Implemented comprehensive evaluation framework combining multiple methods:
- Semantic Similarity: BERT Score and other embedding-based metrics measure content quality
- Traditional Metrics: BLEU, ROUGE provide reference-based comparison
- LLM-as-Judge: Use capable LLMs to evaluate generation quality, alignment, and pedagogical value
- Human Review: Subject matter experts assess whether generated content meets educational standards
Human-in-the-Loop Workflow
Designed system assuming educators remain decision-makers. Generated assignments route through review interface where instructors accept, modify, or reject content. This keeps educators in control while reducing repetitive work.
Research Outcomes
The research demonstrates that fine-tuned LLMs can generate educationally aligned content when combined with proper evaluation and human oversight. Presented as accepted poster at ICoDT2 conference in December 2025.
Technologies & Methods
- Models: Llama family, Mistral
- Fine-Tuning: LoRA (Low-Rank Adaptation), QLoRA (Quantized LoRA)
- Evaluation: BERT Score, BLEU, ROUGE, LLM-as-judge, human review
- Workflow: CLO-aligned generation, human-in-the-loop review
- Framework: PyTorch, Hugging Face transformers, PEFT library
Publication
Accepted poster presentation at ICoDT2 (International Conference on Digital Transformation) in December 2025. The research contributes to understanding how generative AI can augment rather than replace human expertise in education.
Lessons for LLM Integration
- Fine-tuning matters: Domain adaptation through LoRA/QLoRA significantly improves generation quality for specialized tasks compared to zero-shot prompting.
- Evaluation needs multiple lenses: No single metric captures LLM quality. Combining semantic, traditional, automated, and human evaluation provides comprehensive assessment.
- Human-in-the-loop is essential: For high-stakes domains like education, LLMs should augment expert judgment, not replace it. Workflow design must preserve human decision authority.
- Alignment is explicit work: Generating content that meets specific objectives (like CLOs) requires structured prompting, fine-tuning examples, and evaluation criteria explicitly targeting those objectives.
- Research methodology applies to production: Rigorous evaluation frameworks from research transfer directly to production LLM systems requiring quality assurance.
Discuss Your LLM Application
If you need content generation aligned with specific criteria, evaluation frameworks for LLM outputs, or human-in-the-loop workflows, we can assess feasibility and propose an implementation approach.
Start a Conversation →