Explore the vast range of titles available.

A review is textual feedback provided by a reviewer to the author of a submitted version. Peer reviews are used in academic publishing and in education to assess student work. While reviews are important to e-commerce sites like Amazon and e-bay, which use them to assess the quality of products and services, our work focuses on academic reviewing. We seek to help reviewers improve the quality of their reviews. One way to measure review quality is through metareview or review of reviews. We develop an automated metareview software that provides rapid feedback to reviewers on their assessment of authors’ submissions. To measure review quality, we employ metrics such as: review content type, review relevance, review’s coverage of a submission, review tone, review volume and review plagiarism (from the submission or from other reviews). We use natural language processing and machine-learning techniques to calculate these metrics. We summarize results from experiments to evaluate our review quality metrics: review content, relevance and coverage, and a study to analyze user perceptions of importance and usefulness of these metrics. Our approaches were evaluated on data from Expertiza and the Scaffolded Writing and Rewriting in the Discipline (SWoRD) project, which are two collaborative web-based learning applications.

Over the past two decades, there has been an explosion of innovation in software tools that encapsulate and expand the capabilities of the widely used student peer assessment. While the affordances and pedagogical impacts of traditional in-person, “paper-and-pencil” peer assessment have been studied extensively and are relatively well understood, computerized (online) peer assessment introduced not only shifts in scalability and efficiency, but also entirely new capabilities and forms of social learning interactions, instructor leverage, and distributed cognition, that still need to be researched and systematized. Despite the ample research on traditional peer assessment and evidence of its efficacy, common vocabulary and shared understanding of online peer-assessment system design, including the variety of methods, techniques, and implementations, is still missing. We present key findings of a comprehensive survey based on a systematic research framework for examining and generalizing affordances and constraints of online peer-assessment systems. This framework (a) provides a foundation of a design-science metatheory of online peer assessment, (b) helps structure the discussion of user needs and design options, and (c) informs educators and system design practitioners. We identified two major themes in existing and potential research—orientation towards scaffolded learning vs. exploratory learning and system maturity. We also outlined an agenda for future studies.

Peer assessment means students giving feedback on each other’s work. Dozens of online systems have been developed for peer assessment. They all face many of the same issues. The PeerLogic project is an effort to develop specialized features for peer-assessment systems so that they can be used by multiple systems without the need for re-implementation. In addition, the project maintains a “data warehouse,” which includes anonymized peer reviews from different peer-assessment systems, which are freely made available to researchers.

This work-in-progress research-to-practice paper explores the integration of Large Language Models (LLMs) into the code-review process for open-source software projects developed in computer science and software engineering courses. The focus is on developing an automated feedback tool that evaluates student code for adherence to key object-oriented design principles, addressing the need for more effective and scalable methods to teach software design best practices. The innovative practice involves leveraging LLMs and Retrieval-Augmented Generation (RAG) to create an automated feedback system that assesses student code for principles like SOLID, DRY, and design patterns. It analyzes the effectiveness of various prompting strategies and the RAG integration. Preliminary findings show promising improvements in code quality. Future work will aim to improve model accuracy and expand support for additional design principles.

This Innovative Practice full paper explores how Large Language Models (LLMs) can enhance the teaching of code refactoring in software engineering courses through real-time, context-aware feedback. Refactoring improves code quality but is difficult to teach, especially with complex, real-world codebases. Traditional methods like code reviews and static analysis tools offer limited, inconsistent feedback. Our approach integrates LLM-assisted refactoring into a course project using structured prompts to help students identify and address code smells such as long methods and low cohesion. Implemented in Spring 2025 in a long-lived OSS project, the intervention is evaluated through student feedback and planned analysis of code quality improvements. Findings suggest that LLMs can bridge theoretical and practical learning, supporting a deeper understanding of maintainability and refactoring principles.

Evaluating the performance of Large Language Models (LLMs) is a critical yet challenging task, particularly when aiming to avoid subjective assessments. This paper proposes a framework for leveraging subjective metrics derived from the class textual materials across different semesters to assess LLM outputs across various tasks. By utilizing well-defined benchmarks, factual datasets, and structured evaluation pipelines, the approach ensures consistent, reproducible, and bias-minimized measurements. The framework emphasizes automation and transparency in scoring, reducing reliance on human interpretation while ensuring alignment with real-world applications. This method addresses the limitations of subjective evaluation methods, providing a scalable solution for performance assessment in educational, scientific, and other high-stakes domains.

Self-assessment is a powerful mechanism for enhancing learning. It encourages students to reflect on how their own work meets the goals set for learning concepts and skills. It promotes metacognition about what is being learned, and effective practices for learning. It encourages students to think about how a particular assignment or course fits into the context of their education. It imparts reflective skills that will be useful on the job or in academic research. Most other kinds of assessment place the student in a passive role. The student simply receives feedback from the instructor or TA. Self-assessment, by contrast, forces students to become autonomous learners, to think about how what they should be learning. Having learned self-assessment skills, students can continue to apply them in their career and in other contexts throughout life. While self-assessment cannot reliably be used as a standalone grading mechanism, it can be combined with other kinds of assessment to provide richer feedback and promote more student “buy-in” for the grading process. For example, an instructor might have students self-assess their work based on a rubric, and assign a score. The instructor might agree to use these self-assigned grades when they are “close enough” to the grade the instructor would have assigned, but to use instructor-assigned grades when the self-grades are not within tolerance. Self-assessment can also be combined with peer assessment to reward students whose judgment of their own work agrees with their peers’. In Calibrated Peer Review, students are asked to rate the work of three of their peers, and then to rate their own work on the same scale. Only after they complete all of these ratings are they allowed to see others’ assessments of their own work. CPR assignments are often configured to award points to students whose self-ratings agree with peers’ ratings of their work. The Coursera MOOC platform employs a similar strategy. Recently a “calibrated self-assessment” strategy has been proposed, that uses self-assigned scores as the sole grading mechanism for most work, subject to spot-checks by the instructor. Self-assigned grades are trusted for those students whose spot-checked grades are shown to be valid; students whose self-assigned grades are incorrect are assigned a penalty based on the degree of misgrading of their work. In self-assessment, as in other kinds of assessment, a good rubric is essential to a good review process. It will include detailed criteria, to draw students’ attention to important aspects of the work. The criteria should mention the goals and keywords of the assignment, so that students will focus on their goals in assessment as well as their writing. This paper will cover the benefits of self-assessment, and then provide several examples of how it can be combined with other assessments.

Peer review is becoming more common across the curriculum. But, if it is to be effective, students need to know how to provide effective feedback. It is important to construct a rubric that draws students’ attention to the important points of the work they are reviewing. But that is not enough; students also need instruction in how to provide comments that their peers can and will use in revising their work. This involves learning how to provide constructive suggestions. It also means understanding the way the author will react to the review, and using gentle enough terminology so that the words do not get in the way of understanding the reviewer’s advice. Authors can also help reviewers learn the ropes by giving them feedback on the effectiveness of reviews of their work.

This full research-to-practice paper explores approaches for developing course chatbots by comparing low-code platforms and custom-coded solutions in educational contexts. With the rise of Large Language Models (LLMs) like GPT-4 and LLaMA, LLM-based chatbots are being integrated into teaching workflows to automate tasks, provide assistance, and offer scalable support. However, selecting the optimal development strategy requires balancing ease of use, customization, data privacy, and scalability. This study compares two development approaches: low-code platforms like AnythingLLM and Botpress, with custom-coded solutions using LangChain, FAISS, and FastAPI. The research uses Prompt engineering, Retrieval-augmented generation (RAG), and personalization to evaluate chatbot prototypes across technical performance, scalability, and user experience. Findings indicate that while low-code platforms enable rapid prototyping, they face limitations in customization and scaling, while custom-coded systems offer more control but require significant technical expertise. Both approaches successfully implement key research principles such as adaptive feedback loops and conversational continuity. The study provides a framework for selecting the appropriate development strategy based on institutional goals and resources. Future work will focus on hybrid solutions that combine low-code accessibility with modular customization and incorporate multimodal input for intelligent tutoring systems.

A Human-in-the-Loop (HITL) approach leverages generative AI to enhance personalized learning by directly integrating student feedback into AI-generated solutions. Students critique and modify AI responses using predefined feedback tags, fostering deeper engagement and understanding. This empowers students to actively shape their learning, with AI serving as an adaptive partner. The system uses a tagging technique and prompt engineering to personalize content, informing a Retrieval-Augmented Generation (RAG) system to retrieve relevant educational material and adjust explanations in real time. This builds on existing research in adaptive learning, demonstrating how student-driven feedback loops can modify AI-generated responses for improved student retention and engagement, particularly in STEM education. Preliminary findings from a study with STEM students indicate improved learning outcomes and confidence compared to traditional AI tools. This work highlights AI's potential to create dynamic, feedback-driven, and personalized learning environments through iterative refinement.