Explore the vast range of titles available.

A review is textual feedback provided by a reviewer to the author of a submitted version. Peer reviews are used in academic publishing and in education to assess student work. While reviews are important to e-commerce sites like Amazon and e-bay, which use them to assess the quality of products and services, our work focuses on academic reviewing. We seek to help reviewers improve the quality of their reviews. One way to measure review quality is through metareview or review of reviews. We develop an automated metareview software that provides rapid feedback to reviewers on their assessment of authors’ submissions. To measure review quality, we employ metrics such as: review content type, review relevance, review’s coverage of a submission, review tone, review volume and review plagiarism (from the submission or from other reviews). We use natural language processing and machine-learning techniques to calculate these metrics. We summarize results from experiments to evaluate our review quality metrics: review content, relevance and coverage, and a study to analyze user perceptions of importance and usefulness of these metrics. Our approaches were evaluated on data from Expertiza and the Scaffolded Writing and Rewriting in the Discipline (SWoRD) project, which are two collaborative web-based learning applications.

Peer assessment means students giving feedback on each other’s work. Dozens of online systems have been developed for peer assessment. They all face many of the same issues. The PeerLogic project is an effort to develop specialized features for peer-assessment systems so that they can be used by multiple systems without the need for re-implementation. In addition, the project maintains a “data warehouse,” which includes anonymized peer reviews from different peer-assessment systems, which are freely made available to researchers.

This work-in-progress research-to-practice paper explores the integration of Large Language Models (LLMs) into the code-review process for open-source software projects developed in computer science and software engineering courses. The focus is on developing an automated feedback tool that evaluates student code for adherence to key object-oriented design principles, addressing the need for more effective and scalable methods to teach software design best practices. The innovative practice involves leveraging LLMs and Retrieval-Augmented Generation (RAG) to create an automated feedback system that assesses student code for principles like SOLID, DRY, and design patterns. It analyzes the effectiveness of various prompting strategies and the RAG integration. Preliminary findings show promising improvements in code quality. Future work will aim to improve model accuracy and expand support for additional design principles.

This Innovative Practice full paper explores how Large Language Models (LLMs) can enhance the teaching of code refactoring in software engineering courses through real-time, context-aware feedback. Refactoring improves code quality but is difficult to teach, especially with complex, real-world codebases. Traditional methods like code reviews and static analysis tools offer limited, inconsistent feedback. Our approach integrates LLM-assisted refactoring into a course project using structured prompts to help students identify and address code smells such as long methods and low cohesion. Implemented in Spring 2025 in a long-lived OSS project, the intervention is evaluated through student feedback and planned analysis of code quality improvements. Findings suggest that LLMs can bridge theoretical and practical learning, supporting a deeper understanding of maintainability and refactoring principles.

Self-assessment is a powerful mechanism for enhancing learning. It encourages students to reflect on how their own work meets the goals set for learning concepts and skills. It promotes metacognition about what is being learned, and effective practices for learning. It encourages students to think about how a particular assignment or course fits into the context of their education. It imparts reflective skills that will be useful on the job or in academic research. Most other kinds of assessment place the student in a passive role. The student simply receives feedback from the instructor or TA. Self-assessment, by contrast, forces students to become autonomous learners, to think about how what they should be learning. Having learned self-assessment skills, students can continue to apply them in their career and in other contexts throughout life. While self-assessment cannot reliably be used as a standalone grading mechanism, it can be combined with other kinds of assessment to provide richer feedback and promote more student “buy-in” for the grading process. For example, an instructor might have students self-assess their work based on a rubric, and assign a score. The instructor might agree to use these self-assigned grades when they are “close enough” to the grade the instructor would have assigned, but to use instructor-assigned grades when the self-grades are not within tolerance. Self-assessment can also be combined with peer assessment to reward students whose judgment of their own work agrees with their peers’. In Calibrated Peer Review, students are asked to rate the work of three of their peers, and then to rate their own work on the same scale. Only after they complete all of these ratings are they allowed to see others’ assessments of their own work. CPR assignments are often configured to award points to students whose self-ratings agree with peers’ ratings of their work. The Coursera MOOC platform employs a similar strategy. Recently a “calibrated self-assessment” strategy has been proposed, that uses self-assigned scores as the sole grading mechanism for most work, subject to spot-checks by the instructor. Self-assigned grades are trusted for those students whose spot-checked grades are shown to be valid; students whose self-assigned grades are incorrect are assigned a penalty based on the degree of misgrading of their work. In self-assessment, as in other kinds of assessment, a good rubric is essential to a good review process. It will include detailed criteria, to draw students’ attention to important aspects of the work. The criteria should mention the goals and keywords of the assignment, so that students will focus on their goals in assessment as well as their writing. This paper will cover the benefits of self-assessment, and then provide several examples of how it can be combined with other assessments.

Peer review is becoming more common across the curriculum. But, if it is to be effective, students need to know how to provide effective feedback. It is important to construct a rubric that draws students’ attention to the important points of the work they are reviewing. But that is not enough; students also need instruction in how to provide comments that their peers can and will use in revising their work. This involves learning how to provide constructive suggestions. It also means understanding the way the author will react to the review, and using gentle enough terminology so that the words do not get in the way of understanding the reviewer’s advice. Authors can also help reviewers learn the ropes by giving them feedback on the effectiveness of reviews of their work.

This full research-to-practice paper explores approaches for developing course chatbots by comparing low-code platforms and custom-coded solutions in educational contexts. With the rise of Large Language Models (LLMs) like GPT-4 and LLaMA, LLM-based chatbots are being integrated into teaching workflows to automate tasks, provide assistance, and offer scalable support. However, selecting the optimal development strategy requires balancing ease of use, customization, data privacy, and scalability. This study compares two development approaches: low-code platforms like AnythingLLM and Botpress, with custom-coded solutions using LangChain, FAISS, and FastAPI. The research uses Prompt engineering, Retrieval-augmented generation (RAG), and personalization to evaluate chatbot prototypes across technical performance, scalability, and user experience. Findings indicate that while low-code platforms enable rapid prototyping, they face limitations in customization and scaling, while custom-coded systems offer more control but require significant technical expertise. Both approaches successfully implement key research principles such as adaptive feedback loops and conversational continuity. The study provides a framework for selecting the appropriate development strategy based on institutional goals and resources. Future work will focus on hybrid solutions that combine low-code accessibility with modular customization and incorporate multimodal input for intelligent tutoring systems.

A Human-in-the-Loop (HITL) approach leverages generative AI to enhance personalized learning by directly integrating student feedback into AI-generated solutions. Students critique and modify AI responses using predefined feedback tags, fostering deeper engagement and understanding. This empowers students to actively shape their learning, with AI serving as an adaptive partner. The system uses a tagging technique and prompt engineering to personalize content, informing a Retrieval-Augmented Generation (RAG) system to retrieve relevant educational material and adjust explanations in real time. This builds on existing research in adaptive learning, demonstrating how student-driven feedback loops can modify AI-generated responses for improved student retention and engagement, particularly in STEM education. Preliminary findings from a study with STEM students indicate improved learning outcomes and confidence compared to traditional AI tools. This work highlights AI's potential to create dynamic, feedback-driven, and personalized learning environments through iterative refinement.

Resources for Flipping Classes“Flipped” classes have surged in popularity over the last three years, driven by the ease ofrecording and posting video content for students to watch, and the need during class timeto compete with distractions from portable electronic devices. The instructor who desiresto “flip” a class confronts two issues: how to locate or create content for use outside ofclass, and how to use time effectively during class. This paper is not intended to presentthe benefits of flipping, or describe particular techniques for doing so, but rather tosample the resources available for flipping, and suggest strategies for class time thatreaders may wish to explore on their own.A large amount of content for practically any subject is available on YouTube, but thelevel of coverage and quality is variable, so searching it takes time. TedEd(http://ed.ted.com) has videos on a large number of subjects, but coverage is quiteuneven; for example, there is nothing on “logic design” or “Fourier transform.”TeacherTube (http://www.teachertube.com) seems to have more engineering-orientedmaterial, even though its focus is on K-12. Academic Earth (http://academicearth.org)has a more advanced focus. It archives complete sets of classroom-capture videos from13 prestigious universities (many Ivies, Stanford, Carnegie Mellon, etc.). Currently, ithas 81 engineering courses. More focused video collections can be found atSolidProfessor, for CAD classes (SolidWorks, MasterCAM, AutoCAD), etc. Diligentinchas no videos, but rather sets of projects assignable in ECE courses. Khan Academy(http://www.khanacademy.org) has videos on individual topics, rather than classroomcapture. They seem easier to drop into classes, since they are more self-contained.It is also possible to use videos from MOOCs. This requires permission of the copyrightholder, but in the author’s experience, this is not difficult to obtain. Sometimes, a class atone university can use an entire course of videos produced at another, as in the wellknown San Jose State experiment with the MIT Engineering, Electronics, and Circuitscourse.However, most instructors record at least some of their own videos. Camtasia Studio isthe most popular application for creating them. It allows multiple-choice and fill-in-the-blank quizzes to be embedded in videos. It can also generate captions from PowerPointnotes panes; this facilitates developing accessible content. A variety of other platformsare used: iMovie, Office Mix, ScreenFlow, Snag It, Snapzpro, and Tegrity also haveuseful features for recording.Among articles and web sites, the Flipped Learning Network (http://flippedclassroom.org)is most frequently mentioned. It collects articles, press reports, and research studies on the“flipped” model, and also has an “Online Community of Practice” with over 22,000participants.To make sure students prepare for class, they may be given a quiz or a reflection activityat the start of class (e.g., summarize what you learned from the online material). Classtime can be spent in peer-instruction activities, similar to lab experiments. Various kindsof reflective activities can be used to end class, and/or introduce the topic for the next setof video lectures.

Scalable Assessment: Present and FutureA perennial problem in teaching is securing enough resources to adequately assessstudent work. In recent years, tight budgets have constrained the dollars available to hireteaching assistants. Concurrent with this trend, the rise of MOOCs, has raised assessmentchallenges to a new scale. In MOOCs, it’s necessary to get feedback to, and assigngrades to, thousands of students who don’t bring in any revenue. As MOOCs begin tocredential students, accurate assessment will become even more important. These twodevelopments have created an acute need for scalable assessment mechanisms, to assesslarge numbers of students without a proportionate increase in costs.There are three main approaches to scalable assessment: autograding, peer review, andautomated essay scoring. Autograding is provided, on a basic scale, by any LMS that canadminister quizzes made up of multiple-choice or fill-in-the-blank questions.Autograding is also done by publishers’ apps like Wiley Plus, third-party grading systemslike Webassign, and content-specific grading applications like Web-CAT for computerprogramming assignments. Autograding scales very well, because almost all of the effortis expended in creating the quiz, which can then be administered to any number ofstudents automatically. Autograding systems can randomize the ordering of questionsand answers, and randomize parameters to numeric problems, making it harder to cheat.With a little more effort, multiple-choice distractors can be keyed to specificmisconceptions. Advanced analytics can identify which material the students are havingtrouble with.Peer review is supported by almost every LMS, as well as standalone systems likeCalibrated Peer Review, Peerceptiv, and Peer Scholar. It is also a feature of severalMOOCs, notably Coursera and Canvas. Its greatest strength is the ability to give eachstudent formative assessment on their work. When used for summative assessment inMOOCs, its accuracy is questionable, especially for assignments that the students do notunderstand well. However, researchers are working on better techniques to identifyaccurate reviews, which would improve summative scores. Researchers are alsodeveloping techniques to allow peer reviewers to annotate, and respond to annotations, onsubmitted documents. Other research involves natural-language processing techniques toestimate the quality of a review before it is submitted, and give feedback to the revieweron how to improve the review.Automated essay scoring uses software that predicts how an instructor would score apiece of prose, by using metrics such as correlation of its vocabulary with essays scoredhigh by humans, average word length, and number of grammatical errors. An instructorneeds to train an AES system by scoring a number of essays (e.g., 100) to teach thesystem which characteristics are important. AES is used by MOOCs such as EdX, aswell as for high-stakes educational testing, and research is ongoing in applying similartechniques to the grading of non-prose submissions.The presentation will cover the current capabilities and research directions for thesesystems, and also show how they can be used to improve assessment in any context, byproviding greater formative feedback to students.