Automated formative and summative feedback: Mechanical Engineering problem sets

Student homework sheets were not marked in the past on this module, so automation allows feedback on formative assessments where previously there was none. More specifically, it automates ‘low level’ feedback that stops students getting stuck on questions, and allows them to go deeper into their homework before requiring help. The subsequent contact time with teachers is higher quality due to deeper discussions. See this article for more information.

Automation for summative assessments improves staff time management at a busy time of year and increases the speed and consistency of feedback to students.  

The reason for the low stakes assessment is two-fold; it is an entry point to the technology for the teacher (allowing development of the system), and it introduces students to this type of assessment and feedback without a high mark penalty if they get things wrong. The progress tests are a good vehicle for this innovation as they are summative (so students take them seriously), but the weighting is low. 

Initially the assessments were hosted on Mobius platform. However, the following issues were identified:  

• General user experience (student and staff) 
• Restriction on programming language to program feedback (Maple only). 
• Restriction on error-carried forward capability. 
• Restriction on feedback capabilities - because it is ‘grading’ focussed. 
• Restriction on analytics. 

The Lambda feedback software was designed to meet the needs that Mobius couldn’t. In addition to that the software will soon allow the teacher to parameterise a question, so that the exact content seen by a student is unique to them and so the correct answer is different for each student. There are obvious issues with the integrity of remotely-delivered summative tests, so this system allows teachers to change some (or none) of the questions, which prevents students colluding on their answers.  

For the self-study exercises, the same considerations were made as for any worksheet-style assessment, including, from the student perspective: the purpose of the assessment e.g. which Intended Learning Outcome is being assessed (what concept, what skill, what knowledge, what misconception?); how long will it take to complete?; How clear are the instructions for the assessment? How much interest/engagement does it generate for the student? How easy is it to navigate through the assessment, e.g. how good is the user experience?  

From the staff perspective, the teacher needs to consider how to construct a question to enable generation of the formative automated feedback, so must consider the structure of the question, and any potential ambiguity in the answer, e.g. use of different symbols for mathematical concepts.  

The formative feedback assessment is new and innovative; the feedback provided does not cause  any known disruption to any other module, and supports students’ learning of essential concepts in a way that is beneficial to the higher level modules that they take in later years in their degrees.   

For the summative assessments, changes (such as from paper based to computer based and from manually marked to automatically marked) were applied across the whole cohort to ensure consistency of the marking.  

As preparation for the formative assessments students are given general advice on study methods and feedback literacy. Feedback literacy is a two-way thing between students and teachers: students need to receive a lot of feedback before they get skilled at knowing how to react to it; teachers need to provide quality feedback. Students can check their understanding of the module content and concepts during bi-weekly tutorials with staff.  

As preparation for the summative assessments students are given a briefing in a lecture and are given written information about it on Blackboard; they are directed towards the ILOs, the syllabus and the module descriptor. They also take a readiness test (to check they can use the software). For one cohort (who were particularly stressed) a mock test was run too. Students are encouraged to complete the formative worksheets as preparation for the summative assessment.  

As all the assessments are online, useful analytics can be easily generated to track student engagement, e.g. showing peaks in online activity on the day of the lecture; showing that many students access the homework sheet as soon as it is set, then engagement tails off; that there is a gradual decline in student engagement through the time, but that overall students do keep working and keep trying to catch up.  

Feedback for both formative and summative assessments is automated, as discussed above in the explanation of the Lambda feedback software. It is completely consistent and objective, but also manually checked. Any changes that need to be made (e.g. if an error is spotted in a question) are applied algorithmically to the whole cohort. Students can ask questions about their feedback during tutorials throughout term, or in a dedicated tutorial feedback session in January, after the summative assessment has been completed.  

For the summative assessment the feedback is not instant, as teachers check the grading before releasing marks. Feedback is currently limited to grades (marks) with no comments. 

The summative tests were designed to be delivered online, and can be taken remotely (they are currently taken in-person, in computer rooms, but worked well entirely remotely during the pandemic). The online mode of assessment hasn’t affected the design of the assessment, except that it increases the importance of parameterised questions (which are bespoke for each student) for the summative test.