Experiences with informal student panel for continuous formative course evaluation

Background: Summative student evaluations of teaching is widely used to evaluate course content and instruction. However, this feedback does not benefit the students who are pro­ viding the feedback, and may postpone feedback such that the evaluation is not useful in correcting issues in the course. There are several issues with this type of feedback that are tacled in this work: (i) The evaluation is not a help for the current students. (ii) The feedback is often imprecise. (iii) Evaluations focus on the quality of the teaching, rather than the impact of the learning. Previous work [1, 2] have shown that continuous student involvement can improve the feedback received. The idea in this project is to perform this student panel interaction in an informal setting, conti­ nously throughout the semester. Methods: This intervention has been performed for two consecutive semesters in different courses. During the first lecture of the semester, students were recruited to participate in a weekly informal meeting to discuss the instruction in the course. Weekly meetings were held for one full semester, with each meeting lasting around 45 minutes. Notes were taken on all actionable items, and a summarizing report was written. This intervention has been evaluated through the inspection of actionable items in the notes, as well as the implemented items. Results: 104/116 actionable items were identified during spring/fall 2020, of which 57/65 were im­ plemented immidiately. Participation in the student panel group was high and stable.


INTRODUCTION
One of the most prominent challenges we face as educators, is getting useful feedback on our teaching. Student evaluation of teaching is widely used in Universities globally. This evaluation often takes the form of a questionnaire sent out to students after the exam. Some particularly enterprising institutions also include a questionnaire during the semester. These methods of evaluating teaching have several weaknesses: (i) The evaluation is done too late to help the students who are doing the evaluation, (ii) The feedback is often unclear about underlying issues, (iii) the feedback generally does not include sug gestions for improvements, (iv) when interventions are implemented, the same students are not available to evaluate if the interventions helped learning. To alleviate these issues, many lecturers turn to peers, to sit in on a lecture and give feedback. This method of evaluation improves points (i) and (iii), but has little effect on the remaining issues. In an attempt to provide a more holistic evaluation of courses, we have implemented student panels in two courses. These panels comprised one meeting per week, where a group of volunteer students met with the course staff, and discussed the lectures, assignments, and group work that has been done the last week. In this paper, we will argue that this method of course evaluation is more efficient than both questionnaires and peer evaluation for identifying and improving issues with courses.

METHODS
This project has been implemented as a tool during a normal semester, as a part of the evaluation of two courses. As such, the methods described here are intended as the methods of implementation of this type of intervention. We describe the recruitment, meetings, and analysis in more detail. * edvard.pedersen@uit.no

Recruitment
During the first lecture of the course, a few minutes were devoted to explaining what the student panel is, and attempting to recruit students who were willing to spend an hour per week discussing issues, improvements, and other things related to the course. This was repeated when the attendance to the weekly meetings dropped below 4, resulting in two inlecture recruitment periods per semester.

Meetings
Meetings were primarily held physically, at one of the meeting rooms on campus. Due to the pandemic, around half the meetings in the first semester were held digitally, as well as a smaller portion of the second semester meetings. The digital meetings were held on Discord. The meetings were headed by the staff. An outline of the notes were shared, which included a list of topics (e.g. "Lectures", "Assignment", "Group work in last lecture"). These topics were discussed freely, to identify which parts worked well, and which parts could be improved, as well as how improvements can be implemented. During the discussion of the topics, notes were taken and displayed, so that errors or misunderstandings could be reduced in the notes. The discussion took the form of the course responsible asking a question about the topic (e.g. "What did you think of the lecture last week?"), and the discussion flowed freely from there, with additional questions asked where appropriate.

Analysis
The points raised during the meetings are split into several categories: • Feedback, which encompasses points on which parts of the course work well (i.e. what not to change).
• Immidiately actionable, which includes points which are implemented immidiately (for the next lecture or assignment) and points which are implemented during the semester.
• Actionable, which includes retrospective points, and changes that require more preperation or re structuring than can be achieved within the current semester.
• Nonactionable, which includes points where no feasable solution or mitigating actions have been identified through our dicussion (i.e. "unsolvable" problems).
The determination of which items belonged to which category was done after the fact. The results of this classification are presented as the results of the student panel intervention. No further strict analysis is performed on the data. However, the distribution of types of feedback throughout the semester is visually inspected.

RESULTS
In the first semester, 12 meetings were performed. 76 feedback, 57 implemented during the semester, 47 actionable, 7 were nonactionable. (83 nonactionable, 104 actionable) In the second semester, 15 meetings were performed. 122 feedback, 65 implemented during the semester, 51 actionable, 14 were nonactionable. (136 nonactionable, 116 actionable) For both semesters, there were around 5 regular student participants in each student panel meeting. The meetings were held both physically and digitally, with a small number using both physical and digital participation. No clear patterns were found in the data, due to the relatively high variability.

Course evaluation
These numbers are not significant without some background information about alternatives. Since the second semester course was also subject to a traditional course evaluation in the same semester as the stu dent panel was used, we can use the number from this, in the same classifications. The course evaluation from the administration provided 1 feedback, 4 actionable, 2 nonactionable. (2 already implemented before the semester started, so the background for the feedback is impossible to determine) Comparing the results from the student panel meetings and the "standard" course evaluation is not an applestoapples comparison, so the results should be taken with a grain of salt. We have also not con sidered the regular feedback from students during the course. These choices were made due to lacking alternative The methodology of the meetings is a potential issue. Due to the involvement of the lecturer in the discussion, students may be hesitant to bring up certain feedback, and the success of the group is very dependant on the culture of the group being such that students are not worried about bringing up top ics. This is the reason for the informal structure of the discussions, to attempt to generate an environment where this issue can be mitigated. The notetaking and classification may be imprecise, and does not fully represent the discussions. This is a compromise between keeping the discussion flowing, and recording precicely. Audio recordings of the meetings could alleviate this, but would make the already significant time requirements of the student panel arrangement more severe, as well as potentially limiting discus sion. Due to the voluntary nature of the student panels, some bias is expected in which students provide feedback. The time taken per week may be daunting for some educators, and a prerequisite for this is being able to spend one hour per week on this, which may be difficult to achieve at some institutions. The overall methodology is intended as a realistic implementation of the intervention. As such, there is little basis for comparison with other interventions with the same group of students. However, this provides the benefit that the results are expected to be similar in other courses.

CONCLUSION
Despite the limitation described in the discussion, the results indicate a significant benefit for the lecturer and (indirectly) students in performing student panel discussions. The student panel intervention should be considered as a supplement to, or replacement of, traditional course evaluations.