PH350 DMcK Analysis-Section2-1.pdf
Analysis of the feedback forms showing a generally positive reaction to most of the changes that have been implemented so far.
Summary
Reorganisation of 3rd year teaching lab including changes to assessment and feedback practise and allocation of student experiments.
Context
Department of Physics
Faculty of Science
Contact Details
Dr David McKee
http://www.strath.ac.uk/staff/mckeedaviddr/
Themes
Themes
Rationale
The 3rd year lab is an important transition point in the development of UG students as practical experimentalists, moving from 1st and 2nd year labs where experimental methods are spelt out in detail to 4th year projects which are undertaken in research groups on active research topics. The aim is to increase the level of independent thought, critical analysis of data and refinement of lab techniques.
At the time of taking over management of the lab there had been a long period of student dissatisfaction with various aspects of the lab which was being regularly raised at staff-student committee. Whilst only some of the complaints were justified in reality, my role was to alleviate both real and perceived flaws in the running of the lab thereby reducing complaints and improving student satisfaction with the learning experience. Of course, there was a corresponding requirement to do so without reducing the quality of the educational experience and without any increase in staffing level.
Student complaints centred on difficulties in accessing demonstrating staff (low numbers and staff spending time marking lab books during labs), perceived differences in the marking between different demonstrators (not borne out by statistical analysis of marking) and difficulties in the allocation of experiments with some being perceived as being much harder / easier than others. The latter complaint was somewhat reinforced by a points system that required students to reach a minimum points tally over the year, with different experiments awarded a points value between 1-3 depending on perceived difficulty. Further issues relate to the age of equipment and the quality of lab notes.
The lab was assessed by demonstrators marking lab books. Students were also required to produce a single formal lab report, building up each section with feedback and advice from demonstrators over the course of the year.
Under the previous format, students would complete an experiment, have it marked in the lab by a demonstrator who could perform a mini-viva in parallel and then be allocated a new experiment, though the availability was restricted to those currently unoccupied at that particular time. Students quickly formed an opinion of which demonstrators were lenient, which experiments were hard and the perception of an uneven playing field was rapidly and repeatedly formed.
My primary initial goal has been to reduce the scope for this perception of an uneven experience. Experiments have been collated into 4 broad themes. The student cohort has also been separated into 4 groups, and each group cycles through the lab with students performing one experiment from each of the themes. Experiments are randomly allocated.
Two demonstrators have been allocated to each experimental theme, with one being required to mark lab books and the other marking formal lab reports. Students perform 4 experiments over the course of the academic year and produce a formal lab report at the end of each experiment. This means that each student is marked once by each of the demonstrators, eliminating the perception of imbalance between demonstrators. Student feedback on these changes has been very encouraging to date.
These changes have the additional benefit of ensuring that students have more time to complete each experiment (5 weeks total) giving more time to address experimental design, uncertainties analysis and preparation of a formal report. Under the previous system there was no time constraint on how long it took a student to complete an experiment which often resulted in experiment blocking.
I have changed the structure of the formal report, requiring them to produce a camera-ready research paper in the style of a camera-ready journal paper. The intention is to provide stronger emphasis on the need to present results in a formal setting, further develop scientific writing style and report structuring and thereby strengthen preparation for final year projects. This means that students are being assessed regularly on formal writing skills than previously, and it also gives staff an opportunity to provide regular feedback and students multiple opportunities to improve performance.
Successes
Feedback from students reporting to the Staff-Student committee has been very positive. Complaints about the 3rd year lab have essentially stopped. There appears to be a better sense of an even playing field.
We have actually delivered the lab this year with one fewer member of staff than last year.
A number of 5th year students have recently volunteered to provide peer-support in the lab spending up to two hours a week demonstrating. I have received positive comments from many of these with respect to the changes that have been implemented, comparing their previous experience of the old structure with current arrangements.
Challenges
Not all members of the demonstrating staff have fully accepted the new feedback procedure. I ask them to provide students with a sheet giving marks against the detailed marking scheme and feedback comments on the reverse side. Some staff find this too much effort and have simply refused to use the feedback forms and others have been reluctant or unable to provide marks and feedback within the Faculty-set two week limit. This is an on-going issue that will be addressed with support from the Teaching Director with the intention being to achieve full compliance in the next academic year.
Lab notes are of variable quality and finding time / volunteers to update these is a major challenge.
Equipment is old and creates a poor perception with some students. In fact, the equipment is generally well maintained and is of a vintage that allows students to directly manipulate and control readings in away that might not be possible with modern, computer controlled experiments. However, managing the perception is a real challenge in this case.
Students are performing at a range of levels, with a significant fraction finding it difficult to establish a clear understanding of the different requirements for lab books and lab reports.
Lessons Learnt
I am generally happy that the changes that have been implemented have been largely successful. Going forward we need to refine our guidance on how a lab book should be maintained and how to structure a formal report. I may have to prepare a couple of additional lectures to discuss scientific writing style and provide more examples of good practise. The Department is looking to define a standard approach to keeping a lab book from year 1 onwards so that by the time students enter 3rd year this will be ingrained.
After the class completes, I intend to bring together the demonstrators to get their feedback on how the changes have proceeded and to look at any ideas for further improvement. I expect to have some discussion about feedback mechanisms and the need to ensure uniformity of experience across the lab and adherence to expectations for quality and timing of feedback.
I think the main lesson to take from the experience to date is that students are very quick to react to perceived imbalance and have responded well to efforts to manage that perception. I don’t believe the marks they will receive from this version of the lab would be significantly different from the previous version – they will, however (I hope!), feel that they have received the same opportunity as any of their cohort and will generally be happier about the whole experience.
Scalability
We have 80+ students in the 3rd year lab, with 4 demonstrators operating in the lab on each of the two afternoons the lab takes place. This is a pretty low staff: student ratio (in my opinion) that inevitably limits the level of support that can be offered. Of course, we are trying to increase the student’s level of independence, so there are silver linings of sorts.
I am describing an attempt to manage student expectations and perceptions by putting in place highly visible changes that are specifically designed to give a clear impression of a level playing field. The current system works as we have the right number of staff for dividing the experiments and associated marking workloads evenly. A big rise or fall in student numbers would potentially impact on the symmetry of the current set-up, but this could be addressed (to some extent) by bringing in limited degrees of flexibility in experiment assignment and marking duties. Any significant erosion of the current symmetry would, I believe, result in an increase in student complaints of the previous nature. Scalability is therefore an issue to be considered very carefully.
Suggestions for Transferability
Structuring assignment of assessed materials and assessors to provide marks and feedback to maximise student perception of equal treatment is certainly a consideration that could be brought to the attention of staff in other areas. In many cases it will be difficult to achieve for practical reasons. Whilst I am content that the current changes I have introduced have a had a positive impact on student perceptions, a much greater goal would be to develop a greater sense of trust in the professionalism of staff to provide fair and balanced assessments. This is one reason why it would be very beneficial for all staff to use feedback forms properly and to adhere to the guidance on returning marks and feedback on time.
The experience of marking other people’s work and comparing your marking with those of colleagues is a very revealing process. It may be the case that greater use of carefully managed peer marking would help to foster a better understanding of the marking process and thereby greater faith in the ability of staff to ‘get it right’ more often than not.
Attachments
