As the 2020 Academic Year turns to face the final stretch, assessment will once again become a feature of remote teaching and learning experiences. In July, following a national survey conducted by the Australasian Council on Open Distance and eLearning (ACODE), we learned that ANU was one of 24 Australasian universities (N=47) that implemented a remotely invigilated examination process.
The Proctorio package was one of 11 software solutions and represented 7% (N=2) of the product range. Of the 23 universities that did not use a proctored solution 35% (N=8) augmented Turnitin to resolve their examination needs. Thirty (of the 47) institutions indicated they would be rerunning the same solution in coming semesters. Along with almost 80% of other universities ANU indicated it will review its future examination requirements as we look to future years.
Having made that commitment, the Centre for Learning & Teaching (CLT) in partnership with Planning and Performance Management (PPM), went about building upon knowledge gained through the Student and Teacher surveys of Wellbeing and Remote Learning (SWiRL & TWiRL) conducted in Semester 1, 2020.
In September 2020 the Evaluations Team within CLT, in collaboration with PPM, produced a report titled “Student Experiences of Proctorio”, which analysed a survey of almost 1500 students who had used the invigilation software. During the development of this report, it became clear that to provide a comprehensive evaluation of Proctorio, it would be beneficial to speak with teachers who had used the software, or who were considering using it. In combination, the two reports now provide insights on experiences of learning and Teaching with Proctorio; informing the continued context of remote invigilation at ANU.
Regarding the survey of students, a striking feature was the diversity of experiences students faced in remotely invigilated examinations. There was little evidence that Proctorio consistently predicated a poor examination experience for a student. Likewise, there was little evidence that non-proctored examinations predicated a positive or negative experience. Deeper analysis showed the more influential factors were associated with assessment design and communication practices – these two factors returned the best indication of what worked, and what needed improvement for future iterations of Proctorio.
It was also noted that students were more willing to engage constructively with Proctorio than initially anticipated with very few respondents making general criticisms. This contrasted with much of the debate occurring in Semester 1, 2020 where heightened distrust of remote invigilation caused concerns among students and teachers.
Regarding the teacher survey, titled “Teaching with Proctorio”, this was drawn from interviews with ten academics who had conducted (or contemplated conducting) Proctorio examinations. While small in number, the interviews allowed for analysis to take a simple thematic approach where threads were woven from a close reading of notes taken by the CLT interviewers. The thematic-narrative reflects strengths, weaknesses, and suggestions for remote examination with invigilation software. As revealed, Proctorio’s useful teaching functions include its ability to establish an invigilated context that, from a university-wide perspective, addresses academic misconduct (e.g. mitigates cheating).
At this early stage, the interviewed academics suggested they were tackling a range of issues that include software and network reliability, fit for purpose test types, proctoring transparency, monitoring issues, Wattle integration options, documentation processes, special considerations and student perceptions. With the future in mind, teaching staff would recommend a continued focus on practice examinations, backup submissions, sharing of good practice, and a watchful eye on alternative solutions.
At the October 2020 meeting of the Teaching and Learning Development Committee (TLDC) it was confirmed that Proctorio will remain in place to the end of Semester 1, 2021. As of Semester 2, 2021 the context may well change but ‘remote assessment’ in some form or other is likely to remain an option that needs to be available – on call at least. As such, CLT will continue to monitor the market and scan the scope of opportunities ANU has to improve the online experience.
Linked article: ANU Lunch Vox Testing Times: Exploring Assessment
Tim Grace is the Manager of the Education Communities and Environments (ECE) team – one of the three teams within the ANU Centre for Learning & Teaching (CLT).