Feedback. It is a critical part of any course design and it drives a lot of behaviour and activity at the University, butâwe argueâit is often a large workload for markers, and this effort is lost on many of our students.
When talking to students about problems with feedback across many courses, it was clear to us that where we thought the act of feedback was part of a virtuous cycle of learning, students often felt maligned by the feedback, in many cases could not make sense of it, didnât actually look at it, and many students didnât even know it existed.
Our hypothesis was that if we embed students in a more meaningful, coordinated and visible feedback process, aligned with a coherent pedagogy that values learning above grades, then students would (might) begin to see the feedback processes as part of a virtuous loop of learning.
We found that the tools available to run authentic, collaborative marking and peer review processes were clumsy or simply not available. Frustrated, we began to explore the Database tool in Wattle coupled with a web app called Virtuous Loop to generate meaningful feedback for students and markers.
The idea behind Virtuous Loop was simple – letâs consider feedback as a data analytics exercise, and aim to approach it with the same level of rigour and transparency we expect in our research, for the improvement and learning for all involved.
Virtuous Loop has created over 10,000 individual pieces of feedback in a dozen large and small courses since 2017, including peer reviews, project team member contributions [1] , 360-degree feedback with external project hosts [2], topic diagrams of reflection tasks, and plain assignment marking with built-in moderation analytics.
These tools (and the data structures behind them) allow for enormous flexibility in the way that tasks can be designed, and that moderation of expectations between students, markers and convenors can be managed and represented.
Take peer review for example. A key part of learning in this process is to develop a studentâs capacity to benchmark their own work in relation to other reviewers. Below are graphical representations generated from feedback in a peer review process of Tutorial Facilitations (top left) and Short Essays (top right), and an externally facing Group Project (bottom).
These graphical representations, along with guided comments from the review groups, form a package of feedback that students receive. One of the key ideas that arise from a âmany eyesâ representation is that there is no âtruthâ in any one opinion in the review process – the âmany eyesâ generate âmany truthsâ, and the actual truth is probably somewhere in between. Learning to value and work with such diversity of opinion is an important goal of higher education that can be difficult to access and develop.
Using the Virtuous Loop, we have also been able to help align views within teams of markers, with marker analytics reports allowing a data-rich discussion about moderation across teams of markers, considering aspects like distribution of grades at a criterion level, and even the average length of feedback and text readability by each marker.
Many courses run a reflection activity to guide weekly learning. These reflections are valuable for cementing learning and are in themselves a way for educators to develop a rich understanding of what students are thinking, engaging with, and not understanding. By visualising the topics in the reflections in network diagrams, and giving students back this information at a class and individual level can help students identify their learning in relation to others, and encourage collaboration between students who are of like minds, or struggling with similar concepts.
Would you like to participate? We are looking for people who are interested in participating to help expand the use cases and pedagogy around the Virtuous Loop during 2021.
If youâre interested in exploring new ways of running feedback processes within your course in the upcoming semester or beyond, please contact Chris Brown or Jenny Simmons
For more information: https://anu365.sharepoint.com/sites/VirtuousLoop
Also see the April 2021 interview with Chris and Jenny on ANU’s On&Off Campus.
February 2021
Dr Chris Browne is a Senior Lecturer in the ANU College of Science,
Jenny Simmons is a Senior Tutor in the ANU College of Engineering and Computer Science,
[1] https://interact.weblogs.anu.edu.au/2020/07/31/challenge-of-groupwork/ [2] Browne, C., Johns-Boast, L., Blackmore, K., and Flint, S., 2020, âCapstone Design Projects, the Project Value Map and the Many Eyes Process: Balancing Process and Product to Deliver Measurable Value to Student and Clientâ, International Journal of Engineering Education vol. 36-2
