woman using a laptop and smiling

Building a Virtuous Loop of Feedback in your course

Feedback. It is a critical part of any course design and it drives a lot of behaviour and activity at the University, but—we argue—it is often a large workload for markers, and this effort is lost on many of our students. 

When talking to students about problems with feedback across many courses, it was clear to us that where we thought the act of feedback was part of a virtuous cycle of learning, students often felt maligned by the feedback, in many cases could not make sense of it, didn’t actually look at it, and many students didn’t even know it existed. 

Our hypothesis was that if we embed students in a more meaningful, coordinated and visible feedback process, aligned with a coherent pedagogy that values learning above grades, then students would (might) begin to see the feedback processes as part of a virtuous loop of learning. 

We found that the tools available to run authentic, collaborative marking and peer review processes were clumsy or simply not available. Frustrated, we began to explore the Database tool in Wattle coupled with a web app called Virtuous Loop to generate meaningful feedback for students and markers.  

The idea behind Virtuous Loop was simple – let’s consider feedback as a data analytics exercise, and aim to approach it with the same level of rigour and transparency we expect in our research, for the improvement and learning for all involved. 

Virtuous Loop has created over 10,000 individual pieces of feedback in a dozen large and small courses since 2017, including peer reviews, project team member contributions [1] , 360-degree feedback with external project hosts [2], topic diagrams of reflection tasks, and plain assignment marking with built-in moderation analytics.  

These tools (and the data structures behind them) allow for enormous flexibility in the way that tasks can be designed, and that moderation of expectations between students, markers and convenors can be managed and represented.  

Take peer review for example. A key part of learning in this process is to develop a student’s capacity to benchmark their own work in relation to other reviewers. Below are graphical representations generated from feedback in a peer review process of Tutorial Facilitations (top left) and Short Essays (top right), and an externally facing Group Project (bottom). 

Fig 1: Styles of Peer Review charts that can accompany text feedback

These graphical representations, along with guided comments from the review groups, form a package of feedback that students receive. One of the key ideas that arise from a “many eyes” representation is that there is no ‘truth’ in any one opinion in the review process – the “many eyes” generate “many truths”, and the actual truth is probably somewhere in between. Learning to value and work with such diversity of opinion is an important goal of higher education that can be difficult to access and develop. 

Using the Virtuous Loop, we have also been able to help align views within teams of markers, with marker analytics reports allowing a data-rich discussion about moderation across teams of markers, considering aspects like distribution of grades at a criterion level, and even the average length of feedback and text readability by each marker. 

Many courses run a reflection activity to guide weekly learning. These reflections are valuable for cementing learning and are in themselves a way for educators to develop a rich understanding of what students are thinking, engaging with, and not understanding. By visualising the topics in the reflections in network diagrams, and giving students back this information at a class and individual level can help students identify their learning in relation to others, and encourage collaboration between students who are of like minds, or struggling with similar concepts. 

Fig 2: Network diagram of a semester’s worth of reflections from a class cohort 

Would you like to participate? We are looking for people who are interested in participating to help expand the use cases and pedagogy around the Virtuous Loop during 2021.  

If you’re interested in exploring new ways of running feedback processes within your course in the upcoming semester or beyond, please contact Chris Brown or Jenny Simmons  

For more information: https://anu365.sharepoint.com/sites/VirtuousLoop 

Also see the April 2021 interview with Chris and Jenny on ANU’s On&Off Campus.

February 2021

Dr Chris Browne is a Senior Lecturer in the ANU College of Science

Jenny Simmons is a Senior Tutor in the ANU College of Engineering and Computer Science,  

[1] https://interact.weblogs.anu.edu.au/2020/07/31/challenge-of-groupwork/  

[2] Browne, C., Johns-Boast, L., Blackmore, K., and Flint, S., 2020, “Capstone Design Projects, the Project Value Map and the Many Eyes Process: Balancing Process and Product to Deliver Measurable Value to Student and Client”, International Journal of Engineering Education vol. 36-2