All 2 entries tagged Formative

Using AI for Formative Feedback: Current Challenges, Reflections, and Future Investigation

By Matthew Voice, Applied Linguistics at the University of Warwick

One strand of the WIHEA’s working group for AI in education has focused on the role of AI in formative feedback. As part of this strand, I have been experimenting with feeding my own writing to a range of generative AI (ChatGPT, Google Bard, and Microsoft Bing), to learn more about the sorts of feedback they provide.

The accompanying presentation documents my observations during this process. Some issues, such as the propensity of AI to ‘hallucinate’ sources, are well-documented concerns with current models. As discourse on student use of AI begins to make its way into the classroom, these challenges might provide a basis for critical discussion around the accuracy and quality of the feedback produced by language models, and the need for student to review any outputs produced by LLMs.

Other common issues present different challenges for students using LLMs to elicit formative feedback. For instance, the prompt protocol in the presentation revealed a tendency for AI to provide contradictory advice when its suggestions are queried, leading to a confusing stance on whether or not an issue raised actually constitutes a point for improvement within the source text. When tasked with rewriting prompt material for improvement, LLMs consistently misconstrued (and therefore left absent) some of the nuances of my original review, in a fashion which changed key elements of the original argumentation without acknowledgement. The potential challenges for student users which arise from these tendencies is discussed in more detail in the presentation’s notes.

In addition to giving some indication of the potential role of LLMs in formative feedback, this task has also prompted me to reflect on the way I approach and understand generative AI as an educator. Going forward, I want to suggest two points of reflection for future tasks used to generate and model LLM output in pedagogical contexts. Firstly: is the task a reasonable one? Using LLMs ethically requires using my own writing as a basis for prompt material, but my choice to use published work means that the text in question had already been re-drafted and edited to a publishable standard. What improvements were the LLMs supposed to find, at this point? In future, I would be interested to try eliciting LLM feedback on work in progress as a point of comparison.

Secondly, is the task realistic, i.e. does it accurately reflect the way students use and engage with AI independently? The review in my presentation, for example, presupposes that the process of prompting an LLM for improvements to pre-written text is comparable to student use of these programmes. But how accurate is this assumption? In the Department of Applied Linguistics, our in-progress Univoice project sees student researchers interviewing their peers about their academic process. Data from this project might provide clearer insight into the ways students employ AI in their learning and writing, providing a stronger basis for future critical investigation of the strengths and limitations in AI’s capacity as a tool for feedback.

This is blog 14 in our diverse assessment series, the two most recent previous blogs can be found here:

• Assessments: Capturing Lived Experience and Shaping the Future
• Building knowledge on the pedagogy of using generative AI in the classroom and in assessments

Abigail Ball : 26 Jun 2023 09:00 |  Tags: Assessment Feedback Formative Matthew Voice |  Comments (0) |  Close comments |  Report a problem

How can assessment encourage & motivate learners to succeed, academically & socially? – Jade

Summative and formative assessments can both enable students to succeed, academically and socially. Summative assessments are “cumulative assessments…that intend to capture what a student has learned, or the quality of learning, and judge performance against some standards” National Research Council, 2001). This method of assessment is often used to evaluate student knowledge and understanding at the end of a topic (Gardner, 2010), and is often graded. Consequently, summative assessments may encourage and reassure students who have performed well, or motivate those who have under-achieved to try harder. In my practice, I summatively assess end-of-topic tests, and keep an accurate, up-to-date mark book with comparisons to target grades. This enables student performance to be monitored, which may lead to conversations with students or parents, or interventions, if students are underachieving.

Formative assessment refers to “frequent, interactive assessments of student progress and understanding to identify learning needs and adjust teaching appropriately” (OECD, 2005, p.21). Since formative assessments can inform teachers of student comprehension, this can, in turn, inform planning. For example, recently, after beginning teaching the topic of electrolysis to a year 10 class, through whole-class questioning, I realised that they knew very little about ionic bonding. Therefore, I planned and devoted a full lesson to teaching these principles, before continuing with electrolysis.

Importantly, formative assessments enable students to consider feedback, and improve upon their work and understanding, rather than purely focussing on their grade (Education Endowment Foundation, 2016, p.5). Consequently, I rarely write students’ grades or scores on their classwork or homework, but instead offer positive comments, effort grades and several action points. Students then complete these actions during lessons in a different-coloured pen, for easy comparison to the original work. In my opinion, this method is extremely effective; students have to correct mistakes (scientific or literacy), give more detailed, well-written answers, or complete challenge questions, which encourage higher-order thinking. This also allows for students to reflect on the progress they have made, which may motivate students to continue to learn through making improvements.

In terms of the type of written feedback, rather than offering vague remarks such as “great job” or “nearly there”, I try to give more constructive, specific comments. This type of feedback has been shown to focus students’ attention on certain aspects that require improvement (Hattie and Timperley, 2007). For verbal feedback, I try to give lots of praise (including merits) when students answer questions, even if their answers are wrong. As Mueller and Dweck (1998) argue, it is more important to reward effort than intelligence, and socially, I believe that this makes students feel safe and builds their confidence, meaning that they are more likely to volunteer answers in the future. To further help students succeed socially, I set peer assessment tasks, in which students are encouraged to give each other positive comments, in addition to suggestions for improvement.

Furthermore, I assess students’ learning through a range of a plenary activities. I have found that competition encourages learners to succeed, for both academic and social reasons, and thus, I try to implement a variety of games into my lessons, such as bingo, splat, noughts-and-crosses and team quizzes.

References

Education Endowment Foundation, 2016. A marked improvement? A review on the evidence of written marking. London: Education Endowment Foundation.

Gardner, J., 2010. Developing teacher assessments: An introduction. In: J. Gardner, W. Harlen, L. Hayward, G. Stobart and M. Montgomery, eds. 2010. Developing teacher assessment. New York: Open University Press. pp.1−11.

Hattie, J. and Timperley, H., 2007. The power of feedback. Review of Educational Research, 77(1), pp.81−112.

Mueller, C. M. and Dweck, C. S., 1998. Praise for intelligence can undermine children’s motivation and performance. Journal of Personality and Social Psychology, 75(1), pp. 33−52.

National Research Council, 2001. Assessment in the classroom. [online] National Academies Press. Available at: https: https://www.nap.edu/read/9847/chapter/5 [Accessed 8 February 2018].

Office of Economic Co-operation and Development, 2005. Formative assessment: Improving learning in secondary classrooms. Paris: OECD Publishing.

Abigail Ball : 05 Nov 2018 10:48 |  Tags: Assessment Formative Summative |  Comments (0) |  Close comments |  Report a problem