Scholarship of Teaching and Learning (SoTL): Considering and measuring impact

Scholarship of Teaching and Learning (SoTL): Considering and measuring impact

By Dr Kerry Dobbins (Assistant Professor, Academic Development Centre) and Dr Isabel Fischer (Reader in Information Systems at Warwick Business School)

Questions about how to evaluate and measure impact of Scholarship of Teaching and Learning activities (SoTL) have always generated much lively discussion and debate within the SoTL community. For example, a recent LTHE Tweetchat focused on the topic of ‘Breaking boundaries: career progression and education-focused roles’. At the heart of the discussion was the nature and purpose of SoTL in relation to career advancement. Two specific questions discussed were ‘what forms and outputs can SoTL take?’ and ‘how should quality and impact be measured?’ In recent times, these questions have become more important than ever for all colleagues on education or teaching-focused contracts due to their connection to institutional promotions criteria. Routes to promotion for these colleagues will often involve criteria including:

• Making a significant contribution to pedagogy.
• Gaining recognition for quality and impact of scholarship.
• Demonstrating educational impact and influence within and beyond the University.
• Producing scholarly outcomes that advance learning, teaching and assessment.

Outputs and demonstrable impact of SoTL underpin these criteria and highlight why the recent Tweetchat was such a topical discussion. Considering the various forms or outputs that SoTL can take connects to how impact from those outputs may be measured or viewed to have made a significant contribution. The Twitter community identified a wide range of forms that SoTL outputs can take. These included:

• Conference/webinar presentations
• Publications
• Podcasts, blogs, vlogs
• Infographics
• Content for repositories, e.g. the National Teaching Repository
• Books
• Invited talks
• Staff/student development training
• Well-designed courses/modules

There was much leaning towards open access forms and critical scrutiny occurring through the teaching community, rather than tying outputs largely to journal publications and scrutiny through the standard academic peer review process. This leaning is still in line with the SoTL principles of sharing findings for communal use and review, but importantly the Twitter community placed considerable value on practitioner-focused scrutiny.

There was also recognition that different outputs serve different purposes and that any output can have an impact. This takes us into the realms of questioning what ‘impact’ means in SoTL and how it might be measured. Again, the Twitter community identified various ways that ‘impact’ of SoTL could be considered and/or measured:

• Policy changes – national, institutional, departmental, etc.
• Creation of staff development provision.
• Explicit recognition and reward of teaching through, for example, promotion and recruitment criteria.
• Improved module/course evaluations and student grades.
• Enhanced student engagement.
• Paper citations, usage metrics.
• Engagement in networks or communities of practice.
• Improved resourcing for teaching/development.
• Changes in colleagues’ practices.
• Engagement in collaborative activities.

It is interesting to see from the responses the different levels at which impact may play out (e.g. individual, institutional, etc). Of course, one of the ultimate aims of SoTL is to understand how students learn effectively so that learning experiences can be enhanced. However, SoTL is also recognised as an essential component of academic professional development and a mechanism through which HE teaching as a profession is advanced. As the responses above show, this wide remit of SoTL means that impacts from its activities can take many forms.

It is also apparent that some of the impact examples offered above may take longer to realise than others and be harder to measure in a quantifiable sense. A key question emerging in the Twitter discussion was whether SoTL impact needs to be measurable. This again takes us back to the question of what ‘impact’ means in SoTL and more specifically, what type of impact is being sought.

Different stakeholders (e.g., individuals, community groups, institutions) may of course be seeking different types of impact from SoTL activities but in reality, the needs of the different stakeholders are not so easily separated. There continues to be a challenging balancing act being played out in SoTL between it being an activity to develop individual and community practices, and it becoming a significant feature of aspects like promotions criteria. Whilst inclusion in promotions criteria demonstrates institutional recognition and reward of SoTL, it adds a layer of trying to measure or capture a narrative of identifiable and evidence-based impact that resonates with a panel who may or may not be familiar with the various lenses and dynamics of SoTL.

What all of this leads to is the need to keep engaging in institutional conversations about SoTL so that purposes, outputs and impacts of this activity are not viewed in too reductive terms and the various lenses and levels identified by the Twitter community are not overlooked. This is the vast flavour of SoTL and if we are too reductive, we may only taste elements of the impacts that it could have.

Two further questions should also be asked in the context of these institutional discussions:

1. How to align expectations about output and impact with time allocated for SoTL, e.g. what should be the output for somebody who receives a workload allocation of, say, two to three hours a week versus somebody who receives one to two full days per week?
2. How might expectations about output and impact be aligned on an inter-disciplinary instructional-level to encourage transferability?

It is important to ensure that discussions to define impact more clearly (and broadly) at an institutional level do not deter us from seeing the sharing of SoTL work and findings as an activity worthy in and of itself. At Warwick, we aim to create and encourage opportunities for the sharing of SoTL activities as part of our internal communities of practice. This way inspirational SoTL findings can be spread and impact upon colleagues’ practices, independent of views about measurability.

Abigail Ball : 10 Jan 2022 09:43 |  Tags: Isabel Fischer Kerry Dobbins Learning Scholarship Teaching |  Comments (0) |  Report a problem