All entries for Monday 23 January 2012

January 23, 2012

How to evaluate a VLE

I was recently asked if I could recommend a clear-cut evaluation of the use of VLEs in higher education.

The short answer is no. The long answer is maybe.

The research has been poor concerning VLEs in HE. However, I remembered a paper from ALT-J that used a new methodology to evaluate a custom built VLE at Edinburgh. It tentatively established that where there exists a cohesive and strong 'community of practice', and a VLE is used that fits well with the community and its practices, then it can help students with becoming effective operators within that community. So for example, a VLE that fits with the professional practices of medical education helps students to become practitioners. Which is obvious really. But the more significant message is that where communities of practice are less well defined, or more varied and numerous (as is the case in humanities), a VLE is less likely to be a simple fit, and less likely to help students with becoming practitioners (or multi-disciplinary operators). Which is also obvious!

My own research into designerly thinking and practice in humanities education takes a similar approach, but with an emphasis upon much deeper differences between epistemic-agentic assemblages (more than communities of practice, more like paradigms or Foucault's epistemes). I take a radically pluralistic stance on epistemic diversity. But this paper is still very interesting and really worth reading. It's available free from the new open access ALT site.

Ellaway, Dewhurst, Mcleod, "Evaluating a virtual learning environment in the context of its community of practice", Alt-J, Research in Learning Technology, Volume 12, No. 2, June 2004

Here are a few first impressions...

The authors report on one of the first attempts at evaluating a virtual learning environment in use, considering its impact on a community of practice (learners, teachers and others). In surveying previous research, they conclude that:

"...because VLEs can be used in many different ways, and because much that was implicit in the traditional learning environment becomes explicit in its online equivalent, the evaluation of VLEs has proved to be a particularly complex problem. Furthermore, because of the sheer scale, complexity and cost of VLEs, their adoption and use is increasingly undertaken at an institutional level and any subsequent evaluation, if it is not done at the level of the individual learner, is most often also undertaken at this institutional level." P.126

They argue that:

"Although there has been much published on evaluative work on VLEs, this has until recently rarely gone further than analysing their various features and functions...In presuming that a VLE has intrinsic properties, that the context into which a VLE will be deployed is neutral and that any given VLE will automatically deliver predictable benefits (or otherwise) into that context, the predictive approach is significantly limited in providing a useful perspective of a VLE in a grounded course context. It is important to note that most of these approaches have been directed towards a novit-iate audience looking for the best evidence or advice available to help them select a suitable system to meet their needs." P.126

The authors take an alternative approach, more suited to higher education: Wenger's community of practice model, in which the aim of learning is to induct learners fully into a cohesive and fully functional "community of practice", in which they may eventually become equals (whether that be a professional community, or an academic community of researchers). A complex analysis is used to evaluate the fit between a VLE and the Edinburgh University medical education programme for which it was developed. Unsurprisingly, it is a good fit and contributes to the goal of becoming part of the community of practice (although this does not indicate that the students will be well placed to join the wider community of practice of the medical profession). In this case, the VLE was developed specifically for a large, cohesive, rigid, pre-existing community of practice.

However, such well-formed communities might be the exception in HE.

"It is important to emphasise that this is a theory-based approach, which is predicated on a pre-existing course community of practice. In those situations where this is a valid assumption, for instance in subjects such as medicine, then it has immediate relevance and utility. For other situations, for instance in modular programmes of study, where communities of practice may not equate to a course (or even exist coherently at all) then there may be less relevance in such a study, although a module may in some cases retain a degree of internal coherence as a community of practice." P.142

Humanities disciplines, for example, are much more fragmentory and individualistic. There are significant differences between departments, and even between modules within departments, and sometimes between different tutor groups within modules. For example, in the English Department, creative writing is practised in a significantly different way to The European Novel. One could successfully argue that the value for the students lies in the opportunity to partake in radically different communities of practice.