How to evaluate a VLE

I was recently asked if I could recommend a clear-cut evaluation of the use of VLEs in higher education.

The short answer is no. The long answer is maybe.

The research has been poor concerning VLEs in HE. However, I remembered a paper from ALT-J that used a new methodology to evaluate a custom built VLE at Edinburgh. It tentatively established that where there exists a cohesive and strong 'community of practice', and a VLE is used that fits well with the community and its practices, then it can help students with becoming effective operators within that community. So for example, a VLE that fits with the professional practices of medical education helps students to become practitioners. Which is obvious really. But the more significant message is that where communities of practice are less well defined, or more varied and numerous (as is the case in humanities), a VLE is less likely to be a simple fit, and less likely to help students with becoming practitioners (or multi-disciplinary operators). Which is also obvious!

My own research into designerly thinking and practice in humanities education takes a similar approach, but with an emphasis upon much deeper differences between epistemic-agentic assemblages (more than communities of practice, more like paradigms or Foucault's epistemes). I take a radically pluralistic stance on epistemic diversity. But this paper is still very interesting and really worth reading. It's available free from the new open access ALT site.

Ellaway, Dewhurst, Mcleod, "Evaluating a virtual learning environment in the context of its community of practice", Alt-J, Research in Learning Technology, Volume 12, No. 2, June 2004

Here are a few first impressions...

The authors report on one of the first attempts at evaluating a virtual learning environment in use, considering its impact on a community of practice (learners, teachers and others). In surveying previous research, they conclude that:

"...because VLEs can be used in many different ways, and because much that was implicit in the traditional learning environment becomes explicit in its online equivalent, the evaluation of VLEs has proved to be a particularly complex problem. Furthermore, because of the sheer scale, complexity and cost of VLEs, their adoption and use is increasingly undertaken at an institutional level and any subsequent evaluation, if it is not done at the level of the individual learner, is most often also undertaken at this institutional level." P.126

They argue that:

"Although there has been much published on evaluative work on VLEs, this has until recently rarely gone further than analysing their various features and functions...In presuming that a VLE has intrinsic properties, that the context into which a VLE will be deployed is neutral and that any given VLE will automatically deliver predictable benefits (or otherwise) into that context, the predictive approach is significantly limited in providing a useful perspective of a VLE in a grounded course context. It is important to note that most of these approaches have been directed towards a novit-iate audience looking for the best evidence or advice available to help them select a suitable system to meet their needs." P.126

The authors take an alternative approach, more suited to higher education: Wenger's community of practice model, in which the aim of learning is to induct learners fully into a cohesive and fully functional "community of practice", in which they may eventually become equals (whether that be a professional community, or an academic community of researchers). A complex analysis is used to evaluate the fit between a VLE and the Edinburgh University medical education programme for which it was developed. Unsurprisingly, it is a good fit and contributes to the goal of becoming part of the community of practice (although this does not indicate that the students will be well placed to join the wider community of practice of the medical profession). In this case, the VLE was developed specifically for a large, cohesive, rigid, pre-existing community of practice.

However, such well-formed communities might be the exception in HE.

"It is important to emphasise that this is a theory-based approach, which is predicated on a pre-existing course community of practice. In those situations where this is a valid assumption, for instance in subjects such as medicine, then it has immediate relevance and utility. For other situations, for instance in modular programmes of study, where communities of practice may not equate to a course (or even exist coherently at all) then there may be less relevance in such a study, although a module may in some cases retain a degree of internal coherence as a community of practice." P.142

Humanities disciplines, for example, are much more fragmentory and individualistic. There are significant differences between departments, and even between modules within departments, and sometimes between different tutor groups within modules. For example, in the English Department, creative writing is practised in a significantly different way to The European Novel. One could successfully argue that the value for the students lies in the opportunity to partake in radically different communities of practice.

Robert O'Toole : 23 Jan 2012 22:33 |  Tags: Inspiration |  Comments (0) |  Close comments |  Report a problem

Upad on iPad – a commonplace book for the digital age

Some of my friends in the English Department are keen on reviving the practice known as "commonplacing". It's an approach to studying, creating and living that uses a single "common place book" into which experiences of all kinds are recorded, and allowed to transversally interact. A kind of scrap book for containing the divers range of interesting and unusual experiences that might be encountered by the early-modern scholar. Importantly, its more than just a book, its an approach or attitude to being in the world, and a catalyst for generating ideas and experiments in a more free-form fashion. Wikipedia has a good short history, going back to the early-modern period, and disappearing in the early 20th century.

So why, if its so good, aren't we all busy commonplacing all the time? There are probably many reasons why the practice became less popular. But perhaps a better question would be: what would it take to get an early-21st century student commonplacing? What might the practice be competing against? Personally (speaking as a grad student), i've always been discouraged by my own inability to make engaging, aesthetically satisfying, and legible scribbles. Perhaps its the case that digital technology & visual design, makes us all feel a bit inferior. Certainly my old moleskin notebook (used sporadically) seems to lack colour and expressiveness. The other big challenge with which paper-based technologies cannot cope is the multi-mediality of experience today - we are constantly engaged in a rich world of analogue and digital channels. I sit at home reading the news on my iPad, listening to the radio, and talking with people all at the same time. I have access to and use a huge range of digital texts in many forms and from many sources. Its both more intense and more complex. And to add a further level of complexity, sometimes I do this in collaboration with other people, even other people that I've never actually met.

But I have started commonplacing again. A simple technology has made the difference by addressing these problems. And I'm really profiting from it. All of a sudden my personal workflow has changed, with the commonplace at its core, capturing and developing my stream of experience, inspirations and ideation. I'm using the Upad app on my iPad. It's a beautifully designed user interface for scribbling using a variety of pens and colours (I draw onto the iPad using a Just Mobile stylus). Most importantly, its easy to combine neatly hand written text (through a magnified text entry box) with smart typed notes (in a wide variety of fonts and colours), and images. So I can, for example, take a screenshot of a page from a book in the Kindle app, add it to a note book page, annotate it with arrows and hand written text, add a photo from the internet or from the iPad camera, and add neatly typed notes.

Here's a screenshot of some of my notes in Upad:

You can see some typical pages in the lightbox. One of them is a handwritten "writing experiment", the next is a page from a Kindle book, and finally a complicated diagram combining hand drawn and typed elements. In fact, that diagram is a representation of my research workflow (shown in full below). At the centre of the workflow is Upad. Sometimes I create notes that seem particularly important and relevant. I use the email option in Upad to send them to an online Evernote notebook (for example, I have notebook for collecting workflow diagrams, and another for reading notes). The Evernote notes are then replicated onto my laptop and desktop computers, so that I can use them in my writing. For example, this blog entry uses images drawn in Upad, sent to Evernote and then accessed on my MacBook AIR. Storing notes in Evernote has the further advantage of making them searchable - even when they are handwritten (it has a good recognition engine built into its search engine). I can also send pages from my commonplace book to Twitter (or Facebook if you use it).

Here's the workflow diagram in full:

Robert O'Toole : 07 Jan 2012 07:55 |  Tags: Implementation |  Comments (0) |  Close comments |  Report a problem

Everything as PDF?

Follow-up to 3 essential elements for an e–learning strategy from Inspires Learning - Robert O'Toole

My new found enthusiasm for the PDF format is a consequence of using iAnnotate on the iPad. Here's an example illustrating why I think every event should be accompanied by at least an overview PDF (at least until the tools to create a PDF from any format on the iPad become more effective). Last week I attended a conference at DeMontfort University on the Democratic Learning Conversation (great conference).

I received the agenda as a Word file, imported it into iAnnotate which converted it into PDF. I was then able to add notes to the agenda during the sessions. However, I quickly discovered another very effective trick. During some of the sessions I was inspired to create Keynote slides (the Apple equivalent of Powerpoint) and a Mindjet mind map developing some of my own ideas in response to the speakers. I was able to take image snapshots of these and embed them at the appropriate point in the PDF. So I left the conference with a single PDF document containing rich annotations.

That's good. But the longer term effect is particularly impressive. I lead quite a busy life. Lots of meetings, lots to remember and learn. Annotating the PDF helped me to remember the details of the conference (just in the act of writing the notes down I can remember them better) and to keep alive the thinking that it inspired. If a student were to ask me for a recommendation as to a learning technology that could make a big difference to their capabilities, this would be it: PDF + iAnnotate + iPad.

Here is a snapshot of the PDF that I annotated. You can see a thumbnail of one of the Keynote slides. The slide can be viewed as full screen in the PDF file. Audio annotations may also be added. So for example I could have interviewed another attendee, or perhaps even recorded a whole session.

One additional feature would be useful - to be able to add extra space in the margins of the PDF, or an additional blank page to contain my notes. CORRECTION - ADDING BLANK PAGES IS POSSIBLE IN iAnnotate

Robert O'Toole : 23 Dec 2011 14:13 |  Tags: Ideation |  Comments (0) |  Close comments |  Report a problem