All entries for May 2012

Considering Impact

Two things have triggered a series of discussions about impact recently. One of these is working on the evaluation of a series of JISC projects, the other is working on a bid for AHRC and preparing the Pathways to Impact for that.

I first really came across this as an issue when working on the REF for my university. I pulled out of this (I didn’t see why they should have my publications when they weren’t paying me a salary) so only saw the document I had to respond to … an didn’t actually have to respond. I could see the point, the relevance of the research to the wider community is also important, and also saw the panic of colleagues, none of us had much idea what counted as impact or how we could demonstrate it.

I think the idea of impact is a good one to include for research, as long as this doesn’t take precedence over just finding out stuff for its own sake. I can see why the government brought it in, theyre paying for research so they want to see it doing some good, but it is indicative of a move towards a more mundane pragmatic approach to knowledge, education, thought. Like the employability and enterprise agenda in universities. Yes education should have a practical application too, when i got my MA in consultancy i realised that it was the first qualification I’d got that had a practical use since my cycling proficiency. But that’s not the only thing that’s important. We need to continue to do things and learn things purely because they’re interesting. If we don’t instill that love for pure knowledge in our students we’re failing them as much as if we’re sending them out without a skill they can actually earn a living from.

For a funding body I have more sympathy, paying the piper and calling the tune and so on. If you’re ploughing money into something and have to have something to show for it, then describing impact helps to tell the story that you’ve achieved something. the struggle is in what constitutes evidence.

If you’re doing robust science, you need a control. Measuring impact, you can have no control. Yes you can show things have changed, but alone this falls prey to that post hoc ergo propter hoc fallacy. While I’ve been self-employed I’ve lost 14kg ... so is this the impact of changing my job, or cutting out carbs or going to the gym? Hard to say just by measuring a difference. You’d really need to do some research in this universe and simultaneously observe another universe in which the research was not taking place (and keep every other variable the same) to really be able to measure impact. This could be tricky. And if I had that level of omniscience I’d be doing something else anyway.

The other problem is measurement. What metrics do you use? If you can use a mixture of things with numbers attached (students retained, or staff on prozac) and show improvement, then that’s going to keep a lot of people happy. You can measure beneficiaries, students attended, workshops held, but this isn’t really impact, people may attend but how do you know it makes a difference to them, or that they’re even paying attention. There’s also a lot of types of impact that can’t be measured like that, student satisfaction, inclusion. But showing a shift in responses could work here.

I thnk the key thing is to acknowledge all of the above failings with the idea, and do it anyway. Yes the measurements won’t be robust scientific data, yes the may be spurious or fail to take account of some elements. But trying to account for some of these things will give a sense of the degree of change that may have been caused by the research or project and it does focus the mind on what it is really all for.

: 15 May 2012 17:44 |  Tags: Jisc |  Comments (0) |  Close comments |  Report a problem

Random ramblings

Oh another thought … obviously it’s been a slow day. Was reminded of the Pandora myth again (via the jewellery not the planet). I think the message is that if you’ve let War, Death, Famine and Plague out, that having hope left is somehow A Good Thing. but actually it’s totally the opposite. Like death, supposing you live in a place where life expectancy is 40 and you die at 45, you’re happier than if you hope to live to 80 and still die at 45. My cats get really riled if i take out their food, because they hope to be fed, whereas without the noise of their food rattling they’d be OK a bit longer (not that going for a few hours without their Dreamies constitutes famine, but you get my point). Nope that myth is totally screwed. The Four Horseman may lay you out on the floor, but it’s Hope that follows it up with a swift kick to the nuts.

: 13 May 2012 20:18 |  Comments (0) |  Close comments |  Report a problem