All 5 entries tagged Software

CAA Twenty Years On

Writing about web page http://mathstore.ac.uk/articles/maths-caa-series/jan2006/

Cliff Beevers' article IT was twenty years ago today … covers two decades of work on computer-aided assessment for mathematical disciplines at Heriot-Watt University. The challenges of testing mathematics online are presented as an intelligent and ever-changing compromise between effective pedagogy and the technical limitations of the medium.

The CAA package they have created since 1985 is currently offering formative testing to 25,000 students in Scottish schools. As a registered blind teacher, Cliff is well placed to address the accessibility issues; the Heriot-Watt development team took the mathematical tests to youngsters at the Royal Blind School in Edinburgh last year.

He ends with the following advice:

• Keep the questions fresh through randomisation


• Seek answers using mathematical expressions wherever possible


• Provide optional steps for those that need them


• Report all of this to both student and the teacher alike

• Keep all of these new assessments accessible so that disabled students are not disadvantaged


• Extend where possible to include animations, simulations and explorations so that higher order skills can also be tested electronically

: 10 Feb 2006 11:49 |  Tags: CAA Pedagogy Software |  Comments (0) |  Close comments |  Report a problem

I got it wrong!

Follow-up to Unresponsive Multiple–Response from Computer-aided assessment for sciences

Christopher Munro, who is working with Michael McCabe on the SEXPOT Project, has kindly pointed out that my interpretation of the following "partial grading" formula (for multiple-response questions in software package B)

was wrong. The reality is even more bizarre.

To explore the implications of this formula, let's lay down some ground rules for multiple-response questions (MRQs) and look at a concrete example.

Rule 1: The statements that make up the choices in are either true or false (although Martin Greenhow told me recently that he is exploring a question type for Mathletics that also allows the answer 'undecidable').

Rule 2: For each part of the MRQ, it is equally difficult to decide whether the statement is true or false (unlike competitions in popular magazines, which often make it humorously obvious which statements are false). I usually aim to satisfy this rule in my MRQs, although I freely acknowledge it's not an exact science and anyway, what is 'hard' varies from student to student.

Rule 3: It is equally likely that a statement is true or false (so that, on average for a four-part question, one question in sixteen has all parts false).

In the above formula the "# of correct answers" actually means the number of true statements in the question.

For our concrete example, take a four-part MRQ and a model student who gets all four choices correct. If, say, just one of the four statements is true, the student scores (4–0)/1 = 4 marks. On the other hand, if 3 statements are true and one is false, the student only gets (4–0)/3 = 4/3 marks for being just as clever as before. Maybe I am missing something, but that strikes me as a daft scoring system.

And for the one question in sixteen where all four answers are false, the student scores

Luckily here, the software under discussion violates Rule 3 by insisting that at least one statement is chosen to be true!

: 06 Feb 2006 09:59 |  Tags: CAA Mrqs Pedagogy Software |  Comments (0) |  Close comments |  Report a problem

Unresponsive Multiple–Response

I have been exploring two very different assessment programs recently and was drawn to compare the way they each handle Multiple-Response Questions (MRQs). To put things in context, consider the following naive example of such a question:

Decide whether the following arithmetical statements are true or false:

(For non-arithmeticians, this fourth part is the only true statement.)

The two packages impose their own different marking schemes and I am happy with neither. Here are their inflexible offerings:

Package A

This software is a simple quiz builder, very easy to learn and quick to author. (If you have the questions ready, you could put together a 10-question quiz in 15 minutes, even first time round.)

You answer an MRQ like the one above by checking all the buttons of the statements you think are true and leaving unchecked those you think are false — the buttons toggle on and off like conventional check boxes. Full marks are given if and only if every part is answered corrrectly (with true statements checked and false ones unchecked); otherwise zero is given.

I feel that this all-or-nothing approach is too severe; a student gettting three parts out of four right surely deserves some reward.

Package B

In contrast to the previous package, this one is a behemoth, powerful but hard to tame. (Incidentally, I notice that, unlike the "alleluias" and "slaves" that eBay claimed on Google to be auctioning before Christmas, today it doesn't appear to have any behemoths for sale.) Package B's multiple response offering is part of its MCQ environment — you move from MC to MR by simply ticking the box "allow more than one correct answer". Below each MRQ, a hyperlink partial grading explained appears in red; when clicked, the following message pops up in a new window:

What does formula mean? Are "correct choices" the same as "correct answers"? If not, then perhaps "# correct answers" means "# of true statements"? And what if the grade is negative? It's not clear. (Incidentally, it would save us all a lot of time if questions really "could calculate their grades".)

Imagine for simplicity that

• the above MRQ is the sole question on a test
• a desperate student in a rush launches the test and immediately presses the "submit" (or "grade") button, neither reading the question nor checking any buttons.

If we assume all the buttons start unchecked by default, the desperado gets 3 parts correct and one part wrong, and by the most likely interpretation of the formula, scores two-thirds out of a maximum one point; in other words, 66% !! That's certainly 'owt for nowt' and a fat reward for opportunism — hardly a desirable outcome.

What I would like

In neither package is the author given any choice about the format or a marking scheme for a multiple-response question. You must take it or leave it. But just in case the developers are listening, here is the kind of flexibility I would like to see as standard for MRQs:

1. A drop-down menu in a combo box next to each part of the question with the three options: 'true', 'false', 'no attempt', and with all the boxes initially set to 'no attempt'.
2. The ability to set the marks awarded for (i) a correct answer, (ii) an incorrect answer and (iii) no attempt, in each part of each question (or at least in each question).
3. An option to display this information to the examinee next to each part of each question.

Other Scenarios?

I'd like to hear from other people about their preferred MRQ frameworks.

: 23 Jan 2006 15:25 |  Tags: CAA Pedagogy Software |  Comments (2) |  Close comments |  Report a problem

Where ignorance is bliss, 'Tis folly …

… not to encourage students to acknowledge it. There is a lot to be said for creating a climate where students feel comfortable admitting they don't know the answer.

I have just been trying to get to grips with Maplesoft's relatively new assessment software Maple TA. The default format for multiple-choice questions (MCQs) does not seem to offer an option "No attempt" or "Don't know" . The software allows you to include as many parts in an MCQ as you like, but only one correct answer. So if one of your parts is "No attempt", a student choosing it is marked wrong and awarded zero.

My preferred marking scheme for MCQs is something like: 3 marks for a correct choice, 1 mark for choosing "Don't know", and -1 mark for a wrong choice; negative totals are raised to zero, so that students' scores fall in the range 0 — 30 in a test with 10 MCQs. Under this regime, checking boxes at random can yield a positive expected total score, depending on the number of parts to each question. (Of course, you can normalise the scores by subtracting this expected score from the totals, but this seems petty.) A student who answers "Don't know" to all questions ends up with 10 and some would argue this is unfairly generous. But I believe there is real educational merit in rewarding to those who are willing to confront their ignorance; they have a better chance of doing something about it.

: 16 Jan 2006 10:45 |  Tags: CAA Mcqs Software |  Comments (8) |  Close comments |  Report a problem

No Comment!

"No comment!", the standby of the defensive celebrity, is a nightmare for the serious blogger. Buried none too deep in our bloggers' psyches is the desire to be noticed. Comments are our life-blood — being ignored equates to rejection, failure. Where Scott Adams finds nothing unusual in 500 comments per blog, we are grateful for a meagre one or two.

I am therefore proposing a new service for bloggers called Commentary, initially offered in three packages:

Economy Class: Comments from other blogs with the same tags are randomly recycled as comments to your blog. This service is free and much superior to the undignified alternative of bribing or browbeating your friends and relatives into commenting on your blogs. (Maximum 10 comments.)

Frequent Blogger: You are offered a personal crew of aliases who regularly augment your blogs with pithy comments generated from a database of apposite phrases based on the words in your blog. A suitable program already exists. (Maximum 50 comments.)

All the World's a Blog: With this highly customizable service, you get to orchestrate a witty and lively debate in comments around your blog, with optional insults from egregious bigots, enthusiastic agreement and warm approval from like-minded observers of the human condition, and occasional interjections from famous names, such as Mickey Mouse, Tony Blair, Atilla the Hun, Bridget Jones, Renee Day "I-blog-therefore-I-am" Kart, and so on. (Unlimited comments).

The premium option takes blogging into another realm, and one over which you have complete control. When you are next told to "get a life", this could be your answer.

All that's missing is the appropriate software, so once again, E-Lab, it's over to you.

(Don't hesitate to comment.)

: 22 Nov 2005 16:53 |  Tags: Blogging Software |  Comments (1) |  Close comments |  Report a problem