February 06, 2006

I got it wrong!

Follow-up to Unresponsive Multiple–Response from Computer-aided assessment for sciences

Christopher Munro, who is working with Michael McCabe on the SEXPOT Project, has kindly pointed out that my interpretation of the following "partial grading" formula (for multiple-response questions in software package B)
Grade Formula
was wrong. The reality is even more bizarre.

To explore the implications of this formula, let's lay down some ground rules for multiple-response questions (MRQs) and look at a concrete example.

Rule 1: The statements that make up the choices in are either true or false (although Martin Greenhow told me recently that he is exploring a question type for Mathletics that also allows the answer 'undecidable').

Rule 2: For each part of the MRQ, it is equally difficult to decide whether the statement is true or false (unlike competitions in popular magazines, which often make it humorously obvious which statements are false). I usually aim to satisfy this rule in my MRQs, although I freely acknowledge it's not an exact science and anyway, what is 'hard' varies from student to student.

Rule 3: It is equally likely that a statement is true or false (so that, on average for a four-part question, one question in sixteen has all parts false).

In the above formula the "# of correct answers" actually means the number of true statements in the question.

For our concrete example, take a four-part MRQ and a model student who gets all four choices correct. If, say, just one of the four statements is true, the student scores (4–0)/1 = 4 marks. On the other hand, if 3 statements are true and one is false, the student only gets (4–0)/3 = 4/3 marks for being just as clever as before. Maybe I am missing something, but that strikes me as a daft scoring system.

And for the one question in sixteen where all four answers are false, the student scores


Luckily here, the software under discussion violates Rule 3 by insisting that at least one statement is chosen to be true!

- No comments Not publicly viewable

Add a comment

You are not allowed to comment on this entry as it has restricted commenting permissions.

February 2006

Mo Tu We Th Fr Sa Su
Jan |  Today  | Mar
      1 2 3 4 5
6 7 8 9 10 11 12
13 14 15 16 17 18 19
20 21 22 23 24 25 26
27 28               

Search this blog



Most recent comments

  • The LaTeX equations are a little wobbly in their baselines That's because they use mimeTeX. I am try… by Steve Mayer on this entry
  • Aha! You are right, John. I am familiar with a different terminology: Multiple Choice for what Quizb… by on this entry
  • Haven't you used the wrong question type, Trevor? Your answers look, to my unskilled eye, to be mutu… by John Dale on this entry
  • The usual factors in Information Assurance are: Confidentiality Integrity Availability And for syste… by Max Hammond on this entry
  • Is the workshop next Monday,26 March 07, open to anybody interested in computer aided assessment? If… by Kuldeep Singh on this entry

Blog archive

Not signed in
Sign in

Powered by BlogBuilder