January 23, 2006

Unresponsive Multiple–Response

I have been exploring two very different assessment programs recently and was drawn to compare the way they each handle Multiple-Response Questions (MRQs). To put things in context, consider the following naive example of such a question:

Decide whether the following arithmetical statements are true or false:

\circ\ \ 2+2=5\\ 
\circ\ \ 2+2=22\\ 
\circ\ \ 2+2=8-3\\
\circ\ \ 2+2=0\ (\text{mod}\ 4)

(For non-arithmeticians, this fourth part is the only true statement.)

The two packages impose their own different marking schemes and I am happy with neither. Here are their inflexible offerings:

Package A

This software is a simple quiz builder, very easy to learn and quick to author. (If you have the questions ready, you could put together a 10-question quiz in 15 minutes, even first time round.)

You answer an MRQ like the one above by checking all the buttons of the statements you think are true and leaving unchecked those you think are false — the buttons toggle on and off like conventional check boxes. Full marks are given if and only if every part is answered corrrectly (with true statements checked and false ones unchecked); otherwise zero is given.

I feel that this all-or-nothing approach is too severe; a student gettting three parts out of four right surely deserves some reward.

Package B

In contrast to the previous package, this one is a behemoth, powerful but hard to tame. (Incidentally, I notice that, unlike the "alleluias" and "slaves" that eBay claimed on Google to be auctioning before Christmas, today it doesn't appear to have any behemoths for sale.) Package B's multiple response offering is part of its MCQ environment — you move from MC to MR by simply ticking the box "allow more than one correct answer". Below each MRQ, a hyperlink partial grading explained appears in red; when clicked, the following message pops up in a new window:

What does formula mean? Are "correct choices" the same as "correct answers"? If not, then perhaps "# correct answers" means "# of true statements"? And what if the grade is negative? It's not clear. (Incidentally, it would save us all a lot of time if questions really "could calculate their grades".)

Imagine for simplicity that

  • the above MRQ is the sole question on a test
  • a desperate student in a rush launches the test and immediately presses the "submit" (or "grade") button, neither reading the question nor checking any buttons.

If we assume all the buttons start unchecked by default, the desperado gets 3 parts correct and one part wrong, and by the most likely interpretation of the formula, scores two-thirds out of a maximum one point; in other words, 66% !! That's certainly 'owt for nowt' and a fat reward for opportunism — hardly a desirable outcome.

What I would like

In neither package is the author given any choice about the format or a marking scheme for a multiple-response question. You must take it or leave it. But just in case the developers are listening, here is the kind of flexibility I would like to see as standard for MRQs:

  1. A drop-down menu in a combo box next to each part of the question with the three options: 'true', 'false', 'no attempt', and with all the boxes initially set to 'no attempt'.
  2. The ability to set the marks awarded for (i) a correct answer, (ii) an incorrect answer and (iii) no attempt, in each part of each question (or at least in each question).
  3. An option to display this information to the examinee next to each part of each question.

Other Scenarios?

I'd like to hear from other people about their preferred MRQ frameworks.

- 2 comments by 0 or more people Not publicly viewable

  1. Chris Munro

    I am the project student working on SEXPOT.
    As for the formula:

    #correct choices means the number of correctly chosen responses by the candidate e.g. if option 1 is true and the candidate says it is true that is one correct choice. If the candidate said option one was false when it is correct that would be one incorrect choice (#incorrect choices)

    #correct answers, is the number of choices (or options in the question) which are correct, e.g. in a multiple choice question this would be one.


    01 Feb 2006, 13:36

  2. Alexander McCabe

    Hi Trevor,
    I've been meaning to get back to you for a while about this post. I hope you'll forgive the delay.

    Firstly, I'm inclined to agree with you that the 'All or Nothing' nature of the multiple response questions in Quiz Edition is too severe. In fact I'm coming round to the view that this question type is of practically no use in quizzes at all. I don't quizzes with this kind of question and I don't find them fair!

    So why bother including it at all? Most of the reason is because it's in every other quiz program so it's a must for future compatibility/interoperability and is one of the interactions defined in SCORM. Also, I think quiz authors expect to see it there, even if they never use it for anything. (An exception to this is in surveys, where it is useful for a range of different questions.)

    I think the maths question type you've got is properly regarded as four different questions, or maybe could be represented by a 'matrix' style question in some systems (a 4×3 matrix). Given the requirement for negative marking, it would be a very custom question type if it were available as a template style question in any software package. However, there is hope, you could certainly construct it in Question Writer (Publisher Edition). It would take a few hours, but once constructed, it could easily be copied across and between tests.

    To enter the question on a simpler authoring package, the package would first need to have a multiple choice / multiple response question that allowed for negative marking / partial credit. Then you'd need to break it down into four questions, and ask for a response for each statement. I think that's a better plan than to regard each mathematical statement as a separate option – as each would likely be represented by an image anyway and even if you did have a dream package that allowed all the other things, you might find that you couldn't enter images as options.

    04 Feb 2006, 22:55

Add a comment

You are not allowed to comment on this entry as it has restricted commenting permissions.

January 2006

Mo Tu We Th Fr Sa Su
Dec |  Today  | Feb
2 3 4 5 6 7 8
9 10 11 12 13 14 15
16 17 18 19 20 21 22
23 24 25 26 27 28 29
30 31               

Search this blog



Most recent comments

  • The LaTeX equations are a little wobbly in their baselines That's because they use mimeTeX. I am try… by Steve Mayer on this entry
  • Aha! You are right, John. I am familiar with a different terminology: Multiple Choice for what Quizb… by on this entry
  • Haven't you used the wrong question type, Trevor? Your answers look, to my unskilled eye, to be mutu… by John Dale on this entry
  • The usual factors in Information Assurance are: Confidentiality Integrity Availability And for syste… by Max Hammond on this entry
  • Is the workshop next Monday,26 March 07, open to anybody interested in computer aided assessment? If… by Kuldeep Singh on this entry

Blog archive

Not signed in
Sign in

Powered by BlogBuilder