All entries for January 2006

Sexpot

Follow-up to Fruit Flies Like An Apple … from Computer-aided assessment for sciences

Michael McCabe at the University of Portsmouth is giving some rigorous objective thought to marking schemes for objective tests in his SEXPOT Project (Scoring EXemplars and Principles of Objective Testing). Have a look at his very preliminary findings.

: 30 Jan 2006 16:33 |  Tags: CAA Marking Objective Tests Pedagogy |  Comments (0) |  Close comments |  Report a problem

Fruit Flies Like An Apple …

… time flies like an arrow, which brings me neatly on to the theory of social choice functions (better known to most of us as 'voting systems'). Arrow's Theorem states, in a nutshell, that the only voting system that satisfies three very plausible 'fairness criteria' (for example, if more voters prefer X to Y, then X should appear above Y in the final list) is a dictatorship, where one person gets to decide for everyone else.

Could there be an analogue of Arrow's Theorem for marking objective tests? Is there a marking system that is fair to all? To answer this, one first needs a list of fairness criteria. Any suggestions for these? I'll start the ball rolling with

Criterion 1: If student X 'knows more' than student Y, then X should score more than Y. Of course, the examiner has to specify what is meant by 'knows more'.

: 30 Jan 2006 12:00 |  Tags: CAA Marking Objective Tests Pedagogy |  Comments (4) |  Close comments |  Report a problem

I Wanna Hold You In My Hand

Writing about web page http://www.mathcentre.ac.uk/resources_for_category.php?f=1&c=32

1. I held an iPod loaded with 80 hours of Mathtutor video tutorials yesterday (and still with room for 10,000 so-called 'songs'). A joy to (be)hold — I could read the hand-written equations even without my glasses.

2. Apropos John Dale's blog on Brain Training, I can imagine the Nintendo DS being adapted to wonderfully imaginative maths puzzles and quizzes.

Now marry the two, and you have the perfect hand-held tutor, ready to accompany you anywhere.

: 27 Jan 2006 13:27 |  Tags: CAA Hand Held Toys Maths Online |  Comments (1) |  Close comments |  Report a problem

Unresponsive Multiple–Response

I have been exploring two very different assessment programs recently and was drawn to compare the way they each handle Multiple-Response Questions (MRQs). To put things in context, consider the following naive example of such a question:

Decide whether the following arithmetical statements are true or false:

(For non-arithmeticians, this fourth part is the only true statement.)

The two packages impose their own different marking schemes and I am happy with neither. Here are their inflexible offerings:

Package A

This software is a simple quiz builder, very easy to learn and quick to author. (If you have the questions ready, you could put together a 10-question quiz in 15 minutes, even first time round.)

You answer an MRQ like the one above by checking all the buttons of the statements you think are true and leaving unchecked those you think are false — the buttons toggle on and off like conventional check boxes. Full marks are given if and only if every part is answered corrrectly (with true statements checked and false ones unchecked); otherwise zero is given.

I feel that this all-or-nothing approach is too severe; a student gettting three parts out of four right surely deserves some reward.

Package B

In contrast to the previous package, this one is a behemoth, powerful but hard to tame. (Incidentally, I notice that, unlike the "alleluias" and "slaves" that eBay claimed on Google to be auctioning before Christmas, today it doesn't appear to have any behemoths for sale.) Package B's multiple response offering is part of its MCQ environment — you move from MC to MR by simply ticking the box "allow more than one correct answer". Below each MRQ, a hyperlink partial grading explained appears in red; when clicked, the following message pops up in a new window:

What does formula mean? Are "correct choices" the same as "correct answers"? If not, then perhaps "# correct answers" means "# of true statements"? And what if the grade is negative? It's not clear. (Incidentally, it would save us all a lot of time if questions really "could calculate their grades".)

Imagine for simplicity that

• the above MRQ is the sole question on a test
• a desperate student in a rush launches the test and immediately presses the "submit" (or "grade") button, neither reading the question nor checking any buttons.

If we assume all the buttons start unchecked by default, the desperado gets 3 parts correct and one part wrong, and by the most likely interpretation of the formula, scores two-thirds out of a maximum one point; in other words, 66% !! That's certainly 'owt for nowt' and a fat reward for opportunism — hardly a desirable outcome.

What I would like

In neither package is the author given any choice about the format or a marking scheme for a multiple-response question. You must take it or leave it. But just in case the developers are listening, here is the kind of flexibility I would like to see as standard for MRQs:

1. A drop-down menu in a combo box next to each part of the question with the three options: 'true', 'false', 'no attempt', and with all the boxes initially set to 'no attempt'.
2. The ability to set the marks awarded for (i) a correct answer, (ii) an incorrect answer and (iii) no attempt, in each part of each question (or at least in each question).
3. An option to display this information to the examinee next to each part of each question.

Other Scenarios?

I'd like to hear from other people about their preferred MRQ frameworks.

: 23 Jan 2006 15:25 |  Tags: CAA Pedagogy Software |  Comments (2) |  Close comments |  Report a problem

Where ignorance is bliss, 'Tis folly …

… not to encourage students to acknowledge it. There is a lot to be said for creating a climate where students feel comfortable admitting they don't know the answer.

I have just been trying to get to grips with Maplesoft's relatively new assessment software Maple TA. The default format for multiple-choice questions (MCQs) does not seem to offer an option "No attempt" or "Don't know" . The software allows you to include as many parts in an MCQ as you like, but only one correct answer. So if one of your parts is "No attempt", a student choosing it is marked wrong and awarded zero.

My preferred marking scheme for MCQs is something like: 3 marks for a correct choice, 1 mark for choosing "Don't know", and -1 mark for a wrong choice; negative totals are raised to zero, so that students' scores fall in the range 0 — 30 in a test with 10 MCQs. Under this regime, checking boxes at random can yield a positive expected total score, depending on the number of parts to each question. (Of course, you can normalise the scores by subtracting this expected score from the totals, but this seems petty.) A student who answers "Don't know" to all questions ends up with 10 and some would argue this is unfairly generous. But I believe there is real educational merit in rewarding to those who are willing to confront their ignorance; they have a better chance of doing something about it.

: 16 Jan 2006 10:45 |  Tags: CAA Mcqs Software |  Comments (8) |  Close comments |  Report a problem

Volcanoes and Monkey Puzzles

This clue, which turned up recently in The Week, appealed to me:

Mount Vesuvius strikes, engulfing Naples, taking just seconds (4)

It seems to have appeared originally in a Listener crossword by BeRo which also included the novelty clue "H,I,J,K,L,M,N,O (5)" mentioned below; except BeRo noticed that adding "and P" to the clue provided a definition too! The puzzle was punningly called Pseudo-Clue because its grid includes a 9 X 9 Sudoku when the letters other than H — P are removed. (Who was BeRo?)

Just in case you haven't seen it before, here is one of the flashes of brilliance Araucaria (the sobriquet used by John Graham) is famous for:

Poetical scene has surprisingly chaste Lord Archer vegetating (3, 3, 8, 12)

The poet was Rupert Brooke.

: 09 Jan 2006 10:38 |  Tags: Bagatelle Cryptic Crossword Clues |  Comments (0) |  Close comments |  Report a problem

I meet the Computer Scientists

The Department of Computer Science's Teaching Committee gave up 10 minutes of their precious time in the cause of computer-aided asssessment (CAA) this afternoon. I came to their meeting with gratitude for their support (including a generous donation to the Project budget) and left with two tangible contributions to my wish list:

1. Access to their postgraduate students (I plan to hire a small team — 2 or 3 from within the Science Faculty — to learn to use the CAA software, to help create the assessments, and to train others to do so).


2. The possibility of going live with formative CAA on one of the Department's first-year modules — real students doing tests and getting feedback online the autumn.

My next encounter will be with the Department of Statistics.

: 06 Jan 2006 16:13 |  Tags: CAA Science Departments |  Comments (0) |  Close comments |  Report a problem

Ads Build Up Value

This blog's title is an anagram of David Paul Ausubel, the educational psychologist who made serious play of the fact that you can't teach someone effectively until you know what they already know (and, by implicaton, don't know). In Educational Psychology: A Cognitive View (1968) he wrote: "The most important single factor influencing learning is what the learner knows. Ascertain this and teach accordingly". His dictum is often quoted but rarely acted upon, and I see no harm in running another advertisement to build up its value.

When students submit to any kind of assessment, they reveal information about what they know and don't know. This is often valuable information and usually it goes to waste (think of all those exams where the only feedback is a mystery number between 1 and 100). Does computer-aided asssesment (CAA) offer any remedies? Can it find out "what the learner knows" and then act accordingly? I believe the answer is "yes" and will try to convince you of this with a couple of simple examples.

If a factual(ish) question (eg Which philosopher might have said "Blogo ergo sum"?) is wrongly answered ("Renée Zellweger" perhaps), the same question can be repeated in a later test, with a friendly admonition in the feedback for a second wrong answer. Persistent weaknesses over a series of formative tests can be routinely reported back to the student.

For conceptual questions, CAA can do even better. We will take a simple example to illustrate how computer assessment can identify a student's problem and try to deal with it. Consider the arithmetic question:

Add one third to one half and express your answer as a fraction (i.e. a number of the form m/n).

There are many reasons for getting the wrong answer: a simple error of calculation (unlikely here though), or a failure to understand

(i) the nature of fractions (both as numbers and processes — does the symbol 1/2 mean the number 'one half' or the process of dividing by 2?) and

(ii) the rules for calculating with them.

Let's suppose a student submits the following wrong answer:

It's a fair guess that they have used the following wrong rule (mal-rule) of adding the numerators and the denominators:

We could reinforce this guess with another example, say two-thirds plus a fifth. If they come up with three-eighths, we can be pretty certain we've sussed out the mal-rule they're using.

Next we start to generate doubt by getting them to apply their rule to one half plus one half (giving two quarters, which one hopes they will know equals one half). Something funny going on here! A fluke exception? How about this then:

Having thoroughly undermined their faith in the mal-rule, we eventually return to the drawing board, asking the student how many sixths make a half (three) and how many sixths makes a third (two). If the penny drops that three of a kind plus two of a kind is five of a kind, the following equations take on meaning

More examples will be needed to elicit a thorough understanding of the general rule for adding fractions (including unlearning the mal-rule) and yet more practice for the student to feel comfortable applying the rule as a fast and accurate reflex.

What I have just described is only one possible CAA response to just one of many misconceptions or mal-rules that may be revealed in the simple exercise of adding a half to a third. A student may be guessing or may have mislearnt the rule at an earlier stage of education. An effective face-to-face (human) tutor will patiently probe a student's mistakes and identify the root causes their failures, then painstakingly rebuild their knowledge and understanding on a sound foundation. Intelligent computer assessment can be programmed to do the same and it is nothing if not patient and painstaking.

: 06 Jan 2006 11:54 |  Tags: CAA Feedback Pedagogy |  Comments (0) |  Close comments |  Report a problem