January 23, 2007

Fit for Purpose?

Software for computer-aided assessment comes in many shapes and sizes serving many purposes, ranging from simple quiz-building to the construction of complex question templates involving random parameters that are designed to test deeper understanding and provide intelligent feedback.

It is evaluation time for the software we have be trying out in the Science Faculty at Warwick. Because my project is specifically aimed at science disciplines, we have concentrated the trials on four CAA packages with serious mathematical capabilities: Maple TA, Mathletics, STACK and WeBWorK.

In order to judge the merits of these behemoths, it is important to lay down the criteria we will use. I have therefore started to produce a list of features and qualities that might be considered desirable in CAA software of this kind. PLEASE ADD TO MY LIST OR SUGGEST AMENDMENTS.

I have set out the features and qualities under the following headings:

  • Authoring
  • Pedagogy
  • User-experience
  • Administration
  • Data Security
  • Robustness

I will deal with each heading in separate blogs for ease of digestion. Today I start with:

Authoring

Ease of use (Ability to author questions in browser window, intelligent fully-functional editor (see Work-flow below), quick access to current projects, good GUI and navigation, natural syntax for writing questions, flexible file and folder structure for organising work, automatic save before closing browser, easy user account creation, spreadsheet import and export of both account and assignment data, optimised for accessibililty, re-usable user-created templates for (i) writing tests (ii) sets of properties and permissions.)
Mathematics entry and handling (WYSIWYG maths editor for symbolic and mathematical expressions. GIF-free options – MathML, (La)TeX, or WebEQ with MathPlayer. Platform-independent visually-pleasing rendering of symbols with scalable fonts and colours. Tex quality for both rendering and range of symbols. Intelligent display of mathematical objects (e.g. polynomials).)
Sharing questions and assignments/tests (Import/export of (i) questions created in same software and (ii) text from other applications. Compatibility with QTI and other interoperability standards. Control of permissions for other users,)
Creating assignments/tests (Easy selection from question banks. Easy control of assignment delivery options (ability to permute questions, permute parts of MCQs, choose “single scrollable page” or “one question per page”. Full control over length of test, period of availability, user-access, feedback timing.)
Work flow (WYSIWYG editor with (i) full features (e.g. find and replace) and (ii) instant rendering of modified entry. Cut and paste in all question fields (including mathematical expressions). Regular automatic-saving option. Control over time out. One-click question try-out.)
Testing (Ability to try out questions and feedback exactly as it would be experienced by a user. Separate windows for question testing and editing. Debugging and comprehensive error-reporting.)
Question, assignment and user tagging (Ability to create a number of database fields (e.g. level, topic, subtopic, creation date) for quick search and retrieval of questions from large banks. Likewise for retrieval of users from performance database.)


- 4 comments by 1 or more people Not publicly viewable

  1. Sharon Curtis

    I suggest an additional criterion: accessibility.

    23 Jan 2007, 18:00

  2. Paul Topping

    FYI, MathPlayer works with screen readers used by the blind to read the math in a web page. We are also working on a version of MathPlayer that will do the same for a PDF.

    Paul Topping
    Design Science, Inc.
    http://www.dessci.com/mathplayer

    23 Jan 2007, 20:56

  3. Thank you Sharon. Accessibility is scheduled to appear again under the heading “User Experience” in a later blog. I am pleased to learn about MathPlayer’s compatibility with screen readers, Paul. I wasn’t aware of that.

    24 Jan 2007, 09:46

  4. Juliette White

    We found that the ability to share questions between tests in such a way that if e.g. mistakes were found only one master copy would need to be updated would have been incredibly useful as we wanted to use the same questions for several different groups of students studying different subjects in different years.

    08 Feb 2007, 12:13


Add a comment

You are not allowed to comment on this entry as it has restricted commenting permissions.

Trackbacks

  1. CAA Fitness for Purpose: Pedagogy

    This is the next contribution to my list of criteria for judging whether CAA software, particularly that with mathematical capabilities, is up to the job. Today I look at the heading Pedagogy Question types: MCQs, MRQs, yes/no, hot-spot, drag-and-drop, and s…

    Computer-aided assessment for sciences - 24 Jan 2007, 12:53

  2. CAA Fitness for Purpose: User Experience

    Another heading in my check-list of criteria for judging whether CAA software, particularly that with mathematical capabilities, is up to the job. As usual, I welcome your comments and further ideas. Today I look at the heading User Experience Logging in and…

    Computer-aided assessment for sciences - 29 Jan 2007, 12:32

  3. CAA Fitness for Purpose: Administration

    Administering online assessment can be a nightmareI have lost sleep over it. Although setting the parameters for delivering an exam online will never be entirely straightforward, let me suggest a few desirable features to smooth the way. Administration Us…

    Computer-aided assessment for sciences - 08 Feb 2007, 11:13

  4. CAA Fitness for Purpose: Data Security and Robustness

    Data Security Three kinds of data need to be kept safe: (i) the questions stored for a test; (ii) student answers entered during a test; (ii) submitted answers and results. Keeping tests safe: It is clearly important to keep tests, questions, solutions and …

    Computer-aided assessment for sciences - 24 Apr 2007, 13:25

January 2007

Mo Tu We Th Fr Sa Su
Dec |  Today  | Feb
1 2 3 4 5 6 7
8 9 10 11 12 13 14
15 16 17 18 19 20 21
22 23 24 25 26 27 28
29 30 31            

Search this blog

Tags

Galleries

Most recent comments

  • The LaTeX equations are a little wobbly in their baselines That's because they use mimeTeX. I am try… by Steve Mayer on this entry
  • Aha! You are right, John. I am familiar with a different terminology: Multiple Choice for what Quizb… by on this entry
  • Haven't you used the wrong question type, Trevor? Your answers look, to my unskilled eye, to be mutu… by John Dale on this entry
  • The usual factors in Information Assurance are: Confidentiality Integrity Availability And for syste… by Max Hammond on this entry
  • Is the workshop next Monday,26 March 07, open to anybody interested in computer aided assessment? If… by Kuldeep Singh on this entry

Blog archive

Loading…
Not signed in
Sign in

Powered by BlogBuilder
© MMXIV