All 42 entries tagged CAA
September 06, 2007
Writers about assessment make the distinction between formative and summative modes (see Definitions below). It is too often seen as a black-and-white dichotomy. As part of a plea for shades of grey, let me suggest suggest the watched or moderated mode of assessment.
In watched mode, students are allowed a limited or unlimited number of attempts at a given test (or at several variations of it). All their activity is recorded, and they know that their teacher or assessor has access to their results and that the information may be used to form an interim judgement about their commitment, to discuss their progress, and to provide feedback, even though their marks do not influence their progression or their degree.
Other “shades of grey” please.
Definitions of Formative/Summative AssessmentThere are many descriptions of these concepts. Here are two I feel happy with in the context of a degree course:
- Formative assessment is designed to inform development, and to give learners practice at the assessed activity and feedback on their progress; but it does not contribute to the overall assessment.
- Summative assessment contributes to the final outcome of a student’s degree and may include unseen examinations, essays, dissertations or presentations.
April 27, 2007
Warwick’s Elab has published the first draft of Quizbuilder, its elegant tool for writing simple online tests with the minimum of fuss. You might like to try your hand at this short test of 11 multiple-choice questions on elementary number theory, which took me less than an hour to write. The LaTeX equations are a little wobbly in their baselines, but perfectly fit for purpose.
As part of its nearly-£5m investment in Moodle, the free open-source course management system (CMS, aka VLE), the Open University (OU) is currently adding its in-house assessment software OpenMark to Moodle’s assessment capabilities. It will also incorporate some of the authoring strengths of OpenMark into the Moodle Quiz. The full integration may take some time to complete but will mean that OpenMark becomes open source too.
OpenMark’s strengths include the ability to diplay complicated mathematical and symbolic expressions and to provide graduated targeted feedback in response to multiple attempts at variations of the same question. Given the OU’s high production standards and long term funding, this development can only bode well for the future of online assessment, in particular, the assessment of mathematics-based subjects.
April 24, 2007
Three kinds of data need to be kept safe: (i) the questions stored for a test; (ii) student answers entered during a test; (ii) submitted answers and results.
- Keeping tests safe: It is clearly important to keep tests, questions, solutions and feedback safe from prying eyes, especially if they are to be used in summative mode. So the question database should be encrypted or otherwise made hacker-proof. It should also be regularly backed up in case of hardware failure (having lost questions on a hosted service, I would strongly advise authors to back up their work locally too). If a degree of randomness is introduced to reduce the risk of cheating (via multiple question banks or question templates with parameters, say), then thought should be given to the ease with which determined students could circumvent the protection thus provided (see this blog entry, for instance).
- In-test Security: Some assessment software allows submission of answers to be postponed until the end of the test. This is dangerous. A user who has entered 9 out of 10 answers when the system crashes without saving them would have every reason to be angry. My preferred option is to require an entered answer to be validated (and simultaneously saved) before the user is allowed to proceed to the next question (or at least to be warned that they may lose their answer if they do not validate before moving on). Validation allows the software to interpret the answer and return its interpretation to the user in a standard form; it is an important stage in dealing with answers to questions with symbolic content, where the machine may not be able to cope with the informal context-dependent representation humans used to. Another kind of security involves limiting cheating during a test: impersonation, or copying from a neighbour, for example. Invigilation is still the safest answer to this.
- Securing Results: The most important thing about the results database, apart from the obvious needs to be backed up and proof against hacking, it that it should store every bit of activity engaged in by a student during a test. If a student challenges their test outcome, the examiner needs to be able to trace every step the student took, including false validations, inappropriate mouse clicks (some assessment software swoons at the click of a browser back button). and the relaunching of the test. Although it is a good idea to insist that students jot their work down on paper during a test, this is not much help if a system fault requires a new test to be generated and it comes with different values of th random parameters; When parameters are used, the system should also be able to deliver the same test to a student who, through no fault of their own, is forced to make a fresh start. As I have said elsewhere, it is a great help if the database fields are chosen to optimise efficiency and flexibility in searching the results.
March 28, 2007
For 5 hours on Monday, a score of us shared thoughts about online assessment, especially the assessment of mathematics. Here are some of my headline takes on the day:
- Computer-aided assessment of mathematical knowledge and understanding has special needs but offers special rewards
- SToMP, PROMPT, Mathletics, iAssess, OpenMark, STACK, WeBWorK, Maple TA are among the many maths-friendly CAA packages used by or known to people attending the workshop. These tools have many common features but can rarely talk to each other. So much work duplicated but not shared. Does it really have to be like that?
- Question and Test Interoperatibility (QTI) standards to the rescue? Not with the generic Version 1 at least. A better chance with the developing Version 2, which will admit optional extensions users can create to handle special needs, in particular those of mathematics. Will they ever work sufficiently well to justify the limitations they impose and extra attention they require?
There are plans to produce a proper report of the day. Watch this space.
March 19, 2007
Venue: The Mathematics Institute in the Zeeman Building (find us)
10.30 onwards: Coffee in Maths Common Room
11-00 till 1-00: Short presentations and long discussions in B3.02
1-00 till 1-45: Lunch in the Maths Common Room
1-45 till 3-00: Short presentations and long discussions in B3.02
3-00 till 3-30: Summing up and future plans.
3-30 onwards: Tea in the Maths Common Room
March 14, 2007
On Bill Foster’s initative, we are plannng a small informal workshop at Warwick on Monday, 26th March to discuss priorities for computer-aided assessment (CAA) in Higher Education (HE). Bill accepts responsibility for the acronym MADCAP, short for “Mathematics and Computer-Aided Practice Group”. He has had considerable experience using the i-Assess package for large-scale mathematics assessment at the University of Newcastle. We will be joined by colleagues exploring other approaches at Birmingham, Brunel, Portsmouth, Surrey, The Open University, and Warwick.
Here are some topics we hope to talk about:
1. Assessing symbolic material (in particular mathematics) online. Which tools handle this well? How effectively can their functionality be bent to serve our pedagogic needs? Here are some aspects:
- Authoring. Types of input: LaTeX, Asciimath, MathML, plug-ins
- Student Input. Formal or informal syntax, WYSIWYG, symbolic menus/palettes
- Feedback Making intelligent use of student answers. The role of computer algebra systems such as Maple or Maxima
- Question types and Conceptual Understanding. Is CAA only effective at the early stages of mathematical education with large classes and concomitant efficiency gains? Can we go beyond the standard question types to probe deeper understanding in such different areas as Analysis, Algebra and Statistics
- QTI issues. Are these relevant? Do we care? (See below)
2. How can HE institutions influence and gain some control over the development of assessment tools? Are there models of development beyond “buying out of the box”?
- JISC is funding the major development of an assessment tool which meets the Question and Test Interoperability standards (QTI 2.1), but its specification does not accommodate symbolic input. Is this a problem or should we make do with variants of MCQs and numeric input types to satisfy the assessment needs of mathematics or statistics? Alternative products handling symbolic content are available but commercial software usually means some loss control. (One commercial developer will be present at this meeting to outline plans for joint development of assessment tools with HE institutions.)
- How important is it to develop within QTI standards?
- What policies do universities have towards computer-based assessment and how do they influence the choice of tools?
Follow-up workshops are planned at Heriot-Watt and the Open University based upon the outcomes of this meeting.
February 08, 2007
Administering online assessment can be a nightmare—I have lost sleep over it. Although setting the parameters for delivering an exam online will never be entirely straightforward, let me suggest a few desirable features to smooth the way.
• User Accounts: If a single sign-on (SSO) system, such as the open-software system Shibboleth, can be integrated with a CAA package, an assessment can be made instantly accessible to a group of students registered for a module on the institutional database. At the same time, students signed on to the network have immediate access to all the available assessments for the modules they are registered for. In the absence of SSO, assessment software should make it easy for the details of students permitted to access a given assessment to be uploaded manually, for instance accepting comma-separated values from a spreadsheet containing the appropriate fields. An option to give students permission to create their own assessment accounts is also useful; it should allow them to browse the available assessments and register for any that take their fancy.
• Setting Permissions: When creating an assessment, it should be straightforward for the author to set a whole range of permissions: who can see the test, edit the test, take the test, when they can do so, how long it should last, how many attempts are allowed, who can access the results, and so on. It is helpful if these permissions can be set and subsequently edited in an easily-accessible window, which displays the full range of permissions available. It is also handy to be able to save templates of standard sets of permissions for re-use.
• Sending Feedback: It is vital for an author to have detailed control over (i) the levels of feedback: hints, right or wrong, simple answer, full worked solution and (ii) when it is delivered: directly after an answer is submitted, immediately after the test is completed (feedback, like revenge, is a dish best served hot), or later, after the assessment is closed.
• Answer Records: If the assessment software stores users’ answer files – and only that designed for simple self-assessment doesn’t – then it is very important to be able to search those files efficiently. It should be easy to search all the database fields that are used to create assessments and accounts with all the usual functionality available in a respectable database; thus, for example, it should be possible to pull out all the answers to question 5 on assessment 2 done by students called “Smith” who are either based in the Mathematics Dept or whose students numbers begin with 02 (the year of entry). If the database has a field for email addresses, it should be possible to send emails to selected subsets of registered users containing information about, for example, their results and module administration.
• Analysing Results: I have to confess that I have little experience in this area and that my views on what is desirable and useful are poorly developed. I would welcome some input from more experienced readers here. It is obviously helpful to be able to (i) analyse results in as much detail as the database allows and (ii) present the data in easy-to-grasp numerical and visual ways. A number of standard statistical tests can be applied to the data to provide insight into the success of an assessment and the performance of the users; for instance, one helpful test I have used measures the effectiveness of single a multiple-choice question (as part of a larger exam) in discriminating between students of differing ability (as indicated by their overall performance on the exam). Please let me have your views on the best tests to build into the software, by email or via the commenting option below.
January 29, 2007
Another heading in my check-list of criteria for judging whether CAA software, particularly that with mathematical capabilities, is up to the job. As usual, I welcome your comments and further ideas. Today I look at the heading
• Logging in and Submitting: Within a given intranet, so-called “single sign-on” avoids having to distribute special user IDs and passwords to students, who then have to remember them to access an assignment. With single sign-on, it is easier to call on institutional databases to display personal information (name, number, mug-shot) onscreen as an identity check for student and invigilator alike. Once signed on, students should be able to click quickly to the test they want to take. It should be clear how their answers to questions should be submitted for marking (grading), singly and/or in one final submission, and whether multiple attempts are permitted. Answers should be regularly saved to a local drive in case of network or software failure.
• Navigation and Layout: It should be easy to navigate quickly through the questions (in any order), and to choose to display them all together on a scrolling page or one at a time. From any test page it should be clear which questions (i) have already been attempted and (ii) have been finally submitted. Each page layout should be visually easy to interpret (e.g. displayed equations, clear separation of question statement from answer boxes with hypertext for actions close by, adjacent questions with different background colours). Anchors to keep the right part of a long page in view, drop-down menus, and prompts to open help windows can all improve the user experience and sense of being in control.
• Entering Answers: Entering text with standard keyboard characters is usually unproblematic – answer-boxes should accommodate the longest imaginable answer, they should display an easily-readable font, and should have the focus with a flashing cursor when appropriate. Entering non-standard symbols, in particular mathematical expressions, presents a challenge. There are some well-tried ways of dealing with this: informal entry using pocket-calculator conventions, LaTeX markup, a palette of standard symbols that can be dragged into the answer box. CAA software is unforgiving when trying to make sense of the kind of informal entry easily understood by humans, and so rigorous adherence to correct mathematical syntax (brackets, arithmetical operations, functional notation) is usually required. (For instance, WeBWorK is fairly tolerant of informal entry and includes a summary of user syntax in a pane on the right-hand side of its pages.)
• Recording Progress: There are usually several stages in answering a question online: (1) entering the answer in the appropriate box(es); (2) validating the answer to check that the program correctly interprets it (especially if symbolic expressions are involved); (3) saving the answer (often combined with validation); (4) reviewing the answer and editing it; (5) submitting the answer for marking/grading; (6) making further attempts if allowed; (7) submitting the final attempt. It is important for this progress data to be displayed in a table on every page of the assessment, with direct navigation to uncompleted questions. It is also helpful to record individual question and total scores in this table and to display ”time remaining” in say minutes and an analogue clock.
• Training and In-Test Help: It is desirable to give students a practice assignment in conjunction with an online tutorial to familiarise them with the assessment format and the syntax for entering symbolic notation. This can be delivered in advance of the test or as an initial part of it. A summary of this user guidance should be easily accessible at any stage of the test, perhaps through a help-box or in a separate pane of the test window.
• Accessibility: Here is a short checklist of desirable feature for optimising access to web pages: (1) user control of font styles and sizes (especially important for the display of mathematics, which may be embedded as graphics); (2) text equivalents for graphics and multimedia; (3) simple and logical navigation; (4) control over text and background colour. (5) Compatibility with a screen-reader that handles mathematics and other symbolic notation (programs now exist to read mathematics that is coded in MathML – e.g. Design Science’s MathPlayer: see http://www.dessci.com/en/products/mathplayer/tech/accessibility.htm). Entering mathematical answers is particularly difficult for visually-impaired users and so an intelligent screen-reader for validation of answers would provide helpful reassurance.
January 24, 2007
This is the next contribution to my list of criteria for judging whether CAA software, particularly that with mathematical capabilities, is up to the job. Today I look at the heading
• Question types: MCQs, MRQs, yes/no, hot-spot, drag-and-drop, and so on—the more the merrier! For the assessment of deeper mathematical knowledge, more searching questions can be set when the assessment package can call on the services of a computer algebra system (CAS) – eg Maple TA and STACK. An option for multiple-part questions is valuable, especially if (i) partial credit is available and (ii) answers to later parts can be judged correct when calculations based on incorrect answers to earlier parts are correctly performed.
• Marking/Grading, Scoring: It is important for the author (i) to have complete control over the marking system for each question, (ii) to be able to give the user full information about how each question will be scored, and (iii) to have the option of revealing scores to the user’s at specified stages. Default marking schemes may be useful but should be easy to over-ride and should allow an author to specify a different marking scheme for each question. If an answer involves mathematical expressions, the software should be able to parse equivalent answers.
• Feedback: I believe this to be the most important pedagogical feature of CAA software! The author should be able to provide various types of feedback to each question (e.g. (1) whether the submitted answer was right or wrong, (2) the bare marks scored, (3) the correct answer—for instance, the correct MCQ choice, numerical entry, or symbolic expression, (4) the full worked solution) and to specify the point at which the feedback is made available (e.g. upon submission of a single answer, of a completed assessment, or at some later time). If questions contain variable parameters, the feedback should be tailored to the parameter values used. Another useful feature is an option to provide one or more graded hints after a wrong answer and to adjust the marks accordingly. An advanced feature, explored in Mathletics, is to be able to use a student’s answer to guess at errors or misconceptions (malrules) and to respond to them in the feedback.
• Random features: The inclusion of varying degrees of randomness in the construction of individual questions and whole assignments/tests/exams can significantly enhance the educational value of CAA and simultaneously reduce the risks of cheating. For each question at the assignment level, the software should be capable of selecting randomly from a specific bank of questions which all test the same skill/knowledge/understanding. At the question level, there is considerable scope for randomised variation, using place-holders to vary such things as units, names, even subject contexts; and in mathematical subjects, using parameters within specified ranges of numerical values that require students to carry out different calculations, each testing essentially the same knowledge or skills. Considerable care is needed to ensure the questions make sense for all choices of variables (for instance, avoiding division by zero), but in a science discipline, it is possible to generate millions of different, but educationally-equivalent, questions. This makes copying answers pointless and allows students to have virtually unlimited practice in formative mode. When sufficient randomness is built into a question template, it becomes a “reusable learning object ”, a special case of a reusable learning object (RLO) beloved of educational theorists who study computer-mediated learning.