February 08, 2007

CAA Fitness for Purpose: Administration

Follow-up to Fit for Purpose? from Computer-aided assessment for sciences

Administering online assessment can be a nightmare—I have lost sleep over it. Although setting the parameters for delivering an exam online will never be entirely straightforward, let me suggest a few desirable features to smooth the way.


User Accounts: If a single sign-on (SSO) system, such as the open-software system Shibboleth, can be integrated with a CAA package, an assessment can be made instantly accessible to a group of students registered for a module on the institutional database. At the same time, students signed on to the network have immediate access to all the available assessments for the modules they are registered for. In the absence of SSO, assessment software should make it easy for the details of students permitted to access a given assessment to be uploaded manually, for instance accepting comma-separated values from a spreadsheet containing the appropriate fields. An option to give students permission to create their own assessment accounts is also useful; it should allow them to browse the available assessments and register for any that take their fancy.

Setting Permissions: When creating an assessment, it should be straightforward for the author to set a whole range of permissions: who can see the test, edit the test, take the test, when they can do so, how long it should last, how many attempts are allowed, who can access the results, and so on. It is helpful if these permissions can be set and subsequently edited in an easily-accessible window, which displays the full range of permissions available. It is also handy to be able to save templates of standard sets of permissions for re-use.

Sending Feedback: It is vital for an author to have detailed control over (i) the levels of feedback: hints, right or wrong, simple answer, full worked solution and (ii) when it is delivered: directly after an answer is submitted, immediately after the test is completed (feedback, like revenge, is a dish best served hot), or later, after the assessment is closed.

Answer Records: If the assessment software stores users’ answer files – and only that designed for simple self-assessment doesn’t – then it is very important to be able to search those files efficiently. It should be easy to search all the database fields that are used to create assessments and accounts with all the usual functionality available in a respectable database; thus, for example, it should be possible to pull out all the answers to question 5 on assessment 2 done by students called “Smith” who are either based in the Mathematics Dept or whose students numbers begin with 02 (the year of entry). If the database has a field for email addresses, it should be possible to send emails to selected subsets of registered users containing information about, for example, their results and module administration.

Analysing Results: I have to confess that I have little experience in this area and that my views on what is desirable and useful are poorly developed. I would welcome some input from more experienced readers here. It is obviously helpful to be able to (i) analyse results in as much detail as the database allows and (ii) present the data in easy-to-grasp numerical and visual ways. A number of standard statistical tests can be applied to the data to provide insight into the success of an assessment and the performance of the users; for instance, one helpful test I have used measures the effectiveness of single a multiple-choice question (as part of a larger exam) in discriminating between students of differing ability (as indicated by their overall performance on the exam). Please let me have your views on the best tests to build into the software, by email or via the commenting option below.

- 3 comments by 1 or more people Not publicly viewable

  1. Juliette White

    If you don’t have single sign on, I’d recommend having a way to create accounts for students easily and being able to block out the option of students doing this themselves, even though this sounds like more work. We had a nightmare of a time with students creating duplicate accounts, putting their first and last names the opposite way round from the official records, using different first names from the official lists etc.

    We’d have also found it useful to be able to put students doing the same tests into groups and see the results by group, particularly if you’re using the same test over successive years with different cohorts. We did quite a bit of ‘mastery’ testing, and I’d have liked a way to seen a list of students who had passed the test in one of their attempts and those who hadn’t.

    08 Feb 2007, 12:18

  2. Max Hammond

    If a single sign-on (SSO) system, such as the open-software system Shibboleth,

    Shibboleth doesn’t provide SSO, it provides a mechanism for an identity provider (IdP) to assert the credentials of a user. The Internet2 Shibboleth software contains a rudimentary web sign-on capability. Given the moves within the UK HE and FE communities to move towards federated access management based on Shibboleth, it’s likely that an increasing number of institutions will be able to provide Shib IdP. The other end of the Shibboleth software (Service Provider, SP) isn’t any real use for access control for a dynamic system like an exam, that’ll need coding to speak SAML in its own right.

    08 Feb 2007, 21:56

  3. Thanks for pointing out my error in thinking Shibboleth was a single-sign-on system, Max. A colleague in IT Services has now explained to me how Shibboleth plays just a small part in the SSO we use for Perception 4:

    “Our Single Sign On system has a Shibboleth aspect to it. However, as Max says, it isn’t just Shibboleth, it’s a whole SSO system. We’ve plugged a bit of software on top of Perception that talks to our SSO system, it just so happens that part of that is via Shibboleth.”

    14 Feb 2007, 10:37

Add a comment

You are not allowed to comment on this entry as it has restricted commenting permissions.

February 2007

Mo Tu We Th Fr Sa Su
Jan |  Today  | Mar
         1 2 3 4
5 6 7 8 9 10 11
12 13 14 15 16 17 18
19 20 21 22 23 24 25
26 27 28            

Search this blog



Most recent comments

  • The LaTeX equations are a little wobbly in their baselines That's because they use mimeTeX. I am try… by Steve Mayer on this entry
  • Aha! You are right, John. I am familiar with a different terminology: Multiple Choice for what Quizb… by on this entry
  • Haven't you used the wrong question type, Trevor? Your answers look, to my unskilled eye, to be mutu… by John Dale on this entry
  • The usual factors in Information Assurance are: Confidentiality Integrity Availability And for syste… by Max Hammond on this entry
  • Is the workshop next Monday,26 March 07, open to anybody interested in computer aided assessment? If… by Kuldeep Singh on this entry

Blog archive

Not signed in
Sign in

Powered by BlogBuilder