Multiple Choice Marking

New! D.I.Y. Multiple Choice Forms.

Not surprisingly, interest in using the School's Multiple Choice marking facility is increasing. This Web Page is intended to provide information about, and promote the use of, this system.

If you have a large class and can reformulate all or part of your assessment procedures in multiple choice form, then the efficiency gains can be spectacular. In First Semester 1999 approximately 2400 script hours of examination material were processed in this way.

Other advantages include:

Elimination of manual entry of marks.
Current usage and checking procedures appear to give extremely high accuracy, almost certainly better than manual marking.
Ability to vary number of questions and number of choices.
Facilities for processing non-homogenous sets of answer forms, such as come from a single class using several different sets of questions (to reduce copying, for example).

It seems likely that other useful applications will be found. For example Humphrey Gastineau-Hills has developed a method to speed entry of marks from first year assignments.

Note that the current software cannot handle the difficult problem of reliable recognition of handwritten characters. Instead it is based on simple and reliable Optical Mark Recognition (OMR) technology.

Operational Details

The completed forms are first scanned using a high-speed, batch-feed scanner. The scanner processes around 1 page per second, and the images are stored on the hard disk of the controlling PC. It can also scan both sides of the sheet simultaneously, although this facility has not so far been used.

The project initially used a low-cost flatbed scanner/sheet feeder combination. This was much slower, and prone to misfeeds, but still much faster than manual marking. The present machine seems largely immune to paper misfeeds.

Here is a typical completed multiple choice form, designed in TeX by Roger Eyland (and with student name erased):

Click here for a bigger image The main features of the form are:

Spaces to write name and SID.
OMR area to enter SID.
OMR area for answers.
Limited space for questions, if required. For more than a few questions most people put the questions on a separate sheet.

The OMR software can reliably handle rectangular fill-in arrays of of any size, and interpret the results in various natural ways. In addition to OMR arrays, the software can also read barcodes and capture images from sections of the page. In this example, the OMR process on this image generates the data

9821105,c:\images\2002\sid0033.PCX,3,1,4,4,2,2,4,5,4,1

The first item in this list is the SID. The second item is the pathname of a small image containing the handwritten SID (used for later checking). The remaining items show the selected answers. It is easy to match these against the correct answers with a short python or perl script, or by reading the data into a spreadsheet.

The forms can be further parametrised by adding `office-use only' data in extra OMR fields, or barcodes.

Checking the Results

There are various sources of error which need to be monitored.

Students commonly enter their SID incorrectly. The most common problem is the choice of the first row of the SID array to represent the digit 1. There are several ways to check for this.
- If the forms are sorted alphabetically by name, the SIDs can be matched against a correct list. An out-of-order or unmatched SID signals an error.
- Alternatively images of the handwritten SIDs can be captured, decomposed and sorted according to the corresponding OMR selection. Then errors can be found by visual inspection.
Some students complete the forms in ink, and then cross out, or use whiteout, to make corrections. In the former case the software flags a MULTIPLE selection. In the second case it typically rejects the whole array.
If the form is completed in hard pencil, the mark may not be dark enough to trigger recognition, and the software flags a BLANK.

Fortunately the software is quite conservative and prefers to flag errors rather than make guesses. It also provides a reasonably friendly user interface for checking and correcting these types of recognition failure.

Future Developments

The success of the operation is quite sensitive to some features of the form design. The forms currently in use are the result of some experimentation and tuning. Note that there is now a web interface for customising and printing OMR forms for individual lecturers and courses.

If you are interested in this, or have other suggestions for future developments, please let me know.

Nigel O'Brian
nigel@maths.usyd.edu.au
4/8/99