I am currently attending the IMS Learning Impact conference in Long Beach, California (http://www.imsglobal.org/learningimpact2010/). I attended a presentation by Jeff Borden of Pearson eCollege who introduced (to me) the concept of confidence in assessment which is particularly pertinent to eAssessment.
The confidence matrix is shown left. It is I think self-explanatory but to expand on the idea I comment here on each cell. If a student gives a wrong answer with a low-level of confidence they are in the “Uninformed” category. However if they give a wrong answer but believe they are right then they are classified as “Misinformed”. If they give a correct answer but with a low-level of confidence they fall into the “Doubt” category whereas if they are correct with a high level of confidence they are in the “Mastery” category.
I consider that this approach has the potential to greatly improve the validity and utility of eAssessments where students select from a limited number of possible answers. If as well as giving their selected answer students are asked to give a confidence level the effect of random chance selection of the right answer is reduced and the assessor is given more meaningful information about the students knowledge.
Reflecting on this I consider a 3 point confidence scale (High, Medium, Low) would probably be optimal so that for each question the student would register one of those values to indicate their confidence in their selected answer. There are many options for how the matrix is applied. It could be on a question by question basis but is more likely to be useful if aggregated, by some algorithm, over a set of questions on a related topic.
I would be happy to receive any comments but particularly from anyone who has experience of using such an approach in real assessments.