Confidence in eAssessment

I am currently attending the IMS Learning Impact conference in Long Beach, California (http://www.imsglobal.org/learningimpact2010/).  I attended a presentation by Jeff Borden of Pearson eCollege who introduced (to me) the concept of confidence in assessment which is particularly pertinent to eAssessment.

Top Left = Misinformation; Bottom Left = Uninformed; Top Right = Mastery; Bottom Right = Doubt
Confidence in eAssessment Matrix

The confidence matrix is shown left.  It is I think self-explanatory but to expand on the idea I comment here on each cell.  If a student gives a wrong answer with a low-level of confidence they are in the “Uninformed” category.  However if they give a wrong answer but believe they are right then they are classified as “Misinformed”.  If they give a correct answer but with a low-level of confidence they fall into the “Doubt” category whereas if they are correct with a high level of confidence they are in the “Mastery” category.

I consider that this approach has the potential to greatly improve the validity and utility of eAssessments where students select from a limited number of possible answers.  If as well as giving their selected answer students are asked to give a confidence level the effect of random chance selection of the right answer is reduced and the assessor is given more meaningful information about the students knowledge.

Reflecting on this I consider a 3 point confidence scale (High, Medium, Low) would probably be optimal so that for each question the student would register one of those values to indicate their confidence in their selected answer.  There are many options for how the matrix is applied.  It could be on a question by question basis but is more likely to be useful if aggregated, by some algorithm, over a set of questions on a related topic.

I would be happy to receive any comments but particularly from anyone who has experience of using such an approach in real assessments.

Advertisements

6 thoughts on “Confidence in eAssessment

  1. Little further detail was given in a 30 min talk which introduced other topics on data mining student performance as well. I hope someone will comment on this blog post who has applied such a confidence matrix.

    1. Hi Martyn – I’ve heard Jeff present before. He’s a really good speaker – you should hear his creativity keynote. It’s as motivating a speech as I’ve ever heard in my 20 years of teaching. I got to hear him in Denver last year.

      CBL is an interesting concept. I believe it’s been around as a theoretical approach since the 30’s, but many feel it will help with “unfair” measures such as standardized tests for K-12 by leveling the playing field a bit. The best info I’ve seen on it came from Dr. Shuford (sp?) out of UCLA. I believe he actually created an algorithm based on both confidence and score. I’ll have to check for his paper when I’m back at my office. Until then, thanks for the great blogs!

  2. Hey Martin,

    Hope you’re enjoying the sun in LA!

    There’s quite a bit on Confidence-Based (or Certainty-Based) Marking on the UCL site (http://www.ucl.ac.uk/lapt/), including an explanation of their scoring system which is interesting. And if I remember correctly, I think OpenMark can be tweaked to accept CBM questions.

    The Interwrite voting system units used to support CBM too, but I believe that functionality has been removed from recent releases.

    Sure I had a paper here that described CBM being used with medical students (where confidence in your decisions is quite important). If I can find it, I’ll post a link or send it to you directly.

    😉

    Kenji

  3. Thanks Kenji – I look forwrd to any further information you have. I should have known about OpenMark.

    (BTW my wife and I are on a weeks holiday near Edinburgh in June – I hope we can meet up for a beer or a meal.)

    Cheers, Martyn

  4. Still looking for that paper… had a bit of a PC meltdown a couple of months ago and as a result lost a whole heap of stuff (yes, backing up the data would have been the smart move – sigh).

    Happy to catch up with you in June – just let me know when you’ll be up!

    😉

    Kenji

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s