Notes from “The eAssessment Question 2010” conference

This post will be updated after each conference presentation and thus constitute a near live-blog.

Dates: 17-18 March 2010

Conference web site: http://www.e-assess.co.uk/

Twitter hash tag: #eaqconf

Notes against Agenda

Developing Innovative, Reliable and Robust Solutions

Day One 17th March 2010

Opening Plenary Session, Chair: Colin Deal, Examware

Opening Key Note Address and responses will look at the strategic and policy issues underpinning the expansion of e-Assessment and the impact of technology and the need ensure robustness and reliability in e-Assessment delivery

[Apology to speakers and readers but I was late arriving due to overcrowding on the Tube and the conference centre been marked wrongly on map provided! – Whinge over.]

———————————————————————————————————————-

Highlight notes:

Simon Lebus, Group Chief Executive, Cambridge Assessment

  • Shift from summative to formative assessments largely facilitated by technology revolution – e.g. huge data storage cheap and analysis increasingly easy.
  • Developments in machine language recognition – particular benefits for language teaching
  • How foster trust in key stakeholders?
    • “Sat Nav assessment for …. (I think this phrase referred to technological hand-holding)
  • Children increasingly difficult to motivate by traditional teaching because of there expectations given there experience of technology in their day to day lives
  • Books are still are heart of education but students see books as peripheral
  • Public Confidence – contrast between education and professional domains
    – eAssessment widely used in safety critical professional training, e.g. medicine, aviation, emergency services; why if respect it there not in school education
    – Comment from floor in the current professional areas where there his good penetration of eAssessment largely teaching defined processes
  • Goal must be educational enhancement not overcoming weaknesses and staffing issues in current exam system
  • Respect diverse nature and purposes of education assessment (EU taxonomy 40 different functions REF?)
  • Need to develop the research base (As  researcher I say here, here!)
  • Be ambitious but not necessary to find “big bang” instead a series of incremental enhancements

———————————————————————————————————————-

John Winkley, Alphaplus Consulting (and Gavin ????)

Impact of Policy on Assessment:

  • Functional skills and eAssessment (spans professions and education domains)
  • Law of unintentional consequences
    – e.g. does a survey constitute a service for DDA (Disability Discrimination Act) – hoping to save costs by not addressing accessibility!  DDA Advisors asked were unsure. [MC took exception that they were seeking to by-pass accessibility]
  • Drivers for eAssessment:
    – economies of scale
    -authentic working environment for many students
    – transformation of education/learning environment
  • Need to design eAssessments integrated with learning/curriculum design to avoid law of unintended consequences (examples given but not noted here)

[Note-taking suspended because not core to my interests but plan to challenge the DDA example when it comes to questions! Pushed my buttons!!!] – “Come and see me at the break – apologise if given wrong impression.”

———————————————————————————————————————-

Professor Philip John, Heriot Watt University

“Innovations in mathematics eAssessment”

Celebrating 10th Anniversary of “Scholar” in Scotland

  • Responding to no expectations of students
    – web first port of call for almost everything
    – they are pragmatic not in awe of technology
    – find it strange when exposed to e-learning content and the have to take a hand-written exam
  • Scholar: A new generation of Students / Innovation – context:
    – league tables / national student survey
    – student expectations
    – what about the teachers?
    -large scale
  • Need to move away from eAssessment being synonymous with multiple choice
  • Wanted a seamless transition to summative assessment

Work in Mathematics:

  • Seamless transition – “exam” practice not a dirty word
  • Instant Feedback to students
  • Exploratory activity
  • [The mathematics modules in the Scholar system system was demonstrated on screen]

———————————————————————————————————————-

e-Enabling Assessment and Learning

[This session not blogged because I was involved in extensive and constructive conversation with John Winkley,  Director: AlphaPlus, following up on the accessibility exchange reported above.]

There are many issues underpinning the expansion of e-Assessment and its role in learning, training, teaching and qualification development. This session examines those issues from standards to technological challenges.

    Bob Gomersall, Chair, BTL

    Denis Saunders, MD and Founder, Calibrand

    Graham Hudson, DRS, ‘QA and regulatory implications of scanning’

———————————————————————————————————————-

Mobile Telephony, Learning and Assessment Session and Workshop

Convergent technologies and the growth of mobile telephony presents us with some of the most innovative and exciting developments in e-Assessment. This session will look at those developments and provide a hands-on workshop approach to using these emerging technologies

Gavin Cooney, MD, Learnosity

Fun interactive demo in the room of interactive peer-peer student assessment activities! [Impossible to blog – you have to be here!]

High-stakes summative assessment:

  • Again unintended consequences if key issues aren’t addressed:
    – must be an authentic measure – what trying to assess/measure
    -must withstand scrutiny
    – must be scalable and sustainable (many past projects have failed on this account)
    – must have market value (currency) – students looking to trade outcomes for entry into futher education or work
  • Evaluation model (how move from potential to application):
    – Technology opportunity
    – Assessment domain
    – Evidence trail (scrutiny) – automated or human judgements
    – operational viability (if address this before above can lead to short term gains only)
  • “Validity + Reliability x Viability = Value”
    Stuart Jones, Abel Consulting

    [This presentation not blogged.  Lost Internet.  Concerned use of mobiles devices (e.g. iPhone) for assessments.  Example of driving licence theory test.]

———————————————————————————————————————-

Innovation and People

The conference is about innovation and this final session of the day will examine some of the current leading edge projects; those pushing the boundaries of development, challenging old thinking and opening new areas of assessment and work.

    Bryan Mathers, MD, Learning Assistant, ‘Maintaining QA in e-portfolios usage’

[This presentation not blogged.  Quality Assurance in ePortfolios is not relevant to my role.  However if anyone is interested in accessibility of ePortfolios and standards that underpin this please e-mail me as I have been working with a colleague Andy Heath on this for the last 5 years of so:  m.cooper@open.ac.uk]

———————————————————————————————————————-

Linda Steedman MD, eCOM, ‘High Stakes Assessment is it worth the gamble”

Lessons learnt in high stakes assessment:

  • If is can happen it will! – e.g. forgotten passwords, fire alarms going off mid exam, …
  • Need immediate access to high quality technical support
  • Must be able to prove how authentic assessment is – audit trains, validated questions, comprehensive analytics, etc.
  • Expect appeals (cultural change) – must be able to produce script from system to evidence what was done by student
  • Continuous improvement is required, e.g. frequent reviews of question banks
  • Prepare the learners before the exam –  mocks, communications, etc.
  • Housekeeping important, e.g. mapping of students to PCs
  • All borderline fails must be checked
  • Have good back-up
  • Choose a good IT provider
  • Why take the gamble now?
    – Technology advances
    – more technically savvy students
    – improvements in instructional design
    – students want immediate results
    – cost savings due to computer support for assessment admin.
    IF above lessons followed successful implementation now possible

———————————————————————————————————————-

Mike Dearing and Andrew Stone, C&G, ‘Simulations: a case study of City & Guilds’ newest assessment’

A case study – Simulations:

    • Not leading edge
    • Why simulations? Expand on multiple-choice type assessments
    • Simulations here were to realise a realist IT based solution for the assessment context
    • Scenario based
    • Generic
    • Simple – easily understood by students
    • Engaged company called BTL
    • Integrates to Pearson Platform
    • Pitch to largest audience (choose word-processing/spreadsheets)

Challenges:

  • Fitness for purpose
    – meet C&G’s principles of assessment
  • Must provide adequate practice facilities
  • Vendor neutral
  • Assessing specific software disadvantages
  • Keyboard short-cuts
  • Hoe can assess open ended tasks

What happens next:

  • Embedding
  • Other IT units
  • Function skills ICT
  • Other vocational areas

(I don’t get this – why simulate a propitiatory software? – Will ask the question! – apparently because they wanted to be  be generic but they have not implements functional metrics into their system so not leveraging assessment advantages of developing their own approach.)

(I also raised the issue of keyboard short cuts for accessibility reasons – the reply there was paper based assessments as accessible alternatives.  THIS IS NOT ACCEPTABLE – IMHO)

———————————————————————————————————————-

Prof. Cliff Beevers ‘Update on development of the e-Assessment Association’

History:

Started in 2005 after feasibility study by the exam boards

Aims to promote effective eAssessment

Influence education thinking and policy

  • Objectives:
    • Build up support for those in the field
    • Communication Positive Aspects of eAssessment
    • On stop shop for eAssessment information

News:

  • E-journal appearing later this summer
  • Established set of independent experts

———————————————————————————————————————-

The 2010 AGM of the e-Assessment Association will be held at the end of day one sessions

[Not blogged, minutes to be published by eAA]

eaa logo with name from oct 09 150x96

———————————————————————————————————————-

Day Two 18th March 2010

Quality, Transparency and Accountability in e-Assessment

Chair Comment – eAssessment no mainstream in formative assessment to day we her more about the challenge in “Hight Stakes” summative assessment

With plenary addresses from Ofqual’s head of regulation policy and AQA main board member this opening day two plenary sessions will look at the impact of e-Assessment on high stakes qualifications, their development and delivery and how they link to classroom and workplace learning.

Julie Swan, Head of Regulatory Policy, Ofqual

  • Independent regulators but not a Ministerial Dept. so can challenge Ministers where necessary
  • Regulation awarding organisations not individual courses/assessments
  • Promoting regulated qualifications
  • Ensure regulated qualifications give reliable assessment and consistent over time
  • No distinction between general and vocational qualifications
  • Public confidence very important – there they have an objective to actively promote that
  • Efficiency objective key – value for money, promoting competition in the sector
  • Promote and assure accessibility in assessment
  • Understand the learner perspective[Video of interviews with students]
  • Technology is second nature for today’s learners
  • Can’t put eAssessment in the “too difficult” pile
  • Ofqual – to promote innovation in eAssessment
  • Bring about improvements (not just much current standards) – addressing the benefits
  • Promote close related working with other organisations like BECTA
  • Drivers for innovations
    – efficiency (they will assess this)
    – demand
    – quality
    – learners needs including accessibility
    – the regulator by “nudging” (regulator can block innovation usually by myths and false perceptions)
  • Risks – need to demonstrate improving standards – need to be open and transparent
  • Don’t regulate schools
  • On demand testing – “chicken and egg” situation – concern from established professions

———————————————————————————————————————-

Ruth Goddard, Director, AQA

“Protecting the past or engaging with the future” An awarding body view point

Change:

  • Last at conference 4 years ago and things have changed rapidly since then!
  • Students today depend too much on ink.  Pen and ink will never replace the pencil. 1907
  • Ballpoint pens will be the ruin of education in our country.  Students use these devices and then throw them away. 1950
  • You can’t use those calculators on the test.  If I let you do that, you wouldn’t ever learn how to do tables. 1980
  • Why would you ever want the Internet for student use?  It’s just the latest fad, have them use the library 1995
  • I don’t think most parents will buy their kids a new computer 2000
      • Schools have a responsibility to prepare young people for the world they are going to inhabit and not the one of the past.

        Mark Steed, Principal, Berkhamsted School

  • What the learners say about eAssessment
    –  “It was easier to understand and complete”
    –  “It was less complicated than written   papers”
    –  “We’re more into computers than writing on paper”
    –  “Can we do our exams like this again?”

  • What the teachers say about eAssessment
    –  “We very much enjoyed doing it – the kids were very keen!”
    –  “By doing it on screen, we felt we were raising our students’ achievements”
    –  “Students focus on line.  They use technology everyday and are comfortable with it.”
    –  “Students wanted to do the tests on-screen, so it was better motivation.”

  • Barriers or challenges – fears and preconceptions
    Dumbing down
    – Needing 200 PCs in one place
    – Risk of systems failure
    – Plagiarism
    – More work for teachers and exams officers
    – Lack of certainty

  • Technology enabled assessment must  be for learners!

———————————————————————————————————————-

Martin Ripley, 21st Century Assessment

  • Apologies to Mark and readers  – I missed this presentation attending a Doctor’s appointment
    – they say I will lie a little longer 😉

———————————————————————————————————————-

Developing Quality in e-Assessment

The challenge for e-Assessment is not just to maintain standards of quality but provide added value and improve both the delivery of assessments and the understanding and perception of those assessments.

    Peter Wilson, NIACE
    “How to promote quality in eAssessment”

  • If you can develop a good quality unit you can then develop a good assessment
  • Trends in QCF – reducing number of units per qualification and more and more shared units
  • Very few examples of credit transfers to date (liquidity)
  • In the interest of learners means making sure credits can be transfer; evidences don’t have to be repeated, etc.
  • Skills Funding Agency intends to aligned funding to credit (not guided learning hours) – provided credit system proves robust. No long have to prove how long students sitting in the class
  • eAssessment design has strong potential to support this shift

———————————————————————————————————————-

Geoff Chapman – Director, Calibrand Leading a group presentation including contributions from:

Chris Hedges – Policy Manager, Home Office UK Border Agency

Matthew White – Test Operations Manager, UFI


    – “Fit for Purpose Technology – delivering the Life in the UK test.”

[This talk is not blogged in detail]

  • Mainstream UK press very sensitive to developments and failing in eAssessment / Similarly immigration issues
  • Context of citizenship – this is a high stakes assessment for most candidates
  • Requirement for Language since 1664 (Norman French) – since 1948 requirement for English Language
  • 200,000 applications / year

Concluding Summary of Highlights:

  • Security (of exam, of centre, or residency)
  • Robustness
  • Fairness and Accessibility to those with no IT skills and little educational backgrounds
  • Fit for Purpose (does test accurately reflect the handbook~)
  • Mainstream High Stakes Large-Scale eAssessment
  • Emotions are mainstream around this as well
  • Aspirational aspect
  • Allow people to perform to the best of their ability
  • Dynamism
  • Conform to learner

———————————————————————————————————————-

John Winkley, Alphaplus Consulting

“Quality improvement in eAssessment”

  • Based on work for JISC
  • In HE most concerns are about insuring the basics (REAQ project University of Southampton:  www.ecs.soton.ac.uk/research/projects/619)
  • Over 100,000 items in question banks typical
  • Many professions in the field not comfortable with Data
  • Large banks of data and small number of people shepherding that data
  • Classical matrix provide good indicators where to target
  • Automatic metadata to metrics to improve assessment processing
  • Reading level of questions (established computer technology to assess this)
  • Aim to target 20% that give most problems
  • Learn to love data
  • Reasons for changing the paradigm in Assessment
    – Modularisation
    – Item Response Theory (difficult stuff but must be done)
    – Adaptive assessment depends on good data
  • Schools are now ready for eAssessment

———————————————————————————————————————-

The Value of e-portfolios in Assessment and Learning – eScape Workshop

eScape is one of the most exciting projects in e-portfolio and e-assessment development. In this extended session we will try to put such developments into the context of using technology to provide more effective, fit-for-purpose and robust 21st century assessment and give delegates the opportunity to see the technology in action.

Richard Kimbell, Goldsmiths College, University of London – Matt Wingfield and members of the TAG Development eScape Team

Susan McLaren, University of Edinburgh ‘The Scottish Experience with eScape’
Dale Hinch, Edexcel

    [I am not attempting to live-blog this because it is a highly interactive workshop session.  To my view the eScape e-portfolio system is a very high quality pedagogically rich ePortfolio tool and suggest you review it if looking for such a  system for your institution.]

  • Misfit statistic – measure of how closely each judge conforms to the judgement of the multiplicity of judges. [c.f. SSAA]
  • The challenge of dyslexia in ePortfolios when they focus on write evidence of learning – the multi-modal comment content aspect of the eScape system can can be used to address that to some degree
  • Students tend to have a high level trust of the software more than keep paper portfolios

Edexcel Pilot:

  • Creative and Media Diploma (student need to collect a large amount of evidence)
  • Do the hard things first in the project then most things easy afterwards
  • eScape approach a natural fit
    – previously many ePortfolios imposed extensive restrictions in terms of software, media types and files sizes on what the student can use as evidence
    – tagging of evidence to learning objectives
    – on-line storage – easy to manage technically and more secure
    – intuitive interface
    – Platform independent
    – Constructs a strong narrative within the evidence
  • Task Creation tool gives teacher flexibility and enthuses them
  • Students can collect high quality evidence of many different forms
    – positive experience for teachers and students stimulates effective learning
  • Task Creation -> ….. [iterative cycle]
  • Continuing to work with pilot centres
  • Marking and Moderation
  • Exploring a range of qualification type:
    -GCSE
    Functional skills
    English, ICT, D&T
  • Diploma Pilot Running to summer 2011

We have spent 7 years of out lives doing it – proud of achievements

Additional Note by Martyn Cooper:

Andy Heath and I, at the Open University, have been looking for the last 5 years at accessibility issues in e-Portfolios.  The fact that an ePortfolio system is unusally a many authors to a few reader system with non professional authors presents some chalanged here.  If you want to know more about this work please e-mail me: m.cooper@open.ac.uk.

———————————————————————————————————————-

Closing remarks from Chair

  • Thanks and goodbye

———————————————————————————————————————-

End

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s