Notes from CALRG Conference 2014

The Computers and Learning Research Group (CALRG) at the Open University (UK) has an annual conference.  Today and tomorrow is the 35th such conference.  This post is my notes on the presentations I attend (unfortunately I can not make them all).  There is a conference Twitter feed with hashtag #calrg14.  A temporary conference website is at: with a link to the programme.

CALRG Annual Conference Day One – June 10 2014

Discussant: Prof Rupert Wegerif, Exeter University


Patrick McAndrew (Director of IET) – Own experience over 16 years as an induction to the university.  A catch-up point.  Today the theme is mostly on Open Education.

Session I – Chair: Doug Clow

9:45-10:15 MOOCs, Learning Analytics and Higher Education: Perspectives on a recent study leave visit to the USA
Eileen Scanlon

  • The Americans sometimes slightly hallucinate our experience of Ed Tech
  • First stop – ACM Conference on learning at scale (single track)
  •  Bestthing-keynotefromChrisDide, of Harvard. – “Newwineinnow bottles”
    • It is not about the platform but what you do on the platform
    • Use of metaphors from film
    • Going big requires thinking small
    • Micro-genetic studies of online learning
    • People had forgotten all the learning science previously done
  • DistanceLearningOERs and MOOCs (Eileen’s presentation at conference)
    • The Open Science Lab
    • Edinburgh experience – professional development of surgeons
  • Next stop Berkley (Invitational Summit of 150 people)
    • Impact on residential campus based universities
    • Relying on schools of education to measure student learning
    • ReflectiononEDX platform
      • Transforming the institution (MIT in this case)
      • Learn about learning
      • E.g. required physics course – group learning – lot of use of online assessment
      • Comparison of performance in MOOCs of those taking residential course versus those not
      • Drown in information if Google assessment of EDX
    • Simon initiative at Carnegie Mellon
      • AI and Cognitive Tutors
      • Broader than the institution
      • Global learning council
      • Spin out company called “Cognitive Tutors”
      • Individualized instruction seen as gold standard for education
  • Then visited Stanford
    • TheLytics Lab (Learning Analytics)
      • Using learning science with open educational delivery
      • Moving from fragmented approach to systematic improvement of this type of pedagogy
      • CSCL (conversation) ->MOOC space
      • Scale of work in Stanford on MOOCs is staggering
      • Still individual academic driven
  • Then various other conferences
  • Future Learn Academic Network
    • Originally 26 partners now expanding and going more global
  • ESRC proposal on future of higher education
    • Partners: OU, University of Edinburgh, Carnegie Mellon, Oxford University

10:15-10: 45 Squaring the open circle: resolving the iron triangle and the interaction equivalence theorem
Andy Lane

  • Visual Models
    • How visualization can help with understanding/sense making
    • They can equally conceal
    • The Iron Triangle – sides: Scale, Quality, Cost
      • If one dimension changed significantly it will compromise others
    • John Daniel – open distance learning could break the iron triangle
    • Interaction Equivalence Theorem (EQuiv)
    • Supply-side vs demand-side (what about the students?)
    • Adding a circle of success to the iron triangle
    • A student centred iron triangle
      • motivation, preparation, organisation
    • A student centred Interaction Engagement Equivalence Theorem

10:45-11:15 Exploring digital scholarship in the context of openness and engagement
Richard Holliman, Ann Grand, Anne Adams and Trevor Collins


  • Public engagement with a research mandate
  • Research councils fund catalysts
  • An “ecology” of openness
  • Action Research [Lewin 1946]
  • The Edge tool
  • How do we find ways if assessing where staff are and then support them?
  • Research Questions
    • What methods and technologies are researchers using to: make research public, make public research, enable the public to collaboratively research (citizen science)?
    • how do researchers conceptualize the role of students?
  • Scholarship reconsidered
    • discovery
    • integration
    • application
    • teaching
  • Awareness / Responsibility / Sustainability
  • Institutional strategy for open, digital and engaged scholarship
    • What should we try to change?
  • Types of researcher: the fully wired; the dabbler; the brave trier; the unimpressed
  • “The Open Scholar is someone who makes their intellectual projects digitally visible …”
  • Policies / Procedures / Practices

[The remaining  session of Day 1 I was not able to attend but the programme in included here]

Session II – Chair: Ann Jones

11:30-11:55 The OpenupEd quality label: benchmarks for MOOCs
Jon Rosewell

11:55-12:20 From theory to practice: can openness improve the quality of OER research?
Rebecca Pitt, Beatriz de-los-Arcos, Rob Farrow

12:20-12:45 Open Research into Open Education: The Role of Mapping and Curation
Rob Farrow

12:45-13:10 Strategies for Successful MOOC learning: The Voice from the World Record Breaker
Bernard Nkuyubwatsi
Session III – Chair: Rebecca Ferguson

14:00-14:25 The role of feedback in the under-attainment of ethnic minority students: Evidence from distance education
John T.E. Richardson, Bethany Alden Rivers and Denise Whitelock
14:25-14:50 Evaluating serious experiences in games
Jo (Ioanna) Iacovides
14:50-15:15 Social media for informal minority language learning: exploring Welsh learners’ practices
Ann Jones
15:15-15:30 TEA/COFFEE
Session IV – Chair: Inge de Waard
15:30-15:55 What students want: designing learning to optimise engagement in digital literacy skills development
Ingrid Nix and Marion Hall
15:55-16:20 Recording online synchronous tutorials to support learning
Pauline Bloss, Elisabeth Clifford, Chris Niblett and Elke St.John
16:20-16:45 Open Education needs Education for Openness: a dialogic theory of education for the Internet Age
Rupert Wegerif
16:45-17:00 Discussant – Rupert Wegerif

CALRG Annual Conference – Day 2 – June 11 2014

Session V – Chair: Mark Gaved

9:40-10:05 ‘nQuire-it’: The design and evaluation of a mission-based web platform for citizen inquiry science learning
Christothea Herodotou, Eloy Villasclaras- Fernández , Mike Sharples

Notes from this presentation lost in the ether 😦

10:05-10:30 3D Virtual Geology Field Trips: Opportunities and Limitations
Shailey Minocha, Sarah-Jane Davies, Brian Richardson and Tom Argles

  • Can do things unable to d in a real field trips – e.g. drape maps over mountains, see geological cross sections
  • Us Unity 3D Game Engine to build a 10km x 10km area mapping and imaging the real world (around Skiddaw, England)
  • Can pick up rocks and examine under microscope
  • Includes a chat facility for tutor group communication
  • Leave these tools out of the application so as not to compromise the immersion
  • Addresses accessibility with transcripts and full keyboard only access
  • Able to “fly” and “teleport” (on a real field trip a lot of time wasted travelling between sites)
  • Avatar based environment
  • Students use a paper based notebook as they would in the field
  • Integrate the virtual microscope (existing facility) but now contextualized learning
  • Cloud server can handle up to 500 students at one time

10:30-10:55 Juxtalearn: From Practice into Practice
Anne Adams and Gill Clough

  • Large EU project
  • Driver – not enough taking science and technology at school – employment implications
  • Science and Technology engagement through “creative performance” and reflective learning”
  • Threshold concept (TC)
    • Where students find challenges
    • When they get it it is transformative
    • Irreversible – not readily forgotten
    • Integrative – brings concepts together
  • Learning Pathways and Threshold Concepts (different ways from introduction of concept to internalisation of it)
  • Develop understanding through creative video making
  • Tricky Topic Tool
    • Teachers identify tricky topic
    • Teachers create an example
    • Teachers write down student problems
    • Teachers fill in Taxonomy (linked to student problems)
      • e.g terminology, intuitive beliefs, incomplete pre-knowledge, …
  • Taxonomy scaffolds quiz creation
    • Tool to facilitate this
    • Integrates detailed feedback to the student
  • Demo

Session VI – Chair: Anne Adams

11:15-11:40 Citizen Inquiry: From rocks to clouds
Maria Aristeidou, Eileen Scanlon, Mike Sharples

  • Citizen Science + Inquiry based Learning -> Citizen Inquiry
  • Inquiring – Rock Hunters (Initial Study)
    • 24 participants
    • 12 rock investigations
    • discussion and feedback on chat and forums
    • Data collection – questionnaires, System Usability Scale [John Brooke, 1986], …
  • [Note taking interrupted]

11:40-12:05 Imagining TM351 – Virtual Machines and Interactive Notebooks
Tony Hirst

  • TM351 – New Level 3 30 point module on data
  • Two new things:

1. Virtual machines (to overcome the diversity of machines being used by students)

    • Interfaces increasingly browser based
    • Virtual box installed on student machine and browser used as interface
    • Virtual machine can be on cloud server – then can use on a tablet

2. Notebook Computing

  • Literate programming / reproduce-able code or research
  • Code should be able to be read as an essay (self documenting) – read well to human and executable by the machine
  •  Can’t reproduce data analysis from traditional academic papers – reproduceable research includes the tools to enable this
  • Using IPython
  • Corollary to spreadsheets
  • Task orientated productivity tools
  • Cells
    • write text
    • uses “mark-down” simple text based mark-up
    • other cells contain python code
    • e.g. the software creates the table – avoids errors in production and editing
    • similarly with maps and paths
  • IPython server in VM – interface in browser
  • Exploring using OpenDesignStudio so students can share and critique each others code in executable form (see:
  • Example shown


12:05-12:30 MASELTOV – mobile incidental learning services to support language learning and the social inclusion of recent immigrants
Agnes Kukulska-Hulme, Eileen Scanlon, Ann Jones, Mark Gaved

  •  Using smart phones to support language learning
  • Addressing those with low educational level and from different culture
  • Incidental learning approach
  • MApp: a range of services
    • Field local mapping
    • Social network
    • Information resources
    • Translation
    • Navigation guide
    • Language learning
    • Serious game
  • These are separate apps but integrated in the platform
  • High penetration of smart phone among target audience
  • Technology uncertainty period
    • Many purchase phone ahead of travel
    • Android phones most popular
    • May have multiple phones
    • Seek out free WiFi
    • Word of mouth expertise highly valued
  • Howdowe enable transition from problem solving to reflective learning?
    • relating immediate situation to broader context
    • Feedback and progress indicators
      • Study planning and goal setting
      • Indicating completion
      • Supporting sense of community
      • Building confidence
      • Gamified approach
      • Quizzes
  • What evidence that this approach to language learning is effective?
  • Are there clusters of tools use?
  • Demo

12:30-12:55 Knowledge Transfer Partnership: Booktrust and the Open University
Natalia Kucirkova, Karen Littleton, Teresa Cremin and Laura Venning

  • Ongoing project started this year
  • KTP-objectives:
    • Extending book trust work on promoting reading for pleasure
    • Contribute to digital literacy
    • New knowledge and understanding of digital technologies and the opportunities they provide
  • Synergy of two organisations
  • Looking at books created on iPads (created by children or parents using words and images)
  • The ability to search for meaning is enhanced by creating stories
  • Book Trust:
    • Charity founded in 1920s
    • Encouraging reading for pleasure among children and families
    • Run book gifting programmes
    • Book-start – packs delivered by health visitors and via libraries
    • Reception year programme
    • Now seeking to develop the digital side of their work
    • Undertake research on reading habits and how reading contributes to peoples’ lives
    • Reading Habits survey 2013-14

Session VII – Chair: ?

14:00-14:25 Flipped teachers’ views of the impact of open practices on students
Beatriz de los Arcos

  • Flipped teacher – move the instruction online more discussion and analysis in class
  • Help with “homework” given by experts
  • Survey of OER use by teachers and how impact on students
  • “I do not treat this curriculum as mine – it belongs to the class and the world”
  • Example of a learning activity on Chaucer’s Canterbury Tales – kids turned in 82% of homework on time
  • OER enables new ways of teaching and learning
  • How do we measure the success of the flipped model?
    • A lot of teachers respond to do with student motivation and engagement
  • Most teachers informally adopt OER practice (e.g. uploading to You Tube) but don’t know about CC licenses etc.
  • Does flipping with OER give a better “flip” than working with closed resources?


14:25-14:50 The pedagogical design, user profile and evaluation of a Mobile app to teach beginners’ Chinese characters
Fernando Rosell-Aguilar and Kan Qian

  • Examples of tones in Chinese where same syllable means different things – but context means in practice mistakes not significant
  • About 10,000 characters in common use with typically 12 strokes
  • No space between characters to denote separation of words
  •  Stroke order is important – but this also aids memory of characters – in Chinese primary schools they would chant this
  • Pinyin (Roman letters) is used to teach pronunciation because no correspondence between character and pronunciation
  • Grammar very simple (no past or future tense) – verbs stay the same – no plural singular
  • Rationale
    • To provide an aid to learning
    • To raise profile of the introduction to Chinese course
    •  To fulfill KMi objective to produce revision aids
  • Pedagogical design
    • Bite-sized learning
    • Progressive Learning 20 lessons must be taken in order
    • Integrating writing, listening, reading and vocabulary
    • Gaming feature
    • Personalised learning
  • 4 Sections
  • Challenges of working with App Developers
    • What can be done with what desired
    • Timing issues
    • Technical affordences vs pedagogy
  • User profile and evaluation
    • More males than females (unlike other modern languages more males than females study Chinese)
    • Median Age 30-39
    • 91.9% describe themselves as beginners
    • 75% learning Chinese informally
    • Why learn Chinese:
      • Personal interest
      • Family ties
      • Non-Chinese living in China
      • Business use
    • False expectation of ability to learn fluent Chinese from app
    • App rated positively 86% very good or good
    • Good ratings for learning to write but better for learning to recognise characters
    • 82% app as additional to other learning but 18% using it as their main resource
  • Conclusion
    • Met objectives towards a large degree but no evidence of people using the app then signing up for the course
    • Varied mix of users (gender, age, etc.)
    • Android version limited character set iOS more comprehensive
  • App Chinese Characters First Steps –

14:50-15:15 Models of Disability, Models of Learning, Accessibility and Learning Technologies
Martyn Cooper

My presentation so not noted but slides are available on SlideShare at:


Session VIII – Chair: Canan Blake

15:30-15:55 Computer-marked assessment as learning analytics
Sally Jordan

  • Using in iCMAs in teaching since 2000
  • Ellis (2013) assessment often excluded from learning analytics but this is “stupid”
  • Assessment give deep information about learner engagement
  • Analysis at the cohort level
    • Look at questions that student struggle with (from hard data not student opinion)
  • Example of graphic illustrating number of tries students take to get correct question answer in a maths assessment
  • Look are reasons for repeated wrong answers
  • Measuring student engagement – “750 students used my iCMA”
  • iCMAs in formative use exhibit those that just click on it but don’t engage (about 10%)
  • WhendostudentsuseiCMAs?
    • Strong bias towards cut-off dates
  • Length of response to short answer questions – if say a word limit students tend to write near to that limit (see it as a hint)
  • Student engagement with feedback – comparisons between students and comparison between modules
  • Generally students do what they believe their teachers want
  • Engagement with computer marked assessment can be used as a proxy for deeper behaviour
  • Transcend the testing paradigm and see assessment for learning not assessment of learning

15:55-16:20 Open Essayist: Opening automatic support for students drafting summative essays
Denise Whitelock, John Richardson, Debora Field, Stephen Pulman

  • The SAfeSEA Project, see:
  • Present summaries of students essays back to students to facilitate their reflection
  • Not tell students what to write (or what is right)
  • Identifies Intro, Main Section, Conclusions, Keywords
  • Generates different visual representations of the essay – one research question is what representations the students find most helpful
  • Nodel graphs represent repeated notions
  • Marked contrast between highly marked and low marked essays
  • Nodes closer together in the better essays – vector length represents the connectivity between sentences
  • In 2014 made available to students on MAODE, University of Herts and British University in Dubai
  • Non-native speakers expressed found it very helpful
  • A lot of students do not see how a computer system could help them with their essays

16:20-16:45 Findings from a survey of undergraduate use of mobile devices for OU study
Authors: Simon Cross, Graham Healing, Mike Sharples

  • ePedagogies of handheld devices
  • Document and analyse the patterns of use of OU students
  • Align with other surveys – e.g. OU Student Survey
  • Becoming a longitudinal study
  • Modules like a Lego set – what students do with it may be different than intended and may be influenced by the technologies they use
  • 82% students mobile phones, 50% tablets, 37% e-readers 8% none of these
  • 30% bought tablet for OU study
  • 16% bought e-reader for OU study
  • Evolving data set – resource for future research
  • Insights for module development
  • Evolving survey instrument
  • Evolving analytical framework
  • Technology barriers -> learning barriers

16:45-17:00 CLOSE

No discussant today – shame because I like this feature of CALRG Conferences.

Accessibility and Learning Theories

I am writing a paper for Computers and Education with Jane Seale of Southampton University.  The title is:

E-learning, Accessibility & Pedagogy: In search of the missing tools of practice

The paper seeks to review the interrelation between accessibility and learning theory as it relates to eLearning.  Here I give some key points from the paper for comment if you wish.

Broadly speaking, accessibility in relation to eLearning is understood as ensuring that learners are not prevented from accessing technologies or the content and experiences offered by technologies on the grounds of their disability.

The IMS Accessibility SIG defined Accessibility, in an eLearning context, as the ability of the learning environment to adjust to the needs of all learners [IMS 2002, Guidelines for Developing Accessible Learning Applications]. Accessibility is thus determined by the flexibility of the eLearning system or learning resource to meet the needs and preferences of all users. These needs and preferences may arise from:

– their environment (e.g. working in a noisy environment)

– the tools they use (e.g. assistive technologies such as screen-readers, voice recognition tools, or alternative keyboards, etc.)
– or, a disability in the conventional sense.

Accessibility and different theories of learning
Different learning theories are chosen by different educators to underpin their teaching. A brief survey of high profile learning theories classified according to Mayes & DeFreitas is offered here and their accessibility implications discussed. There are numerous models of learning; this fact reflects that reflect the complexity of what is being modelled. Learning involves perceptual, cognitive, communicative and memory aspects of psychology and these are areas we only currently have a partial understanding of. The models selected for consideration here are principally the ones that in the authors’ experience have received some prominence within teaching at a higher education level. Further they are models that have an implication for then use of technology in mediating the teaching and learning.

A range of learning theories are reviewed and their accessibility implications discussed.  These include:

Behaviourism is the School of Psychology that relates to behaviour as a central component of learning. It began with central findings of Ivan Pavlov (1849-1936) about the ‘conditioning reflex’. Pavlov provided the basis of behaviourism highlighting the importance of stimulus for learning. Later John Watson, an American Psychologist (1878-1958), building on the work of Pavlov outlined a whole new branch of psychology know as behaviourism, denying Freudianism and heredity and instead explaining behaviour and learning as part of nervous ‘wiring’. B. F. Skinner, an American psychologist (1904-1990) extended the behaviourist approach, describing the learning process as beginning when we are babies, then we are a black box upon which experience and conditioning are written. He developed ideas about the ‘operant conditioning’ and ‘shaping behaviour’.

Accessibility and Behaviourism
The essential implication of the behaviourist model of learning is the importance of stimulus. Now this has particular implications especially for people with sensory disabilities. How would Pavlov’s dogs learnt that food was neigh if they could not hear the bell? Now extending this to eLearning highlights the importance of providing stimuli for learning in different modalities.

Constructivism is a theory based upon the thinking of John Dewey, an American philosopher (1859-1952), who questioned traditional epistemology, Dewey instead came to believe that:

“…the theory of knowledge must begin with a consideration of the development of knowledge as an adaptive human response to environing conditions aimed at an active restructuring of these conditions.”

Unlike traditional approaches in the theory of knowledge, which saw thought as a subjective primitive out of which knowledge was composed?

Further reference is made to the work of Seymour Papert, Piaget and Vygotsky here.

Accessibility and Constructivism
The key point here is that a student with disability needs full access to the environmental context of the learning.

The Laurillard conversational model
Contemporary accounts of student learning accept that it is an active process and depends on interaction. Laurillard offers a model of student / tutor / courseware interaction, and this is outlined briefly here.

Laurillard offers a classification of educational media based on a conversational framework (after Pask and with due deference to Socrates), which identifies the activities necessary to complete the learning process. She considers the learning process as a kind of conversation, and asserts that this process ‘must be constituted as a dialogue between teacher and student (or student and student), operating at the level of description of actions in the world’. Her classification system is based on the type of interaction between instructor and student when a particular medium is used. She classifies educational media as discursive, adaptive, interactive and reflective, and raises issues about the nature of feedback, goals and control of student learning. Her review of media asserts that currently only tutoring systems and a combination of tutorials and simulations can claim to address the entire learning process as specified in her model. However her conclusion is not that these are the only worthwhile media, but that educators should consider media combinations to construct learning packages that combine complementary features. Summative and formative assessment can form one aspect of the interaction referred to in the model (although the first author has predominantly used it when analysing practical work in science and engineering education, which of course in itself can be assessed). The teacher constructs the assessment, the student interacts with it and there is feedback via the marking and review

In book introducing her ideas [Laurillard, D. (1993) Rethinking university teaching, Routledge, London], Laurillard highlights how different modes of learning map onto the conversational model. For example “Learning through acquisition” (teacher as storyteller or lecturer) maps to a teacher to student arrow in Laurillard’s diagram from the teachers conceptual knowledge to that of the students. In assessment teacher can implement a wide range of leaning modes depending on the types of examination and question chosen. This could include “Guided discovery” that requires all the conversational components.

Accessibility and the Laurillard Model
Key in the Laurillard Model are the various conversations it embodies. Laurillard uses these to analyse the use of media in learning. However this can be further extended to analyse the accessibility of all the media used to support these different conversations. One aspect of the Laurillard model points to practical forms of assessment where the teacher sets up something in the “real” world for the student to examine, interact and reflect upon. Practical exams are not specifically a topic of this paper however there is a growing trend to make practical available as part of an eLearning context. The second author led a major EU funded project PEARL that argued such remote controlled labs could increase access to practical work, particularly in science and engineering subjects, for disabled students.

Cognitive Learning Theory

Cognitive learning theory closely relates to how cognitive skills develop. This set of theories is underpinned by cognitive science and the development of psychology. While social cognitive theory owes its heritage to social learning theory founded in the1800s; Albert Bandura in 1986 wrote his seminal book: Social Foundations of Thought and Action: A Social Cognitive Theory, which ignited social cognitive learning theory.

Computational Theories of Learning
Computational theories of learning: derive from artificial intelligence and metaphors of computer science rather than from psychology, cognitivism, or philosophy. Starting from Alan Matheson Turing the computer has been used as a metaphor for the human brain and its functioning.

A cybernetic model of learning
Cybernetics was a term coined by Norbert Wiener in his seminal book on systems theory. It is dependent on the concept of feedback which is of course central to the idea of assessment of learning.

Typically a learning activity is followed by an assessment of some form. The results of the assessment are feedback to the student who compares this with their own perception of their learning and they move through the system again either retaking the same learning object and assessment or moving onto a new learning object. The diagram is only shown simplistically with any conditional branching based on the assessment to alternative learning objects or activities omitted.

Inclusion and the Cybernetic Model

The essential implication of the cybernetic model for accessibility is that not only should any learning activity and assessment be accessible but the form of any feedback needs also to be accessible to the student.

Other Models of Learning
There are many models of learning stemming from different schools of thought. In this paper a diverse but limited selection has been made but it makes no pretence at being comprehensive. Other models of learning that are more briefly considered include:

  • Situated Learning
  • Action learning
  • Andragogy
  • Communities of practice
  • Instructionism or instructivism
  • Learning styles
  • Motivation
  • Problem-based learning
  • Socially-mediated learning

Discussion and Conclusions
The conclusions to the paper are still being developed and discussed but cover:

  • Many accessibility related tools exist, but they do not seem to be having much impact on teachers and teaching practice in further and higher education-, evidence by the fact that the accessibility of e-learning in colleges and universities is still incredibly variable;
  • This variability in accessibility suggests that a different set of tools may be needed to help teachers develop their accessibility practice further;
  • It is widely agreed that there is a link between effective pedagogy and effective use of technology. There may also be therefore, a link between pedagogy and accessibility. Whilst the link may be complex, it would seem to be worth exploring further, particularly with a view to ascertaining whether pedagogical tools have a role to play in the development of accessible e-learning and accessibility practices;
  • Pedagogical tools are tools that mediate a teachers’ action, offering clear and detailed principles regarding learning that can be easily and readily translated into teaching practice.
  • Two sets of pedagogical tools that are worth exploring further are models and theories of learning and learning design tools
  • Having explored and discussed these two tool sets we conclude that they maybe useful in changing/developing the accessibility practices of teachers in further and higher education.