Internal project proposal: Learning Analytics for Disabled Students in STEM subjects

I am currently working on an internal project proposal: Learning Analytics for Disabled Students in STEM subjects (LA4DS-STEM). Hopefully it will run from April – December 2014.

The LA4DS-STEM project will review the potential of Learning Analytics in higher education, specifically in STEM, and with an emphasis on supporting disabled students and facilitating accessibility enhancements.

Learning Analytics is defined as the measurement, collection, analysis, and reporting of data about learners and their contexts, for purposes of understanding and optimising learning and the environments in which it occurs.  The LA4DS-STEM project will specifically explore the following STEM application areas for Learning Analytics. A key output of the project will be an external funding bid for a larger-scale collaborative project. The work of LA4DS-STEM will inform pilots in this project. Provide envisaged benefits are confirmed, this should lead to enterprise level implementation within the OU and across HE.

The findings of the LA4DS-STEM project will be disseminated, firstly throughout the Science and MCT faculties, then to the wider university. External dissemination will highlight the OU’s lead in this field.

JISC Digital Festival – Notes (Day 2)

I have spent most of the morning interacting with reps of the various exhibitors here.  Now to rest my legs I have settled down in Hall 1 for the keynote by  Sugata Mitra, Prof. Of Educational Technology at Newcastle University.

Notes from Keynote

Sugata was the originator of the ‘Hole in the Wall Experiment‘. He plans to review the last 15 years of work and review trends.

The hole in the wall experiment

ATM like computer  in a hole in the wall. They (the slum kids in New Dehli) did not know English and the interfaces were in English. Street children were browsing within 6 to 8 hours and teaching each other.  Conclusion groups of children left with a computer would reach the level of the average office secretary in the West in about 9 months. [Video shown of this work].

The children’s achievement of their proficiency happened because not despite of the absence of an adult teacher/supervisor.  After 4 to 5 months the teachers reported that their English was much improved. Discovered they were using a search engine to find quality content and copying it down on to paper. Question – why we’re they copying down the right things?  They seemed to know what they were writing.  Then gave them educational objects.  Working in groups they seemed to be able to locate the right information and select it.  Groups of children could reach educational objectives of their own if they wished to. People supposed that when got to in depth learning or skills acquisition they would need human intervention. However, could not find the limits of this learning.

In England turned the hole in the wall upside down. Created the chaotic environment of the hole in the wall inside the clasroom with just a few computers. Made up some rules: free discussion and free movement allowed. In period 2008-2010 this led to the descriptor of self- organising learning events. E.g. For 7 year-olds “why is a polar bears coat white”.  Given the the choice between a hard and easy question the children opted for the harder questions. They were able to do GCSE questions about 6 to 7 years ahead of time. Called these Self Organising Learning Environments (SOLE).

In other countries around the world similar results.  C.F emergent phenomena or self ordering or spontaneous order in the Natural Sciences.  Tested limit of this method in Southern India. Research Question: can 11 year-olds learn the process of DNA replication?  Experiment was a failure but the students self studied why DNA replication sometime went wrong causing disease.  Pre and post testing showed those working 10 years ahead of their time. Used a non scientist and the method of the grandmother.  Using an older adult to stand behind and encourage.

[Slides: Schools in the cloud]

Constructing 7 pilots trying to level the playing field in primary education comparing India with UK.

Q&A

Experience with older students?  – Used to think method applied to ages 6 to 14 but beginning to show that it is not restricted to this. Experiences reported with 16-18 year olds, in FE and he is using SOLE approaches in his university courses.

Notes from JISC Digital Festival (Day 1)

I attended the JISC Digital Festival in Birmingham, UK.  This blog post is a set of notes from Day 1 (11-March-2014).

Keynote – Diana Oblinger (CEO Educause)

“Why are we still talking about this digital stuff?”

It is about 25 years since we moved from the analogue world to the world of society, education and work being based on digital computing technologies. But we still use the term “digital” because either we see it as something special or we get concerned about “man vs machine”.  It is not just about digital – it is about demographics. In US only 17% of learners are traditional college students. Many are now studying as adults who may not have previously had a positive educational experience.  This changes what we need from education and “how” we deliver that education.

Engagement

When you are engaged you learn better – leading to the hypothesis that face-to-face is always the best solution. But face-to-face often just presents text on a screen (a board or projector screen). However, digital technologies allow greater interaction between students and between them and data about what they are  investigating. We want to promote deep learning and develop skills through practice.  In these online practice environments not only is there student activity but there is data coming back from the students’ interaction. When have massive amounts of data can begin to realise the long-held goal of personalised learning.  This can lead to adaptive learning systems.

Student Empowerment

There are different types of students.  Two example profiles:

  1. ROI Skeptics – not sure education will be worth the effort; external barriers; lack of vision; juggling work and family etc.
  2. Highly motivated students who always expected a college (university) education

Students need help with their complicated lives to be able them to give education the right priority. Case management – dealing with the student holistically; early alert programmes (for e.g.) have significant and lasting impact. Some students are blissfully unaware that they might be at risk of failing the course. Here lies a potential for learning analytics. However, beyond that teachers need support on how to deliver those messages to the student, e.g. mobile phone text message? (BTW 43 words seems to be about the right length of message.)

Too much choice can be the enemy of student success if they choose courses they are not prepared for or at too high a level for where they are in their studies.  (C.F. consumer choice problems).  Software tools are emerging to address.  Example shown of a tool for students and their advisers as they seek to support them and their choices.

Alternative Models

IT in education can spawn over complexity and disorientation.  Interconnected elements:

  • Mission
  • Market
  • Margin

Part of this is coming from outside our institution – e..g. MOOCs; large-scale commercial educational providers.  Many customers (students) feel they are over-served by the traditional university system. Now a big puss on competency based education – prove you have particular skills irrespective of where you acquired them.  However, current IT systems just focus internally to the educational establishment.

Time is an opportunity cost.  Example “Direct to Degree” from University of Kentucky, enabling students to rapidly acquire a degree and reach their employment goals. If students never go to campus where do you provide their support? Another example from College for America that provides the support in the workplace. Can be low-cost models – e.g. 1 student completed degree in 3 month at cost of $1,350.

There is lots yet to do in the digital environment.  Need to design from the digital and with man & machine not man vs machine in mind.  A great frontier for all of us.

Three questions:

  1. What does it take to exceed expectations in this digital world?
  2. What capabilities (personnel, budget, skill) are required to deliver the value from IT?
  3. How do we optimise for a digital future??? – Answer yet to be told!
Steps and clap board Image credit: http://www.livebinders.com/play/play/759398

Response and Reflections on keynote

Prof. Paul Curren VC at City University

Was in a situation of ailing IT and with many mismatch between demand and supply issues.  (Here the focus is on education not research or admin. but had to address all three). How to develop a personalised experience for the students and enable academic staff to give the support they want to provide?  Further, how do we enhance the educational experience? How can we do this in the context engendering a community experience. City has multiple campuses now.

Trying to achieve a clear vision (2016) of where wanted the university to be.   Investing in academic staff, IT and estate.  Focused on having sector leading IT areas in education.  In 2010/11 large IT service, 142 IT staff, £ 14.6 million budget.  But, very devolved, often software developed in-house and not fully documented and big problems occurred when staff move on.

Formulated a strategic plan with projects each under a PVC.

  • Engaging IT services around the student (e.g. brought in Moodle, Office 365) and organised around the concept of “The One City”.
  • Sourced commodity from external suppliers
  • Now spending less on IT, less staff but more junior staff at the coal-face and less IT managers
  • Standard high quality equipment in the student spaces
  • Listened to what students, staff and professional services members wanted
  • Moved core services to a central base in London

Where are we now?:

  • The student registration system now stabilised
  • Monitor student access to Moodle to check on student engagement (reduced drop out rates)
  • Early initiation to ensure things were scalable and multi-platform (including mobile to enable work while travel)
  • Provided easy access to student records
  • Using the big providers (e.g. SAP) linked to increasing in-house skills
  • IT staff now viewed differently – previously seen around their skill base (e.g. UNIX) – now seen around their relationship management and knowledge of integrating systems

Challenges for staff:

  • Academic staff need to move seamlessly between a “digital” and “real” world – provided a lot of training
  • IT staff – understanding their new role around the user/systems integration – again investment in training
  • Outsourcing and agreeing the boundaries

Conclusions

  • On a journey
  • Downtime reduced
  • Student Satisfaction Scores with IT increased

Martyn’s reflections

The key points that stood out for me from these two linked presentations were:

  1. We need to accept digital as being here and now and move to a “Man & Machine” mindset and not a “Man vs Machine” one
  2. Systems integration is key and outsourced systems are often the cost-effective way to go
  3. IT support staff need to see their role as focussed on the users (students, academics, admin staff, management) not on a particular area of technology that forms the core of their skill set
  4. It’s a ‘win-win’ situation of better services at lower cost that is achievable this way

A personal reflection from me:

The IT systems should be the servants of the educational, support and management processes not the other way round!

horzizonal line indicating end of blog post

 

Ethics, Learning Analytics and Disability

Today I have been writing a contribution for a paper requested by the Open University’s Ethics Committee about ethics in Learning Analytics.  This blog post is adapted from that.

There are two broad use case scenarios where learning analytics approaches may benefit disabled students:

  1. Targeting support to disabled students or their tutors (Support)
  2. Identifying online activities that seem to be problematic for some disabled students (Accessibility)

As far as we are aware these approaches are yet to be deployed anywhere world-wide but we are actively researching them here at the Open University where we have approximately 20,000 disabled students.  We envisage that if the early promise of this research holds up, deployment on about a 3 year horizon.  These approaches, especially the accessibility one, are reported in more detail in Section 5. of Cooper et. al. 2012.

Firstly, a few definitions:

IMS Global Learning Consortium offered education-specific definitions of both disability and accessibility when introducing its work on the development of technical standards for accessibility in e-learning:

[…] the term disability has been re-defined as a mismatch between the needs of the learner and the education offered. It is therefore not a personal trait but an artifact of the relationship between the learner and the learning environment or education delivery. Accessibility, given this re-definition, is the ability of the learning environment to adjust to the needs of all learners. Accessibility is determined by the flexibility of the education environment (with respect to presentation, control methods, access modality, and learner supports) and the availability of adequate alternative-but-equivalent content and activities. The needs and preferences of a user may arise from the context or environment the user is in, the tools available (e.g., mobile devices, assistive technologies such as Braille devices, voice recognition systems, or alternative keyboards, etc.), their background, or a disability in the traditional sense. Accessible systems adjust the user interface of the learning environment, locate needed resources and adjust the properties of the resources to match the needs and preferences of the user. (IMS Global 2004)

Thus disability is not an attribute of a person, but an attribute of the relationship between that person and the tools they are using to meet their goals; in this case online learning.  And, accessibility is a property of the learning resources that makes is usable by all, including those traditionally labelled as disabled.

The principle ethical dilemma when approaching learning analytics and learners who might experience a disability in the context of online learning is:

  • For what purpose has the individual students declared their disability to the university or other educational establishment, and is this consistent with how that information is to be used in the learning analytics approaches?

No other literature has been found explicitly addressing this issue.  So this blog post might represent the first public statement of the problem.

At the Open University students who declare a disability so that they can be provided with support in their studies.  This is consistent with the first use case scenario (Support).  It is a moot point if it is consistent with the second use case scenario (Accessibility).  More critically at this stage of development of these approaches it is not obvious that it is consistent with research into these approaches.  Is it ethical to use historic or current data relating to students with disabilities to undertake research into future approaches of applying learning analytics?

References

Cooper, M,Sloan, D., Kelly, B.,  and Laithwaite, S. (2012) A Challenge to Web Accessibility Metrics and Guidelines: Putting People and Processes First, Proc. W4A2012, April 16-17, 2012, Lyon, France. Co-Located with the 21st International World Wide Web Conference.

IMS Global Learning Consortium (2004), IMS AccessForAll Meta-data Overview. Available online at: http://www.imsglobal.org/accessibility/accmdv1p0/imsaccmd_oviewv1p0.html (accessed 17/02/14)

Perspectives on the student as e-learner

Introduction

Today I attended an internal Open University (OU) seminar which I had been much looking forward to.  This blog post is my summary notes from that seminar and some personal reflections on it.  A much repeated “mantra” of mine over the last 10 years has been:

  • We don’t know enough about the experience of our students studying online

My particular interest is the experience of our approximately 20,000 disabled students.  However, in many ways, within the diversity of this cohort and that of the overall student population, their experience is not distinct in most aspects. Putting that more simply, disabled students are people first exhibiting the full range of human diversity in their dislikes and preferences.  For example among all students, and among those declaring disabilities, there are significant proportions that do not like to read long texts on screen.

So I approached this seminar as a way of gaining insights of the student experience from three colleagues who have differing roles.  My hope was that this would provide enriched context for my work.

Abstract

The seminar was part of an Arts and Social Sciences e-Learning Seminar Series. The abstract published in advance is quoted here:

Three presenters will offer different perspectives on students as e-learners. Megan Doolittle, Staff Tutor and Senior Lecturer in Social Policy and Criminology, will offer a module team member’s view based on DD206, The uses of social science. Richard Greaves, Associate Lecturer in the East of England who teaches Arts modules at all three undergraduate levels, will offer the tutor’s perspective on students as e-learners. And Robin Goodfellow, Senior Lecturer in New Technology in Teaching (IET), will examine the way that the faculties’ data wranglers use and interpret information about students.

Discussion around these presentations will focus on the question ‘what does this triangulation tell us about our students?’

Resources

The presenters’ web pages are listed here:

The PowerPoint Presentations and a video recording of the session will be put on line in a day or so and I will post the links to them here then.  These may or may not be accessible to OU staff only.

A note on OU roles (OU colleagues please skip)

Here are a few notes to set the rest of this blog in context for any readers not familiar with The Open University.  They give a simplified picture. For more information see: http://www.open.ac.uk/about/main/

  • An undergraduate qualification with the OU is made up of studying a series of Modules.
  • The Open Degree is made up from 300 credit points, the equivalent of 5 years of 60 credit point modules. The BA (Honours) or BSc (Honours) adds a further 60 credit points.
  • There are requirements for the number of points that must be achieved at each of 3 Levels (1, 2 or 3), that approximately correspond the levels of study you might experience in a conventional British University in 1st, 2nd and final years of a bachelors degree.
  • A typical module is worth 60 credit points (although there are 30, 15 and 10 point modules too).
  • A 60 point module requires 16 hours of study each week over 9 months; equivalent to about three evenings a week and a full day each weekend. (Many students “laugh” at the university’s study time estimates.)
  • Modules are developed in Module Teams each of which has a Module Team Chair (the role represented by Megan Doolittle at this seminar).  The teams are multidisciplinary made up of several, sometimes many, academics from the host faculty, Curriculum Managers (who have a co-ordination and project management role) and other specialists.
  • Learning and Teaching Solutions (LTS) is the centre for the development, production and delivery of creative and cost-effective distance learning materials (both printed and electronic). Specialists in the production of the various media elements, work closely with each Module Team.  They also closely work with IET.
  • Modules are essentially delivered by people the OU calls Associate Lecturers which are tutors.  Richard Greaves represented this role in this seminar.  Students on any given module are split into tutor groups which are assigned one or two tutors.  Traditionally these tutor groups would typically have the opportunity to meet together face-to-face about once a month but the tutor group interaction has moved increasingly online in recent years.
  • The OU makes extensive use of a Virtual Learning Environment (VLE) – based on Moodle[1] and other software tools including online forums, audio conferencing and remote presentation software.  Each module has a presence here and some are presented wholly in this way. Although an approach integrating this with printed materials is more common and sometimes face-to-face activities like summer schools and field trips are included.
  • The Institute of Educational Technology (IET) (where I work) supports all the other faculties and the university’s management in making pedagogically sound use of technology in its teaching and learning.  It does this based on internationally renowned research in this field.  It has many roles but Robin Goodfellow represented in this seminar that of a learning analytics advisor to a faculty.  This is an ongoing role communicating data on student performance and attainment and assisting with its interpretation to the Module Teams and faculty management.  For decades this role has gone on in different guises.  Recently it has acquired the sexy title of “Data Wrangler” and the terminology of learner analytics has come to the fore. [Any apparent cynicism there is all mine.]

Notes taken during the seminar

Megan Doolittle – From the module teams perspective

  • Module Team Chair for  DD206 “The uses of social science”, see: http://www3.open.ac.uk/study/undergraduate/course/dd206.htm
    • 2nd Level, 60 point, more online than past social-science modules
    • 1000+ students first presentation, 84% completed, 95% of those passed
    • Students liked:
      • Integrated assessment strategy
      • The relevance of the online elements
  • Students did not like:
    • Extent of online materials
    • Inadequacies in some software
  • Collaborative Groupwork
    • Students in small groups asked to  find and assessed evidence about a current debate (e.g. the planned UK High speed train line HS2)
    • Assessed (participation in groupwork always higher if assessed)
      • But assessed individually and participation accounted for 15% of grade
    • Tasks for each group (set up by module team – facilitated by tutors)
    • Forum discussions main method of collaboration
    • Issues for offender learners (prisoners) and those that can’t access forums
  • How well did it work?
    • Anecdotally – almost all students participated (at least a bit)
    • Sample group (8 students)
      • 84 posts across a number of threads (given as a hand-out)
      • A larger group of 11 students made 104 posts
    • Whole tutor group considered too big (based on knowledge of face-to-face teaching)
    • Series of lead in activities available not assessed but participation in these much lower
    • Sample exchange shows:
      • Common group dynamic patterns
      • Some posting with no comment
      • Students took task seriously
      • No complaints from tutors or students (although a few said won’t do)
      • Often one student took the lead
      • Most cases kept to focus of thread well
      • They didn’t always stick to roles initially proposed
      • Resources were shared
      • Some evaluation of resources, even if superficial (more like that which is common on social network discussions like Facebook or Twitter)
      • Key discussion points were raised  – sometimes backed up with evidence
      • It was not easy to move towards into more in-depth discussion of the issues and evidence
      • Most groups self-organising
      • Tutors tried various strategies to manage the forums – e.g. e-mail prompts
  • Technical issues
    • Forums are “clunky” at OU end to set up – the tasks had to be simpler than had been envisaged in planning stages because of technical limitations
    • Support from LTS, Module team and tutors needed in presentation
  • Feedback from students
    • (Similar to the feedback you would see in face-to-face group work)
    • Complained about lack of participation from other students
    • Difficulties of those working at different paces or different schedules
    • Not a lot of specific feedback from students just a few grumbles on forums
  • 2/3s teaching materials written for online environment
    • Limited blocks of text – lot of audio and video
    • Enabled a real blend of skills and content to be taught simultaneously
    • Took staff long time to learn to write effective resources (had to see it online to figure if it worked then iterate)
    • Some things can be best taught in book – e.g. abstract concepts
    • Tension between writing styles used online and models of good academic writing for students – referencing an important tool for this (how reference VLE materials?) – Still don’t ask students to write for online environment but rather essays and reports.
  • Student Experience
    • Students  could work though module from beginning to end – the tool has a strong linear push
    • The online annotate tool is currently very poor
    • Harder to dip in and out than print
    • Some students did not want to work on screen; others preferred it – most just got on with it.
Question
  • Issue of the 15% participation mark and offender learners and others who could not get online?
    • Answer – It’s problematic

Richard Greaves – From the AL (tutor) perspective

  • Provision on different modules:
    • AA100 – face to face and cluster forum
    • A150 – face to face and Tutor Group forum
    • A230 Face – to face and cluster forum  and national Forum
  • Module Websites
    • All provided some information
    • Newer modules more online
    • AA100 – “Inside Arts” Works well for nearly all students who access it
    • More problematic for weaker students – need tutor support
      • Feeling about more confident students posts: “I don’t know how they got to that answer I must be thick”
  • Students inexperienced with IT find it confusing
  • Students unwilling to spend time learning to navigate the website (e.g. when encounter different layouts)
  • Important because
    • Can lead to student not accessing materials
    • Student – Tutor Interactions
  • E-contact via forums
  • Tutor “business posts”
  • If tutor gives answer on forum rather that e-mail available to all on forum
  • Tutor posts indication additional materials e.g. relevant TV programmes
  • Online Tutorials
    • N.B.  long posts on forums do not get read
    • Elicit student engagement via discussion and questions
    • Students need an individual response from tutors not summary answers
  • Types of student:
  1. Confident and Intelligent posters
  2. Weak – showing little understanding but still participating
  3. Readers – Who do not post (lurkers is the term I would use c.f. social media)
  4. Non-engagers
  5. Students who see online activity as part of their study
  6. Students see all OU activity as “work” do not see tutorials and forums as support
  7. Students who want to work on their own – they may be bright and work effectively
  8. Students who are weak and struggling but will not engage because of this – student type 1. puts this student off (may prefer to use a Facebook[2] groups – these are sometime a source of disinformation)
  • Element of chance in the mix of students types in any given tutor group
  • What does student resistance look like?
    • I haven’t got time
    • I feel intimidated prefer Facebook
    • I only want peer support Facebook is better for that
  • What does AL (tutor) want?
    • Students boosting understanding and skills
    • Student engaging with forum based tutorials
    • Students able to access peer-support
    • Don’t want – discussion posts that are literally interpretations from other sources copied from sources found by google search
  • Compulsory online activity
    • A150 Wiki assignment
    • Requires online group activity
    • Successful in itself but didn’t (in his case) carry over to later course tasks
    • Students are increasingly instrumental – only engaging when marks involved

Robin Goodfellow – From the IET “Data Wranglers perspective
or “Where is the student in all these numbers?”

I have not noted this section live.  This is partly because I ran out of steam for near verbatim note taking and because of the nature of the presentation included lots of data tables and graphs. Please refer to the slides that will be linked to under resources above as soon as they are available.

Key points:

  • What assumptions are we making about the students when looking at data from VLE?
  • Great graphic illustrating how no correlation between KPI1 (Satisfaction > 87%) and Pass rates.  SEaM[3] survey goes out after final Tutor Marked Assignment (TMA) before the end of module Exam.
  • Leaning design (OU version) structured in 4 categories:
  1. – Guidance and Support
  2. – Communication and Collaboration
  3. – Content and Experience
  4. – Reflection and Demonstration
  • VLE tracking Data:
    • % of students that access VLE in each week
    • Average time (mins) students spend online in each week
    • Forum access – % of students in each week (module teams can specify which forums tracked)
    • Resource bank accessing per week (but don’t know which students are which – so might be different students in adjacent week)
    • What are the students saying?
      • Online collaborative activates usually get low satisfaction levels.

Small group round table exercise:

A set of various data sheets and graphs were given out to each table of about 6 people.  These related to anonymised courses, 3 from Arts and 3 from Social Science, at Levels 1, 2 and 3.  Each table was asked to review the data presentations and:

  • Say what does data show?
  • Identify 2 or more modules  that appear to suggest  different learning experiences
  • Prepare a short  summary  of what the student  on each of the modules is “likely” or “unlikely” to do or experience

Discussion

Time was cut short for discussion because of the inevitable over-running of the three excellent main presentations.  However here are some of the key points that were raised in the short discussion period:

  • Lots of points on the degree of confidence in the learning analytics.
  • What issues are specific to e-learning, many are the same as face-to-face, then how can this be used to improve e-learning?
  • Satisfaction and pass rates a really dangerous indicator! – Difference between learning and passing.
  • There is a danger of reducing the incentive to teach hard things.  This is because of criteria being used to evaluate how good a module is, is so reductive.
  • Does the data analytic view reflect anything? – Is it not just a process measure?

Robin – if we reject the data approaches that are beginning to pervade the university because they reductive we are absenting ourselves from the discussion and steering towards good practice in their use.

“Know thy data!”

Personal reflections

I am not going to make extensive reflections in this blog post.  I have a meeting with Robin Goodfellow booked for the 30th Jan 2014 about the overlap between our respective work, interest and reservations about learning analytics.  I hope to make a further blog post after that which this seminar will contextualize.  However here are a few thoughts:

  1. It was a shame that there was insufficient opportunity to triangulate the 3 perspectives given in the seminar.  This is vital because of the scale that the OU operates at and because unlike most courses at face-to-face universities it is not the authors of the modules that deliver them.
  • Module teams need to know the tutor experience
  • Module teams need to know the students experience as understood by the tutors and the student surveys and other learning analytics approaches
  • Those collecting and interpreting the data about the student experience need to validate their data approaches against the students, tutors and module teams “stories” (I hope to expand on this in the next blog post)
  1. Robin’s concluding comment of “know thy data” was a fitting one.  There was quite a bit of anxiety expressed, as has been widely recognised in other learning analytics work, that the learning analytics was more about management evaluation of the academic staff and their modules than enhancing student support and for continual improvement of module design.  I can understand that anxiety but I agree with Robin that the way to ensure a positive use of the data in improving teaching and learning is to understand the data, what is means and importantly what is does not mean; its limitations.
  2. Unlike many of my colleagues in IET working on learning analytics, I come from an engineering (cybernetics) background and not a social science one.  So, some of my thinking about this data comes from a measurement physics perspective that was part of my education and work as an engineer for about 10 years.  Key questions for the learning analytics approaches from this perspective include:
  • What are you actually measuring?
  • How does what you are measuring relate to what you really want to know?
  • What is your estimate of the error in your measurement?
  • What is the signal-to-noise ratio?
  • When are you approaching the fundamental limits of measurement for the approach in question?
  • How might your measurement approach and instrumentation be interfering with what you are trying to measure and to what degree?

The issue of instrumentation is important in learning analytics.  It is very easy to produce pretty graphs and dashboards that look like they are saying something important and useful, but without understanding the measurement approaches, data and metrics and the limitation of these it is too easy to convey a false confidence level in our learning analytics.

Repeating Robin’s conclusion: “know thy data!”

Footnotes:


[2] Facebook groups are rarely set up by tutors or Module teams but are frequently set up for peer-to-peer support by students themselves

[3] The SEaM Survey is the OU’s main method for collecting feedback about the study experience and administers by IET. It is:

  • an evaluation which asks students to provide feedback on their tutor and their wider module experience in one combined survey managed jointly by the Student Statistics and Survey Team in the Institute of Educational Technology and the AL Services Team in Student Services.
  • sent to all students who successfully studied and completed modules with tutor support circulated via email with an online link to the survey two or three weeks before the end of the module.

This new approach was first used in January 2013 and the results are proving to be very beneficial.

New paper on planning for professionalism in accessibility

Just published in journal Research in Learning Technology is a paper I am a co-author on entitled:

Adapting online learning resources for all: planning for professionalism in accessibility

This blog post is a bit of shameless self publicity for this paper but is shared because we believe it contains important lessons for those seeking to address accessibility for disabled students especially in Higher Education.  The abstract and link to the full text follow:

Adapting online learning resources for all: planning for professionalism in accessibility

Patrick McAndrew, Robert Farrow and Martyn Cooper

Institute of Educational Technology, The Open University, Milton Keynes, UK

(Received 7 May 2012; final version received 24 October 2012; Published 19 December 2012)

Abstract

Online resources for education offer opportunities for those with disabilities but also raise challenges on how to best adjust resources to accommodate accessibility. Automated reconfiguration could in principle remove the need for expensive and time-consuming discussions about adaptation. On the other hand, human-based systems provide much needed direct support and can help understand options and individual circumstances. A study was carried out within an EU-funded accessibility project at The Open University (OU) in parallel with studies at three other European universities. The study combined focus groups, user-testing, management consultation and student survey data to help understand ways forward for accessibility. The results reinforce a holistic view of accessibility, based on three factors: positioning the university as a positive provider to disabled students; developing processes, systems and services to give personal help; and planning online materials which include alternatives. The development of a model that helps organisations incorporate professionalism in accessibility is described, though challenges remain. For example, a recurrent difficulty in providing adequate self-description of accessibility needs implies that a completely automated solution may not be attainable. A more beneficial focus, therefore, may be to develop systems that support the information flow required by the human “in the loop.”

Keywords: inclusion; students with disabilities; services; personalisation; evaluation; virtual learning environments; EU4ALL

The full text is freely available under a Creative Commons license at: 
http://www.researchinlearningtechnology.net/index.php/rlt/article/view/18699/html

Your comments would be most welcome!