Author: Mandy Honeyman

Trying to choose and evaluate a specific technology – as a group!

The learning outcome for this week 4 of H817 was to be able to evaluate a learning innovation. It is now Thursday evening and I have learnt a lot but nothing to do with the intended outcome. However,I do want to get something out of this frustrating experience and the thing that occurs to me is that I would like to break down the activity and try to understand why this went so terribly wrong.
First of all, my own contribution was made too early, was completely wrong and I’m afraid it sent the whole group off into different directions, each equally confusing. I tried to step back through the error and get us back on track but that didn’t work out. And this was the first lesson learnt about collaborative work and is associated to using a forum without editing capability:

Students need to be able to correct their own mistakes

The forum is organised as FIFO (first in first out), by default, this means that anyone starting to read will read from the top down. If they get at all distracted by, for example a link along the way, it is likely they won’t make it to the bottom, even if that is where the correction is. But also that correction might not be placed next to the message that contains the erroneous information.
To be fair, this is particularly a Moodle issue; the majority of other forum systems I have used have allowed the poster to correct and amend their own messages at any time. Moodle is set up on a university wide basis, so even if this module (or any other encouraging student collaboration) wanted something different, it is likely to be quite tricky to change.
But on a broader question, is a forum therefore a useful environment for collaborative work? I have found forums useful as a tutor in order to collaborate with other tutors, but this is because a problem or issue is worked on in one thread only. And this leads to the second lesson:

Breakdown the activity

Various of us tried to break this multi-step activity down; I did, but skipped a step and going backwards didn’t work. Others tried but instead of taking the original activity and starting from the top these attempts were also repeating the whole activity.
What might have worked would have been to break down the steps into different threads from the outset, perhaps drip-feeding them one by one as they became populated with material. The threads could have been something like:
  1. Context & Goal: As a group, you need to decide on the context and the goal you want the technology to support.
  2. Shortlist: suggest items for the management team’s technology shortlist.
  3. Choose a technology: Each (see following lesson for discussion about terminology used in describing an activity) member of the group should choose a technology from the shortlist from point (2) and post the reasons for their choice (to the tutor group forum) here.
    1. At the end of the thread, nominate a group member to collate the suggestions from point (3) in doodle.com.
  4. Reviews: Review all the suggested technologies and why they are being suggested.
  5. Voting: (post the link to the Doodle vote here) Then each member of the group should vote for their preferred technology in doodle.com.
One of the barriers to progression seemed to me to be the way that we also had to wait for contributions from students we didn’t know whether they were lurking, on holiday, just reading everything, too busy, working, lost, looking after family….. and so to the third lesson:

Avoid wording that slows down the process

In point 3 the module team used the word “each”. This assumes that every member of every tutor group is going to a) be active b) take part in a timely manner. It set up the activity for failure from the outset and when I realised what it was requiring I simply dropped out of the discussion. But I also felt that we were being asked to do things in an unnatural order which leads me to the fourth lesson:

Collaborative work needs careful design

If this activity had been designed in a way where one thing flowed naturally on to the next, we might have found it easier to manage and been successful:
  1. Context & Goal: As a group, you need to decide on the context and the goal you want the technology to support.
  2. Shortlist: suggest items for the management team’s technology shortlist.
    1. Include a review
    2. Include a reason for your suggestion linked to the context and goal from point 1.
  1. Voting: 
    1. Nominate a group member to collate the suggestions from point (2) in doodle.com.
    2. (post the link to the Doodle vote here) Then each member of the group should vote for their preferred technology in doodle.com.

As I said, I have stopped working on this activity altogether, apart from the fact that the final selection the group has actually managed to make is one that I have already looked at in detail so feel no need to continue with it, I also have simply run out of time this week. The idea of collaborating on the 2nd part of the week’s activities, to produce a report, fills me with horror and so I will cut my losses and move on.

I feel as though I am letting my fellow students down, that I am letting my tutor down but I don’t feel that I am letting myself down.

When I talk about innovation, what do I mean

Innovation in your context

I am going to write expressly from the context of working as an associate lecturer at the Open University. I work on two modules, tutoring three presentations in all. One module seems in itself to be quite innovative, in approach, in its responsiveness to feedback and current knowledge. The other module is not exploring pedagogy of online learning in nearly as much detail.

On the basis of your own experience:
Do you sense that your innovations (as supporters of learning) have been valued, encouraged, supported?

This question assumes that I innovate in the context of my OU work as an associate lecturer and I am not sure that I really do. I think I make small steps and if I announce these to my colleagues then they are often supported. For example, I have made suggestions to students in a recent batch of marking that was noticed by a course team member during monitoring and he is thinking about including those in future presentations. But is my innovative nature, the one I like to think I possess, exploited, encouraged and expressly valued, no I don’t think so. 

On the other hand, I am also taking part in the Higher Education Academy, Open Professional Academic Development initiative (HEA OPAD) ,I believe that the innovations that I explore through that piece of action research is being supported by being assigned a mentor, by being encouraged by my staff tutor and I am offered opportunities to explore my practice via staff CPD events.

I also believe that were I to produce resources, that perhaps were innovative, and share them with my colleagues in my modules, these would be warmly received and that would encourage me to do more. These might be valued on a professional level, through acknowledgement of my efforts rather than on a monetary one.What evidence do you have to support your view?
As I mentioned, I have a mentor assigned to me. I don’t have to pay to make an application to the academy for membership and if I don’t succeed the first time I will get feedback about how to improve my application. The OU has created a supportive environment on an institutional level for anyone wanting to take up this opportunity. 

My staff tutor invited me to lead a staff CPD session about my chosen research topic, which also enabled me to distribute a survey and gather qualitative responses. I was paid for this extra work, so yes, that would indicate on a simple level that it is valued.
From the perspective of your context:
How widespread is innovation in your organisation?
One of the joys of working for the OU is the sense that it is a world leader, innovating and teaching, in the area of online learning. Two reports have been written “Innovating Pedagogy” by OU faculty. This would indicate that the organisation is supporting innovation and as an open report, it is not only supporting its own. 
Are there policies or statements that relate to innovation?
A policy does exist about how data associated with the OU’s open and free educational resources.
If yes, how are they implemented?
These would be implemented at an organisational level, i.e. as modules are prepared to be made open certain aspects of the material are filtered for an open audience.
What implications, if any, does this have for your attitude towards innovation?
None.

Summary:
I think that the organisation I work for is very innovative, and has the reputation, policies and examples to prove it. However, in my role in that organisation I don’t believe that we are institutionally encouraged to be innovative, in fact almost the opposite. But that doesn’t mean that on an individual level we would not get support for any small innovations we undertook ourselves, for example, approaching using OU Live in a way that was not part of published guidelines. Examples of this might be having students moderate the Online Room, or using application share for students to engage in peer support (after OU Live training with their tutor). But these small innovations would be over and above our job specifications and if we committed extra time to them, they would be unlikely to receive any reward other than professional acknowledgement by our own peers.

Podcast: – how to find out if students are engaged

http://mandyhoneyman.podbean.com.


References:

Beer, C. Clark, K., & Jones, D. (2010). Indicators of engagement. In C.H. Steel, M.J. Keppell, P. Gerbic & S. Housego (Eds.), Curriculum, technology & transformation for an unknown  future. Proceedings ascilite Sydney 2010 (pp.75-86). http://ascilite.org.au/conferences/sydney10/procs/Beer-full.pdf (accessed 20/11/2012)
Green R (2010). Incite Intentional Learning with Learning Management System Features http://coggno.com/learning-management-system/incite-intentional-learning-with-learning-management-system-features.html (accessed 20/11/2012)
Educause (2011) Things you should know about… First-Generation learning analytics http://www.educause.edu/eli
Music: Grapes (2008), I dunno, ccMixter, licensed under  http://creativecommons.org/licenses/by-nc-sa/3.0/

Transcript: 

I am Mandy Honeyman, a student on H808,  and in this podcast for Unit 5 – I am going to talk about three aspects of evaluating engagement in online learning from both the perspective of the student and teacher.
The first one is monitoring which is more about systems simply because systems are beautifully set up to do this for us and that will involve looking a systems or systems methodology that provide access to that kind of data. Another thing I am going to look at is engagement from the perspective of how we think about designing activities that might help engage learners learners or will engage us as learners. Finally I want to talk about how ensuring engagement actually  remains the responsibility of the learner and this is where I will talk about keeping a journal or reflective log.
One of the things I have noticed as an OU tutor on the new version of the VLE is that we are able now to monitor access on the forums, this means that I can look very quickly and assess very early on whether students have engaged at all. All I want to see is that they have made one post at this early stage, but the problem in the past is that you had to go through every single thread on the forum and check against a list who has posted anything or not – it was time consuming and tedious. Whereas now, all I have to do is press one button and the information is there in front of me. The impact of that has been , in the course I am currently teaching, that I was able to tell in the first two weeks who had engaged more, who had engaged minimally but most importantly I could tell who had not engaged at all. And I was able to then address those issues really fast, I could speak to the students I was concerned about by email, on the phone and this meant that they felt that they had someone to talk to. Then I was able to efficiently assess whether they would be able to engage at the level necessary to be successful (i.e. simply complete the course). So that was absolutely fantastic. I am very aware that there are a number of systems where this is not available at all, and certainly not possible to apply against a database of a list of students or group of students. This is quite a simple area of learning analytics, and analytics certainly get far more complicated than this, but just this kind of small facility  would certainly be something very high in my list of criteria if I were evaluating different learning or course management systems would be 
Posting to a forum is quite a high-level online activity for a student, I wonder how to create material that is engaging for students that might also serve to assess whether there has been engagement. So I have already covered that a little bit in terms of being able to monitor, but I think there is probably something one can do with the design of activities on a course to monitor engagement as a student is working through material. And I am particularly thinking of that in compulsory education levels because teenagers are notoriously hard to keep focussed. How to you know in an online setting whether students have not only learnt the material  but what they thought of it as they went along, whether they found it interesting or boring. How could we get information from them that can help us to improve the course and course delivery. And I think that is especially important in an online setting, because in a classroom you can look around and see very quickly, usually, whether or not students are listening or bored and distracted, whereas if you are online you have absolutely no idea what the student is doing at the same time as trying to learn, concentrate or focus. It seems that at the moment the only way to discover this is by outcome, so you can only find out whether they have engaged by assessing what they have learnt. I would like to experience evaluation of whether or not they like what they have just experienced. For example, when we have used the elluminate sessions in this course or any other, one of the things that is really helpful is when people engage by clapping, using smilies, ticks, crosses, all of these kind of fast question/response things certainly helps me as a learner to focus on what is happening and if I am speaking, to find out how my audience is responding. So I think this needs to work from both perspectives, tutor and learner, as a learner I want to be helped to remain focussed and interested. This also applies to the length of time learning objects last for and how long a student is expected to remain focussed on something. There is significant amounts of research that have questioned the amount of time that people at different ages can concentrate, this is even more true when working at computers as health and safety issues also come into play.
Finally, I am going to talk about the use of learning journals, reflections and how to assess engagement from the point of view of the learner being responsible for it. How do you make reflective activity prove itself to be so worthwhile to the learner that they are motivated to engage with it. How do you convince that it is a powerful object other than being something that being assessed. I think it is probably an individual process that each learner has to go through for themselves, I don’t know that it can be forced. My own response to the blogging that we were encouraged to do at the outset was once I knew it was no longer for assessment I stopped, and I put my time and energy elsewhere. But of course I will go back now, because as I talk now I am horribly aware that I have been very lax. The reflective log is the proof of engagement and I think many learners would be able to figure out for themselves that this is a highly worthwhile activity. I wonder about how it can be made to be more engaging and more fun for students at other levels than HE? Surely those of us who have chosen to study at Higher levels, understand what we have got ourselves into, the commitment to learning itself and this can be said for any post-compulsory level, whereas those students in compulsory education do not have the same vision. They are, arguably, very focussed on summative assessment, and specifically final results, whether that be coursework or exams. So maybe the link between the activity of making notes and engaging in deeper reflective learning needs to be made earlier in order to improve the quality of the learning, the direct pay off being that they will get better results. Perhaps this would help younger people engage with reflection? This would be joined up thinking, I just don’t know if we are taught at an early enough age, that there is a payoff from reflecting more deeply on what you have learnt and therefore would be able to perform better in examination and assessment.