Trying to choose and evaluate a specific technology – as a group!

The learning outcome for this week 4 of H817 was to be able to evaluate a learning innovation. It is now Thursday evening and I have learnt a lot but nothing to do with the intended outcome. However,I do want to get something out of this frustrating experience and the thing that occurs to me is that I would like to break down the activity and try to understand why this went so terribly wrong.
First of all, my own contribution was made too early, was completely wrong and I’m afraid it sent the whole group off into different directions, each equally confusing. I tried to step back through the error and get us back on track but that didn’t work out. And this was the first lesson learnt about collaborative work and is associated to using a forum without editing capability:

Students need to be able to correct their own mistakes

The forum is organised as FIFO (first in first out), by default, this means that anyone starting to read will read from the top down. If they get at all distracted by, for example a link along the way, it is likely they won’t make it to the bottom, even if that is where the correction is. But also that correction might not be placed next to the message that contains the erroneous information.
To be fair, this is particularly a Moodle issue; the majority of other forum systems I have used have allowed the poster to correct and amend their own messages at any time. Moodle is set up on a university wide basis, so even if this module (or any other encouraging student collaboration) wanted something different, it is likely to be quite tricky to change.
But on a broader question, is a forum therefore a useful environment for collaborative work? I have found forums useful as a tutor in order to collaborate with other tutors, but this is because a problem or issue is worked on in one thread only. And this leads to the second lesson:

Breakdown the activity

Various of us tried to break this multi-step activity down; I did, but skipped a step and going backwards didn’t work. Others tried but instead of taking the original activity and starting from the top these attempts were also repeating the whole activity.
What might have worked would have been to break down the steps into different threads from the outset, perhaps drip-feeding them one by one as they became populated with material. The threads could have been something like:
  1. Context & Goal: As a group, you need to decide on the context and the goal you want the technology to support.
  2. Shortlist: suggest items for the management team’s technology shortlist.
  3. Choose a technology: Each (see following lesson for discussion about terminology used in describing an activity) member of the group should choose a technology from the shortlist from point (2) and post the reasons for their choice (to the tutor group forum) here.
    1. At the end of the thread, nominate a group member to collate the suggestions from point (3) in
  4. Reviews: Review all the suggested technologies and why they are being suggested.
  5. Voting: (post the link to the Doodle vote here) Then each member of the group should vote for their preferred technology in
One of the barriers to progression seemed to me to be the way that we also had to wait for contributions from students we didn’t know whether they were lurking, on holiday, just reading everything, too busy, working, lost, looking after family….. and so to the third lesson:

Avoid wording that slows down the process

In point 3 the module team used the word “each”. This assumes that every member of every tutor group is going to a) be active b) take part in a timely manner. It set up the activity for failure from the outset and when I realised what it was requiring I simply dropped out of the discussion. But I also felt that we were being asked to do things in an unnatural order which leads me to the fourth lesson:

Collaborative work needs careful design

If this activity had been designed in a way where one thing flowed naturally on to the next, we might have found it easier to manage and been successful:
  1. Context & Goal: As a group, you need to decide on the context and the goal you want the technology to support.
  2. Shortlist: suggest items for the management team’s technology shortlist.
    1. Include a review
    2. Include a reason for your suggestion linked to the context and goal from point 1.
  1. Voting: 
    1. Nominate a group member to collate the suggestions from point (2) in
    2. (post the link to the Doodle vote here) Then each member of the group should vote for their preferred technology in

As I said, I have stopped working on this activity altogether, apart from the fact that the final selection the group has actually managed to make is one that I have already looked at in detail so feel no need to continue with it, I also have simply run out of time this week. The idea of collaborating on the 2nd part of the week’s activities, to produce a report, fills me with horror and so I will cut my losses and move on.

I feel as though I am letting my fellow students down, that I am letting my tutor down but I don’t feel that I am letting myself down.

Podcast: – how to find out if students are engaged


Beer, C. Clark, K., & Jones, D. (2010). Indicators of engagement. In C.H. Steel, M.J. Keppell, P. Gerbic & S. Housego (Eds.), Curriculum, technology & transformation for an unknown  future. Proceedings ascilite Sydney 2010 (pp.75-86). (accessed 20/11/2012)
Green R (2010). Incite Intentional Learning with Learning Management System Features (accessed 20/11/2012)
Educause (2011) Things you should know about… First-Generation learning analytics
Music: Grapes (2008), I dunno, ccMixter, licensed under


I am Mandy Honeyman, a student on H808,  and in this podcast for Unit 5 – I am going to talk about three aspects of evaluating engagement in online learning from both the perspective of the student and teacher.
The first one is monitoring which is more about systems simply because systems are beautifully set up to do this for us and that will involve looking a systems or systems methodology that provide access to that kind of data. Another thing I am going to look at is engagement from the perspective of how we think about designing activities that might help engage learners learners or will engage us as learners. Finally I want to talk about how ensuring engagement actually  remains the responsibility of the learner and this is where I will talk about keeping a journal or reflective log.
One of the things I have noticed as an OU tutor on the new version of the VLE is that we are able now to monitor access on the forums, this means that I can look very quickly and assess very early on whether students have engaged at all. All I want to see is that they have made one post at this early stage, but the problem in the past is that you had to go through every single thread on the forum and check against a list who has posted anything or not – it was time consuming and tedious. Whereas now, all I have to do is press one button and the information is there in front of me. The impact of that has been , in the course I am currently teaching, that I was able to tell in the first two weeks who had engaged more, who had engaged minimally but most importantly I could tell who had not engaged at all. And I was able to then address those issues really fast, I could speak to the students I was concerned about by email, on the phone and this meant that they felt that they had someone to talk to. Then I was able to efficiently assess whether they would be able to engage at the level necessary to be successful (i.e. simply complete the course). So that was absolutely fantastic. I am very aware that there are a number of systems where this is not available at all, and certainly not possible to apply against a database of a list of students or group of students. This is quite a simple area of learning analytics, and analytics certainly get far more complicated than this, but just this kind of small facility  would certainly be something very high in my list of criteria if I were evaluating different learning or course management systems would be 
Posting to a forum is quite a high-level online activity for a student, I wonder how to create material that is engaging for students that might also serve to assess whether there has been engagement. So I have already covered that a little bit in terms of being able to monitor, but I think there is probably something one can do with the design of activities on a course to monitor engagement as a student is working through material. And I am particularly thinking of that in compulsory education levels because teenagers are notoriously hard to keep focussed. How to you know in an online setting whether students have not only learnt the material  but what they thought of it as they went along, whether they found it interesting or boring. How could we get information from them that can help us to improve the course and course delivery. And I think that is especially important in an online setting, because in a classroom you can look around and see very quickly, usually, whether or not students are listening or bored and distracted, whereas if you are online you have absolutely no idea what the student is doing at the same time as trying to learn, concentrate or focus. It seems that at the moment the only way to discover this is by outcome, so you can only find out whether they have engaged by assessing what they have learnt. I would like to experience evaluation of whether or not they like what they have just experienced. For example, when we have used the elluminate sessions in this course or any other, one of the things that is really helpful is when people engage by clapping, using smilies, ticks, crosses, all of these kind of fast question/response things certainly helps me as a learner to focus on what is happening and if I am speaking, to find out how my audience is responding. So I think this needs to work from both perspectives, tutor and learner, as a learner I want to be helped to remain focussed and interested. This also applies to the length of time learning objects last for and how long a student is expected to remain focussed on something. There is significant amounts of research that have questioned the amount of time that people at different ages can concentrate, this is even more true when working at computers as health and safety issues also come into play.
Finally, I am going to talk about the use of learning journals, reflections and how to assess engagement from the point of view of the learner being responsible for it. How do you make reflective activity prove itself to be so worthwhile to the learner that they are motivated to engage with it. How do you convince that it is a powerful object other than being something that being assessed. I think it is probably an individual process that each learner has to go through for themselves, I don’t know that it can be forced. My own response to the blogging that we were encouraged to do at the outset was once I knew it was no longer for assessment I stopped, and I put my time and energy elsewhere. But of course I will go back now, because as I talk now I am horribly aware that I have been very lax. The reflective log is the proof of engagement and I think many learners would be able to figure out for themselves that this is a highly worthwhile activity. I wonder about how it can be made to be more engaging and more fun for students at other levels than HE? Surely those of us who have chosen to study at Higher levels, understand what we have got ourselves into, the commitment to learning itself and this can be said for any post-compulsory level, whereas those students in compulsory education do not have the same vision. They are, arguably, very focussed on summative assessment, and specifically final results, whether that be coursework or exams. So maybe the link between the activity of making notes and engaging in deeper reflective learning needs to be made earlier in order to improve the quality of the learning, the direct pay off being that they will get better results. Perhaps this would help younger people engage with reflection? This would be joined up thinking, I just don’t know if we are taught at an early enough age, that there is a payoff from reflecting more deeply on what you have learnt and therefore would be able to perform better in examination and assessment.