Final Reflection

Year Two Project Reflection

Finding a research area

The 2013 Horizon Report led to the inspiration behind my research proposal. I was intrigued by technologies such as learning analytics, game based learning and internet of things. Coming from an I.T background I saw the potential in these technologies, it simply a question of finding an area within the technologies for research that was feasible, relevant and suitable for my own profession and that could be carried out within the specified time frame.

I developed a growing interest in the area of learning analytics and attended a seminar in DIT entitled “Perspectives on Education Seminars 2013 - focus on retention- Using technology to enhance retention”. The words “using technology to enhance retention” struck a chord with me. Vast quantities of data from academic performance results, progression rates and student surveys were provided at the seminar and it was apparent from the seminar that we as staff members in DIT could do more to harvest and analyse student data to learn about our students. It was about this time that my “eureka” moment occurred – the readings prior to the seminar highlighted a noticeable lack of research exploring the inbuilt reporting features of Learning Management Systems and this, together with the need to analyse student data, identified an area within this field for exploratory study and so my research question was born – How can DIT staff use LMS data and reporting features  to make better decisions around student learning.This marked the first significant milestone in the research study.

Towards the end of the research methods module we were presented with the opportunity to discuss the research proposal with our course co-ordinators and classroom peers. It was clear from the feedback I received that I needed to narrow the scope of the study. It was too broad and unrealistic within the time frame. The feedback helped crystallise my research proposal and facilitated submission and acceptance of it.

First Problem encountered in second year

One of the major obstacles I faced at the commencement of second year was the requirement for ethical approval prior to approaching potential student and staff participants. Given the sensitive nature of analysing student data, obtaining clearance from the Research Ethics Committee (REC) was paramount and became the immediate priority following my first consultation with my project supervisor. Details of this process are available on the ethics section of the e-portfolio page. Awaiting REC approval proved frustrating at times as I was eager to get participants on board and I was concerned by what I perceived to be “red tape” issues. However my concerns were alleviated by discussion with the project supervisor. She advised me to focus my efforts on getting the necessary documentation submitted to the REC and to resume the literature review. The ongoing readings resonated with me; they highlighted the delicate nature of analysing student data. The ECAR report cited students ‘ lukewarm attitudes towards learning analytics while at the first annual Learning and Analytics Knowledge Conference (LAK) all delegates in attendance agreed that it raises deep and complex privacy issues. Ironically, whilst initially frustrated with the REC process, through the readings I developed a greater understanding of the work they are tasked with carrying out.

During the course of the M Sc I developed a real appreciation of the opportunity to present ideas and take feedback on board from the course tutors, supervisor and classroom peers. With this in mind I saw the November work in progress (WIP) presentations and interim WIP report as an ideal opportunity to get a handle on the research and avail of invaluable feedback from my peers. This was the appropriate time to evaluate the research study to date, project plan and academic writing, if there were issues with any aspect now was the time to address them. On the eve of the project presentations I was granted clearance from the REC which marked another significant milestone as I could now officially approach potential participants. The WIP presentation and interim report were reasonably well received though concern was expressed about the decision to migrate my e-portfolio from Mahara to Yola.

Migration of e-Portfolio

Technical issues with Mahara such as customising layout, navigation and problems encountered with embedding digital media content and Web 2.0 technologies led to the decision to migrate from Mahara to Yola platform in early second year.  I had debated the pros and cons of such a move in correspondence with my project supervisor. It was a decision not taken lightly in that it took considerable time to migrate the content across to the new platform. By early December the migration was complete and I now had a platform which provided me with the flexibility and user interface I required with fewer technical problems.

Blackboards New Early Warning System – Retention Center (Development of Resource)

Blackboard’s release of the Retention Center marked an important milestone in 2^nd year.  This new analytic feature represented an improvement on the previous year’s “Early Warning System” as it offered more tailored solutions for lecturers to create their own “at-risk” table. From informal discussion with colleagues in the Institute I sought to gauge lecturers’ awareness of these features and sensed many lecturers were not aware of them; hence the idea of developing a resource consisting of a workshop and a website (http://webcoursesanalytics.wordpress.com) ensued. The pedagogy behind the design and implementation of the workshop was influenced by the paper I wrote as part of the learning theories module. In writing the paper I reflected and evaluated my role as a training provider and concluded that a blend of learning theories, notably behaviourism and social constructivism were the best fit for me in my role as a training provider and in this instance represented the best solution to achieve the learning objectives of the workshop.

Initially the workshop was very structured and instructor led .Participants were provided with clearly defined learning objectives and emphasis was placed on repetition and practice in order for the participants to become knowledgeable with usage of the tools. The second phase of the workshop was much more of a collaborative effort. The focus was very much on what “outcome” the participants wanted from the workshop. This phase facilitated an engaging and stimulating learning environment as we discussed and debated the merits of each report, learnt from each other, identified weaknesses in the reporting and attempted to tailor the workshop to meet the needs of the participant and his/her student cohort.

The feedback on the workshop was very positive. One of the unique aspects of this study was that lecturers were given the opportunity to explore the tools and use the website resource after the workshop and in advance of the interviews, which led to a more detailed evaluation of same. I believe this was a key decision as it provided the participants with time to reflect on the workshop, the tools and assess its impact for their own teaching.  It was particularly pleasing in that all participants used the reporting features in the interim to inform themselves about their student cohorts, particularly to track Blackboard engagement of potential at-risk students, other participants analysed the usage reports to get a feel for what parts of the Blackboard modules were the most popular. The combination of hosting a workshop and adopting a blend of qualitative and quantitative techniques gave a comprehensive view of lecturers’ evaluations and perceptions of Blackboard’s analysis features.

The study was not without its limitations, confining data analysis to LMS does not capture face to face dialogue between lecturer and student, nor does it capture student engagement with social media tools that reside outside LMS. The option to integrate data from different data sources such as Student Information Systems, library and CAO data is not available within Blackboard Learn. Integration of these data sources would provide a richer student profile and whilst a limitation of the study it does highlight areas for future research, possibly through exploring or trialling the offerings of another Blackboard platform namely Blackboard Analytics.

To conclude the last two years represented a challenging but enjoyable learning experience. I believe that the key ingredients to completion of the MSc were consistent work rate, perseverance and engagement with colleagues within the M Sc. My positive experience has whetted the appetite for further studies and rather than this course marking the end of a learning journey, I believe it has triggered a yearning for advancing my professional development. The sense of “camaraderie” among the group made it an enjoyable and rich learning experience; there was a real willingness by all to help each other particularly when providing feedback and using each other as sounding boards. This sense of community, I feel, played a significant role in its completion and will remain one of the enduring memories of the M Sc.

 MSc Critical Reflection

In writing this final reflection I discuss each of the programme learning outcomes in light of my experience in the M Sc. I discuss how my understanding of elearning has evolved and its implications for the final year research project and my own profession.

Demonstrate a thorough understanding of the theory of, and best practices in eLearning in a range of educational contexts

Apply a thorough grounding in theory and practice of eLearning in a range of contexts.

Manage the design, development and implementation and evaluation of a number of appropriate eLearning resources

Demonstrate an awareness and understanding of current eLearning technologies and the challenges and opportunities associated with each.

Design specific elearning applications/resources and evaluate them to determine their value according to specified criteria 

Steen contends that the process of designing effective eLearning is “part art and part science” and identifies adopting an appropriate learning theory as an important characteristic in designing effective eLearning solutions (2008, p 526).The opportunity in the Learning Theories module to critically discuss and evaluate relevant learning theories in light of my own profession provides me with a rich and fascinating insight into my own teaching/training techniques as a technical officer.Developing a knowledge and understanding of the theories of learning has resulted in paying placing closer attention to the teaching methodologies I employ in my role as an IT trainer. These theories also influenced the design and development of the resource I created as part of the final year project.

The design and development of the workshop and website adopted a blend of behaviourist and social constructivist principles. Initially the workshop was very structured and instructor led. Lecturers were provided with clearly defined learning objectives and emphasis was placed on repetition and practice or what Conner refers to as a “drill and practice” approach thus enabling the lecturers to become more knowledgeable with usage of the tools.

The second phase of the workshop was much more of a collaborative effort. The focus was very much on what “outcome” the lecturers sought from the workshop. This phase facilitated an engaging and stimulating learning environment as we discussed and debated the merits of each report, learned from each other, identified weaknesses in the reporting and attempted to tailor the workshop to meet the needs of the participant and his/her student cohort.

A website resource http://webcoursesanalytics.wordpress.com is developed to facilitate the workshop. It is designed as a resource guide to staff who have either (a) already participated in the workshop and want to recap on the lessons learned from it  or (b) staff regarded as "IT savvy”, (they would be able to learn from the videos and user guides provided on the website).

The website incorporates social constructivist principles. Discussion is encouraged via a "blogging feature" to help develop a sense of community and collaboration. Social media tools such as Twitter, Facebook and Google plus are also embedded within the website. In line with Steen (2008), the website incorporates a blend of style, color and video usage that captures lecturers’ attention without distracting from the learning experience. The instructional design models we looked at over the course of the MSc have been really helpful in guiding the development of eLearning content. It is difficult to pick one design model over another. ADDIE, although, very structured provides an excellent framework for creation of eLearning material. I tend to use this design process myself but tweak\extend it to be more flexible and include an evaluation\reflective aspect at each of the first four stages. Rapid prototyping is another interesting technique which I hope to explore in the near future.

The key questions I ask myself before I commence the design, development and implementation of an eLearning resource are what, why, who and how:

(a) What exactly is the resource, what purpose will it serve and in what context? What are the proposed learning objectives? What instructional design method should I consider – AIDDIE, ICARE?

(b) Why am I proposing it - Is there a real requirement for it? Does it already exist? Why is it important?

(c) Who is my target audience?

(d) How am I going to design, develop, implement and evaluate the resource? How will the learning activities align with the learning outcomes?

These are the key questions that guided the development of my applied elearning project and were informed by my experiences in the Learning Theories and Instructional Design module. The applied aspect of the M Sc in being able to I expect they will guide me in the creation of future resources. 

The MSc was a real eye opener to see the wide range of technologies relating to technologies.The experience of using Articulate and Screencast-O-Matic to design eLearning content is something I plan on using again. The interactive features that be incorporated into the eLearning resources using Articulate are excellent and offer a lot of opportunity to create engaging, interactive content. One example of this is available from the one Skydrive modules I created in the ID module. Whilst screen capture software such as Screencast-O-Matic provide a superb means of explaining/demonstrating learning material to a potential end user. I have uploaded a Technology Exploration section on to my eportfolio which discusses some of the other tools I used over the past two years.

The standout tools for me personally are the web 2.0 technologies. Collaborative tools such as blogs, discussion boards and wikis offer enormous potential to create online communities and share knowledge and skills with one another. Interestingly, my experience over the past few years has highlighted the difficulties of trying to apply these collaborative tools in an educational context. In the Instructional Design module, I along with other members of team really engaged with our collaborative blog. It was our “go-to” area for brainstorming, sharing ideas and thoughts. We would struggled to have been as productive without this resource. 

Due the success of the tool I tried to incorporate it into a subsequent module which involved team members from The MA in Higher Education. Despite my obvious enthusiasm, my co members failed to see the potential in it, they much preferred face to face interaction. Promoting eLearning technologies such as the uptake of web 2.0 tools and trying to change the mindset of learners who fail to see the benefits of such tools remains a challenge. A huge part of this is, I feel, down to the individual’s lack of experience in tech tools or technology in general. My role as an IT advisor is to guide these individuals in becoming more skilled with such tools, highlight the benefits that can be gained by using them and encourage them to experiment/trial with them. 

One of the requirements of the research was to capture lecturers’ evaluation of the workshop and/or website. The workshop represented the first time that these analytic tools were explored in any real detail by lecturers involved in the study. The knowledge acquired during the course of the workshop enabled lecturers to make immediate use of the analytic features to monitor student engagement with their respective Blackboard modules. Two of the three interviewees managed to review the website with interviewee1 commenting “It will be of great assistance to all staff intending to use the analytical features” (personal communication, April 16, 2014). Interviewee3 highlighted the need for staff training and asserted the resource and further workshops should be accessible to other staff within the Institute, “I think this definitely shouldn’t go on a shelf... it needs to be integrated into LTTC [Learning, Teaching & Technology Centre] training … possibly as part of a Blackboard module… so that staff can understand its potential” (personal communication, May 14, 2014).

I’m hoping the opportunity to host a workshop at the upcoming eLearning Summer School will raise awareness of the resource website. The immediate goal of this website is to promote the blog and feedback features of the website to (a) improve it (via feedback) and (b) develop sense of collaboration and community.  By working together we can learn of each other, highlight new features that other people may not be aware of, identify problems and share experiences with the main aim of informing our educational practice.

Design a constructively aligned module integrating the appropriate use of eLearning technologies

In describing constructive alignment Biggs & Tang (2007, p7) suggest that we start with what learning outcomes we want students to learn and align planned learning activities and assessment tasks with these outcomes.

This principle was very much at the heart of the resource designed to teach staff participants about the data analytic features of Webcourses.  The resource, consisting of a workshop and website, utilised a number of learning activities with the intention of:

(a) the lecturers in the study becoming proficient with the use of the analytic tools.

(b) achieving the learning outcomes as outlined on the lesson plan.

The learning activities carried out as part of the workshop aligned with these learning objectives. Whilst there wasn't an official assessment element to the workshop, I was able to gauge each of the participant’s proficiency with the analytic tool through observation. The staff participants were encouraged to engage the website resource after the workshop and carry out the learning activities themselves, using the website resource as a guide.

It was crucial that the lecturers carried out these learning activities themselves in order to become adept with the tools.  This was later evidenced during the interview stage when it emerged that the lecturers had continued to use the reporting features of Blackboard to monitor their students engagement with their respective modules.

Conduct critically focused literature reviews relevant to the use of elearning within the selected discipline area

Inspired by the technologies cited in the 2012-2014 Horizon Reports, a comprehensive literature review was conducted into the field of Learning Analytics (LA). Cited in four consecutive Horizon Reports as an emerging technology to impact on teaching, technology and research (Johnson, Smith, Willis, Levine & Haywood, 2011; Cummins, Johnson & Adams, 2012; Johnson, Adams Becker, Cummins, Estrada, Freeman & Ludgate 2013,Johnson, Adams, Estrada, & Freeman, 2014), LA provided an interesting field for further research. To narrow the scope of the study, the research focused on LA in the context of the Learning Management System (LMS) in use in DIT namely Blackboard Learn 9.1.

The literature review provides an overview of the field of LA highlights its importance for education, details of previous data mining exercises, examples of LA initiatives undertaken by other Higher Educational Institutes and identifies issues and concerns raised by previous studies and experts in the field.

The literature review was one of the most rewarding aspects of the M Sc.  It enlightened my understanding of the field of LA. This subsequently informed the methodology section and enabled me to relate my own findings back to those of previous studies outlined in the literature review. The experience of engaging in a detailed literature review has developed my writing and critical thinking skills. I now feel much more equipped to conduct literature reviews in future studies.

Sustain from the research evidence obtained from the undertaking of an e-learning  project, a reasoned argument and draw consistent and coherent conclusion.

The final year research project can be viewed as a LA initiative with the main objective of capturing lecturers’ evaluation of the data and reporting features available within Blackboard Learn 9.1 to determine what, if any, information we can glean from it.  The main themes to emerge from the literature and the interviews are also addressed in the study .They include student data privacy, engaging with “at risk” students, dispersed information sources and lecturers’ evaluation of the workshop and website resource.

The readings highlight a noticeable lack of studies exploring the data and reporting features available within Learning Management Systems (LMS). Given the technical difficulties and limitations cited by Dawson & Mc William (2008, p. 6) in extracting the LMS data via third party tools, I was surprised by the lack of literature exploring the inbuilt reporting features of LMS. Dawson & Mc William referred to Blackboard reporting as “complex and confusing” (2008, p7).

However it is important to highlight two key points with their findings (a) the study was conducted in 2007/2008 and (b) it refers to Blackboard v8.0. The Institute is currently using Blackboard Learn v9.1 and it is reasonable to suggest these features would have advanced in the interim. Furthermore in a study conducted to determine the important factors for measuring Blackboard success, Tella (2011, p72) identifies factors such as content, system and service quality. Notably data analytic or reporting features of Blackboard do not feature in this study.

Given that learning analytics is such a relevant issue in education today I am surprised by the lack of studies exploring the inbuilt reporting features of LMS and feel I have a genuine reason to explore this area further. The results of the research show that analytic features of Blackboard can facilitate lecturers in learning more about their students’ online engagement. They provide academic staff with further knowledge and monitoring capabilities regarding their students’ interactions, or lack thereof, with the Blackboard Learn 9.1. Furthermore, analysis of LMS variables namely login frequency, MCQ scores and hit activity within Blackboard indicated statistically significant positive correlation with student performance. These findings suggest that activity within LMS, measured by logins, hit activity and results in MCQ’s provide indicators of student academic performance. 

Reflect self-critically on the process and outcomes of a development and learning implementation project

Overall the final year project has represented a steep learning curve. I have encountered difficulties along the way, notably seeking Ethics approval, getting staff and students to partake in the study and meeting the project deadlines that I set for myself. Interpreting the correlation analysis in a statistically sound and recognised manner represented a major challenge for me. Initially I struggled to interpret the findings in a manner that would be statistically recognised and accepted , it was crucial to interpret these findings so that they would hold up to scrutiny from academic peers. Fortunately this is where the literature proved invaluable as I was able to cite previous studies and reports which conducted similar analysis techniques. Using a Critical Value Table I was able to determine if the results produced by the correlation analysis occurred by chance or if their value represented statistically significance (Siegle, 2014). The literature review has been a deeply rewarding aspect of the M Sc and something I really immersed myself into. If anything it has whetted the appetite for further study within the field of Learning Analytics as there are so many research opportunities within this area. 

Overall with the learning experience of the M Sc I feel well equipped to engage with research projects and the development of eLearning resources in the future. I have developed the pedagogical and technical skills required to undertaken the design, development, implementation and evaluation of eLearning resources. The process of identifying an area for study, researching it, developing a proposal and then implementing it and seeing it come to fruition is very satisfying. I would like to take this opportunity to thank all the course lecturers for their support and particularly my project supervisor her constant feedback and guidance over the past 10 months.

Engage in research to evaluate the effective use of eLearning resources within a Higher Education environment

An objective of my final year research project was to determine what variables within Blackboard Learn 9.1 are indicative of academic success.Student variables extracted from Blackboard included multiple choice quiz scores, frequency of logins, hit\click activity and overall time spent by students within a module.Correlation analysis of these variables against examination results established a positive correlation between hit activity, login activity and student examination results.

Another eLearning technology used by one of the lecturers participants with his/her student cohort is discussion forums. Two of the modules involved in the study used this feature. The SNAPP (Social Networks Adapting Pedagogical Practice) tool was used to analyse discussion forum activity for two of the student groups involved and identified a total of 286 posts, averaging 6.5 posts per student. A positive correlation was established between discussion forum activity and students examination results. Although the correlation coefficient value was quite low (r.242, n=44), the result suggests a link between discussion forum activity and final grade. 

These findings suggest that activity within Blackboard, measured by logins, discussion forum activity, hit activity and results in MCQ’s can provide indicators of student academic performance.

 Identify instances and conditions where eLearning would be appropriate and evaluate potential and use, within different contexts

Create and evaluate strategies for the effective use of elearning in a range of higher education learning environments

Woodhill (2007) contends that a “lack of a requirement for e-learning” or “lack of identification of a real issue” as shortcomings in the design phase of e-learning resources. Personally this was a major consideration is designing the Webcourses Analytics website. 

This led me to question is there a requirement or demand to build a resource for lecturing staff. Perhaps carrying out a series of workshops with lecturing staff might suffice? Two factors that played an important role in developing the website were: 

1.      Investigation of existing online resources available to assist educators with the analytic tools of Blackboard. I discovered excellent YouTube help videos and user guides on all different aspects of the Blackboard platform but no specific resource that captured the content I needed within the one website.  

2.      Following consultation with the Learning Teaching and Technology Centre (LTTC), I was advised the LTTC do not have any user guides for DIT staff on the “Webcourses User guides” section of their website. If one was created it could be made available to lecturing staff via their website.

Further evidence to support an eLearning resource emerged during the qualitative phase of the research study.

One of the main concerns to emerge from the study is the lack of awareness of the existence of the analytic tools in Blackboard Learn 9.1.  Therefore as part of the final write-up I recommended that further promotion and awareness of the analytics features should be disseminated to lecturing staff in DIT.  In identifying possible avenues to endeavour to promote these analytic tools, I recommend two suggestions namely:

 1.    To host a series of training workshops throughout the academic year with lecturing staff

2.    Host a workshop as part of the DIT eLearning Summer School. I feel this would capture a wider audience and provide an ideal platform to raise awareness of the tools.

At the time of writing this reflective summary I am delighted that LTTC have accepted this recommendation and invited me to host a workshop at this years Dublin eLearning Summer School.I intend to use this opportunity to promote the resource website - http://webcooursesanalytics.wordpress.com.It is envisaged that this website resource will be hosted on the “Webcourses User guides” section of the LTTC website upon successful completion of the M Sc. This I feel is testimony to my ability to identify instances where eLearning would be appropriate and identities strategies for the effective use and dissemination of same. This will enlighten the development of future eLearning resources in which I will be involved.

References

Biggs, J. B., & Tang, C. S. (2007). Teaching for quality learning at university what the student does. (3rd ed.). Maidenhead: McGraw-Hill/Society for Research into Higher Education & Open University Press.

Dawson, S., & McWilliam, E. (2008). Investigating the application of IT generated data as an indicator of learning and teaching performance in higher education. Queensland: Australian Learning and Teaching Council. Retrieved from http://www.olt.gov.au/project-investigating-application-it-qut-2007

Johnson, L., Adams Becker, S., Cummins, M., Estrada, V., Freeman, A., & Ludgate, H. (2013). NMC Horizon Report: 2013 Higher Education Edition. Austin, Texas: The New Media Consortium. Retrieved from http://www.nmc.org/pdf/2013-horizon-report-HE.pdf

Johnson, L., Adams, S. B., Estrada, V., & Freeman, A. (2014). Horizon Report: 2014 Higher Education Edition. Austin, Texas: The New Media Consortium. Retrieved from http://cdn.nmc.org/media/2014-nmc-horizon-report-he-EN-SC.pdf

Johnson, L., Adams, S., & Cummins, M. (2012). The NMC Horizon Report: 2012 Higher Education Edition. Austin, Texas: The New Media Consortium. Retrieved from http://www.nmc.org/publications/horizon-report-2012-higher-ed-edition

Johnson, L., Smith, R., Willis, H., Levine, A., & Haywood, K. (2011). The 2011 Horizon Report. Austin: The New Media Consortium. Retrieved from http://net.educause.edu/ir/library/pdf/hr2011.pdf

Siegle, D. (2014). Critical values of the Pearson Product-Moment correlation coefficient. Retrieved March 1, 2014, from University of Connecticut Neag center for gifted education and talent Development: http://www.gifted.uconn.edu/siegle/research/correlation/corrchrt.htm

Steen, H. L. (2008). Effective eLearning Design. MERLOT Journal of Online Learning and Teaching , 4 (4), 526-532.

Tella, A. (2011). Reliability and Factor Analysis of a Blackboard Course Management System Success:A Scale Development and Validation in an Educational Context. Journal of Information Technology Education , 10, 55-80. Retrieved from http://www.jite.org/documents/Vol10/JITEv10p055-080TELLA897.pdf

Woodill, Gary. (2007). The shortcomings of e-learning 1.0 and the emerging possibilities of e-learning 2.0. Brandon Hall.