November 2013 Work in Progress Presentation

How can DIT academic staff use Blackboard data and reporting to make better informed decisions around student learning?

Work in Progress Report

 Submitted November 2013

Student Name:                                   Pat Walsh

Project Supervisor:                           Dr. Frances Boylan

Introduction

SoLAR, (Society for Learning Analytics and Research) defines learning analytics as the “measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimising learning and the environments in which it occurs” (SOLAR, 2013, p. 1).

Learning Analytics (LA) offers a lens into student learning and has been identified as one of the “fastest growing areas of research into technology-enhanced learning” (DEHub, 2012, p. 8). It was cited in the 2013 Horizon Report (Johnson et al., 2013, p. 24) as an emerging technology predicted to impact on teaching, learning and research in the next two to three years. In response to the increasing demands for student success and institutional accountability, Higher Education Institutions (HEI) have witnessed  a surge of interest in data mining and learning analytic technology (Campbell & Oblinger, 2007, p.3;DeBlois, Campbell, & Oblinger, 2007, p.41).

The focus of the research is to explore and evaluate the data and reporting features available through Blackboard to determine the possibility of assisting lecturers and students in making better informed decisions relating to student learning. The report summarises work carried out to date and provides an outline for the project plan going forward.

Aims and Objectives

The aims and objectives are summarised below

·         Analyse student engagement with course content and compare with end of module results.

·         Host workshop demonstrating data analysis features of Blackboard and measure effectiveness of same.

·         Evaluate data analysis features of Blackboard e.g. retention center, performance dashboard and module reports.

·         Capture lecturers’ perceptions of learning analytics and their evaluation of the data analysis features of Blackboard.

Literature Review

The adoption of Learning Management Systems (LMS) across education provides HEI with a wealth of data sets that capture learner behaviour in an online environment. In line with this Swan (2012 p.5-6) contends LMS to be on the “frontline of LA” due to its ability to capture and store vast amounts of data in real time. The literature provides numerous examples of studies analysing student data which sheds interesting insight into the behaviour of online students within LMS.

A report published by the Australian Learning and Teaching Council (Dawson & Mc William, 2008, p. 4) identifies positive correlation between student participation in discussion forums and final results. Similar studies identify active site engagement as an effective predictor of course outcomes in LMS (Smith, Lange, & Huston, 2012, p. 60; Beer, Jones, & Clark, 2012, p.82; Dawson, Tan, & McWilliam, 2008; Fritz, 2013, p. 1). It will be interesting to observe what hidden patterns, relationships, interdependencies or trends, if any are revealed during the quantitative phase of research.

Dawson, Heathcote & Poole (2010, p. 7-9) raise a salient point regarding the integration, or lack thereof, of current disparate institutional systems for data mining purposes. The authors assert HEI should not limit data analysis to LMS but rather seek to integrate data from other sources such as student information systems or records of previous academic history, therefore providing academia with more detailed and greater understanding of their student cohort. One of the flagship projects for LA, Purdue University’s Course Signals (Arnold, 2010, p. 2), integrated student data from a range of instructional systems including student information system (SISs), library system and LMS to implement an early warning system. Data integration from other ICT systems is not a consideration for this project in its current state but potentially identifies an area for further research.

The literature highlights a noticeable lack of research investigating the inbuilt reporting and data analysis features of LMS. Given the technical difficulties and limitations encountered in extracting data from LMS via third party tools (Dawson & Mc William, 2008, p. 6) I find it remarkable that more research has not been conducted into the analysis of inbuilt reporting in recent years. The aforesaid report is critical of Blackboard’s analysis features, referring to the reportage of data as “complex and confusing” (Dawson & Mc William, 2008, p. 6 ). This report, citing (Mazza & Dimitrova, 2007) refers to the extraction and reporting of LMS data as being fragmented.  It is important to raise two key points with this assertion, firstly this study was conducted in 2007/2008 and secondly it refers to Blackboard v8.0. The Institute is currently using Blackboard Learn v9.1. Therefore it is reasonable to suggest these features would have advanced in the interim. Tella (2011, p. 72) cites some important factors for measuring Blackboard success such as content, system and service quality. Notably data analysis or reporting features of Blackboard do not feature in this study. I find this remarkable considering the data mining and learning analytics is such a relevant issue in education today

Another significant report uncovered during the literature review was the ECAR 2013 report. The report revealed intriguing findings, in that in two successive years students reported LMS “most pervasive and valued of technology resources” (Dahlstrom, Walker, & Dziuban, 2013, p. 11). Of the HEI surveyed, 70% view LA as a major priority but only 10% collect system generated data needed for analytics (p. 35). In line with this, (Siemens & Long, 2011, p. 32-33) affirm that HEI have traditionally been inefficient in their use of data. This is further proof that LA is still in its infancy and why it may take two/three years before we see its impact in HEI (Johnson et al., 2013, p. 24)

Despite the promise of LA the literature highlights some concerns.

·         Data privacy is a major concern for LA. At the first Learning and Analytics Knowledge (LAK) Conference in 2011, all attendees present agreed it raises “deep and complex privacy issues” (Brown, 2011, p. 3). Understandably students may perceive LA as an invasion of privacy as the monitoring and tracking of student online engagement raises the “specter of a digital big brother” (Norris, 2011, p. 2), while Diaz & Brown argue that LA is “tantamount to snooping” (2012, p. 8) . Despite this (Arnold, 2010, p. 8) claims during the Course Signals Project five years of research not a single student asked about how their privacy is protected. I find this claim startling given the issue of data privacy prevalent in the literature. One particular facet of the ECAR 2013 report that resonates with me is the discussion regarding students’ lukewarm attitude to LA. Dahlstrom, Walker, & Dziuban address student’s concerns regarding data privacy, suggesting that if we fail to approach LA in an open, transparent and thoughtful manner then students’ data privacy concerns will transcend any benefits derived from LA projects (2013, p. 32-36) This discussion facilitated the submission of documentation to Research Ethics Committee. The study will ensure the privacy of participants is protected, data anonymised and the study strictly adheres to data privacy and ethical guidelines.

·         Students and instructors may feel that LA “takes assessment out of the realm of human judgment and reduces it to numbers and statistics” (Norris, 2011, p. 2). The readings uncovered similar findings with Campbell & Oblinger’s (2007, p. 16) assertion that L.A projects may be perceived as “dehumanising the educational process”. My thoughts on LA closely align that of  (Brown & Diaz, 2012, p. 3; Fritz, 2013, p. 9), who argue that the role of LA is to support decision making not supplant it. It should not be used by academics to abrogate their responsibilities in decision making  as educators.

Methods

The study will adopt a sequential mixed method approach with a qualitative follow-up phase building on the initial quantitative phase (Creswell, 2009, p. 211). 

The quantitative phase will involve the measurement and analysis of data gleaned from Blackboard reporting to identify key statistical information regarding student activity. Tracking of LMS data variables will include login times, grades, frequency of logins, user activity in forums and discussion boards, etc.

Interviews to be conducted with staff participants will provide the qualitative data for this study. It is anticipated interviews will be carried out in March 2014. They will follow a semi- structured approach with the objective of receiving feedback from participants on their evaluation of Blackboard data analysis features.

Criteria for selection of staff participants will be based on frequent users of the Blackboard system. The snowball/referral sampling technique will be utilised to identify staff. It is proposed staff participants will be confirmed before mid December 2013.

Resource

Staff participants are required to attend a workshop hosted by the researcher. The workshop will demonstrate the data analysis features of Blackboard namely module reports, performance dashboard and retention center. No real student data will be used for this workshop. To facilitate the workshop, the Learning Teaching and Training Centre in D.I.T have created a test module and test accounts within Blackboard. Lecturers will be requested to monitor and where possible utilise the retention center in their respective modules in advance of the interview phase to provide feedback for the qualitative study.

ePortfolio

After much deliberation, the decision was taken to migrate data from Mahara to Yola .This alternative is deemed to offer more flexibility and ease of use. Yola will serve as the main hosting website for the ePortfolio. All data will be migrated across to this platform by January 2014.

Limitations

The focus of the study is confined to Blackboard and therefore in isolation of other ICT systems such as student information system or enrolment system. Although a limitation, it highlights the possibility for future study. Leveraging data captured from other ICT systems would construct a more detailed picture of the student learning experience.

The study does not capture face to face dialogue between lecturer and student nor does it capture student engagement with social media tools that reside outside Blackboard. Previous studies have avoided Blackboard’s inability to capture face to face query issues by focusing solely on online students (Beer, Clark, & Jones, 2010, p. 79). This is not an option as it would eliminate too many potential participants.  To avoid the issue of not capturing data within other social media tools that reside outside LMS, the study will be targeted at lecturers using web 2.0 tools within Blackboard.

Ethics

Following a second submission to the DIT Research Ethics Committee, ethical approval was granted on 1/11/13.

Some of the agreements outlined as part of the submission are summarised as follows

·         The course lecturers will authorise the researcher with instructor access to their respective modules for a period of 24 hours. This period will enable the researcher to run reports for each of the modules involved in the study. The reports will only include staff and students who provide written consent to partake in the study.

·         All student and instructor data will be anonymised when the reports are run within Blackboard. Interviews conducted with staff will be anonymised.

·         Data will be encrypted using DIT Mc Afee Endpoint Encryption Software.

·         No student or academic staff member will be receiving payment for this study.

·         Participation in the study is voluntary and participants are free to withdraw at any time, without giving a reason for withdrawing and without affecting their future relationship with D.I.T. 

Target Journals

Submission of journal article will be targeted at MERLOT Journal of Online Learning and Teaching and the Journal of Information Technology Education

These journals are peer reviewed, deal with emerging technologies in online education and fulfil the criteria of 5000-7000 words and APA style referencing.

The journal article will in the first instance be submitted to the Journal of Information Technology Education as authors are notified within one week if the submission advances to the first round of reviews.

Project timetable

Below is an outline of the project timescale, entries and date may be subject to change.

References

Arnold, K. (2010, March 3). Signals: Applying Academic Analytics. Educause .

Beer, C., Clark, K., & Jones, D. (2010). Indicators of engagement. ASCILITE, (pp. 75-86). Sydney.

Brown, M. (2013). LEARNING ANALYTICS: THE COMING THIRD WAVE. EDUCAUSE Learning Initiative (ELI) , 1-4.

Brown, M. (2012, July 24). Navigating the waters of learning analytics. Retrieved from Educause Learning Initiative. http://www.educause.edu/blogs/mbbrown/navigating-waters-learning-analytics

Campbell, J. P., DeBlois, P. B., & Oblinger, D. G. (2007). Academic Analytics A New Tool for A New Era. Educause.

Campbell, J. P., & Oblinger, D. G. (2007, October). Academic Analytics.  Educause.1-24

Chalex, J., Essa, A. H., & Norris, D. M. (2012, April 12th). ELI Focus Session The LMS Perspective. (M. Brown, Interviewer) ELI (Educase Learning Initiative).

Creswell, J. W. (2009). Research design: Qualitative quantitative, and mixed methods approaches (3rd ed.). Thousand Oaks, CA: SAGE.

Dawson, S., Heathcote, L., & Poole, G. (2010). Harnessing ICT potential The adoption and analysis of ICT systems for enhancing the student learning experience. International Journal of Educational Management , 116-128.

Dawson, S., & McWilliam, E. (2008). Investigating the application of IT generated data as an indicator of learning and teaching performance in higher education. Australian Learning and Teaching Council,1-45.

Dawson, S., Mc William, E., & Tan, J. P.-L. (2008). Teaching Smarter: How mining ICT data can inform and improve learning and teaching practice. In Hello! Where are you in the landscape of educational technology? (p. 222). Melbourne: ASCILITE.

DEHub. (2012, May). Learning analytics: following the trail of evidence in digitised education. (11, Ed.) DEQuarterly-Winter-2012-Edition-No-11 , 8-10.

Diaz, V., & Brown, M. (2012). Learning Analytics A Report on the ELI Focus Session. EDUCAUSE Learning Initiative.

Fritz, J. (2013). USING ANALYTICS AT UMBC: ENCOURAGING STUDENT RESPONSIBILITY AND IDENTIFYING EFFECTIVE COURSE DESIGNS. EDUCAUSE Center for Applied Research , 1-11.

Griffiths, M., & Graham, C. (2009). Patterns of User Activity in the Different Features of the Blackboard CMSacross All Courses for an Academic Year at Brigham Young University. MERLOT Journal of Online Learning and Teaching , 5 (2), 285-291.

Heirdsfield, A., Walker, S., Tambyah, M., & Beutel, D. (2011). Blackboard As An Online Learning Environment:What Do Teacher Education Students And Staff Think? Australian Journal of Teacher Education , 36 (7), 1-16.

Johnson, L., Adams Becker, S., Cummins, M., Estrada, V., Freeman, A., & Ludgate, H. (2013). Horizon Report 2013.

Lonn, S., Krumm, A. E., Waddington, R. J., & Teasley, S. D. (2012). Bridging the gap from knowledge to action: putting analytics in the hands of academic advisors. Proceedings of the 2nd International Conference on Learning Analytics and Knowledge, (pp. 184-187). New York.

Mazza, R., & Dimitrova, V. (2007). CourseVis A graphical student monitoring tool for supporting instructors in web-based distance courses. International Journal of Human-Computer Studies , 125-139.

Norris, Donald M. (2011, December 6). 7 THINGS YOU SHOULD KNOW ABOUT FIRST-GENERATION LEARNING ANALYTICS. Educause , 1-2.

Siemens, G., & Long, P. (2011, September). Penetrating the Fog: Analytics in Learning and Education. Educause Review , 31-40.

SOLAR. (2013). Society for Learning Analytics and Research. Retrieved April 2nd, 2013, from Society for Learning Analytics and Research: http://www.solaresearch.org/mission/about/

Swan, K. (2012). Introduction to the Special Issue on Learning Analytics. Journal of Asynchronous Learning Networks , 16 (3), 5-7.

Taylor, L., & McAleese, V. (2012, July 18). Beyond Retention: Using Targeted Analytics to Improve Student Success. Educause Review Online , 1-9.

Tella, A. (2011). Reliability and Factor Analysis of a Blackboard Course Management System Success: A Scale Development and Validation in an Educational Context. Journal of Information Technology Education , 72.