How can DIT academic staff use LMS data and reporting to make better informed decisions around student learning?
Participant Name: Patrick Walsh
Student Number: D07112362
Year on Programme: Year 1
Module Title: Research Methods Module
Module Tutor(s): Roisin Donnelly & Claire McAvinia
Date of Submission: May 28th 2013
Introduction
SoLAR, (Society for Learning Analytics and Research) defines learning analytics as the “measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimising learning and the environments in which it occurs” (SOLAR, 2013, p. 1).
Learning Analytics (LA) offers us a lens into the student learning experience and has been identified as one of the “fastest growing areas of research into technology-enhanced learning”(DEHub, 2012, p. 8). In line with this,LA was cited in the 2011 Horizon Report (Johnson, Smith, Willis, Levine, & Haywood, 2011) as an emerging technology predicted to impact on teaching, learning and research in the far-term horizon (four / five year window). Subsequently it wascited as a technology predicted to enter mainstream use in the mid-term horizon (two / three year window) in the 2012 (Cummins, Johnson, & Adams, 2012)& 2013 Horizon Reports (Johnson, Adams Becker, Cummins, Estrada, Freeman, & Ludgate, 2013).
The focus of the research paper is to explore LA in the context of Dublin Institute of Technology(DIT) and its learning management system (LMS) namely Blackboard Learn.
LMS such as Blackboard, Moodle and Desire2Learn track and store vast quantities of data on students and their engagement with course content .This potentially can provide insight intostudents and assist lecturers in making informed decisions regarding their pedagogical practice .The proposal aims to examine the data and reporting features available through Blackboard to determine if it assists lecturers and students in making better informed decisions regarding the student learning experience.
The purpose of the research proposal is not to peddle LA as the solution to every problem in education but to support the decision making process with regard to student learning.
Rationale
The research questions in this proposal focus on the usage of LMS data and reporting features to gain greater insight into students.
Digital footprints left behind by studentsin their online engagement provide vast amounts of data. Information gleaned from Blackboard include log-in information, the frequency by which students access online materials, the results of assessment, student engagement with course content and discussion forum activity amongst others. In addition Blackboard provides reporting features illustrated in a later diagram (Figure 1). Analysis of the data identifiesstudents who are actively engaging, succeeding or struggling with course content.
Dawson, Mc William, & Tan (2008,p.222) assert the goal of LA is no longerto simply generate data and make it available to the necessary stakeholders e.g. students, lecturersand courseco-ordinators but rather to effectively analyse and interpret such data and translate its finding to practice.
The research includes developing a resource for academic staff highlighting the main functionality of Blackboard reporting and demonstrating its main features. The resource will equip staff with the requisite skillset, knowledge to run queries, reports, set up early warning systems and other features of LMS (refer to Figure 1)
Macfadyen & Dawson (2010, pp. 589-590) cite the importance of the social nature of learning referring to educators increasing recognition of the benefits associated with learning and teaching that embrace socio-constructivist principles. Social Networks Adapting Pedagogical Practice(SNAPP) isa social network analysis tool providing visual aids that depict the interactions amongst staff and students (Bakharia, Heathcote, & Dawson, 2009). It captures engagement/participation in online learning communities as it contructs relations based on forum interactions and therefore identify where socio-constructivist goals are achieved. The proposal aims to incorporate the SNAPP tool into Blackboard.It envisages the tool will provide real time visual representation of participation among students in discussion forums on Blackboard. Social network analysis has been “demonstrated to assist educators in identifying learner isolation and community formation”. (Dawson & Bakharia, 2011, p. 3)
Numerous studies conducted highlight the necessity for exploring LA. The following points, summarised below, are additional evidence for further research.
- A recent survey conducted by International Data Corporation produced a staggering statistic which estimated just 1% of the world’s data had been analysed (O’Brien, 2013).Clearly there is an enormous challenge ahead for both business and education to adopt effective data mining techniques.
- LA tools takes the tracking of student online engagement out of the hands of the lecturer as they monitor , track and record more data than a course instructor can alone (Educause, 2011).
- (Brown, 2013)highlights the increasing emphasis on the use of metrics in Higher Educational Institutions (HEI) in order to demonstrate student learning and progression rates. There is compelling evidence indicatingHEI are not making sufficient use of data mining techniques. Iwitnessed further evidence of this recently at a seminar I attended in DITthat focused on student retention.Vast quantities of data from academic performance results, progression rates and student surveyswere provided at the seminar.There is a greater need to adopt more data analysis techniques to enable all stakeholders to make better informed decisions regarding student teaching and learning (O’Rourke & Russell, 2013). In line with thisSiemens & Long (2011) affirm that HEI have traditionally been inefficient in their use of data. The findings closely align with my personal views that vast amounts of data exists that we can use more efficiently to identify at risk students. Such knowledge can assist course co-ordinators and lecturers to adopt early intervention techniques to target at risk students.
- LA can reach beyond the focus of individual students. It can enlighten academics regarding their pedagogical practice.It can identify which teaching techniques are more effective than others (Campbell & Oblinger, 2007). LA can identify the learning objects/resources that prepare students for achieving learning objectives whilst highlighting ineffective learning resources. The information derived from LMS reporting assists lecturers in removing any barriersin existingcourse content or delivery.
Aims and Objectives
The aims and objectives of this project are summarised below.
- Promote the utilisation of Blackboard data, its reporting featuresand measure effectiveness of same (see Figure 1)
- Build a resource for lecturers that demonstrates reporting features of Blackboard
- Track and monitor student engagement with course content in order to determine if/how this correlates with assessment grades.
- Capture lecturers’ perceptions of LA, Blackboard data and reporting functionality.
Figure 1 Built-in reporting features of Blackboard
Literature Review
The annual Learning Analytic & Knowledge (LAK) conferences commenced in 2011 and have contributed significant research to this area.Papers submitted to the conferences,together withEducause Learning Initiative,provided invaluable sources for literature. Alfred Essa’s explanation of the LA tools available resonate with me in that he describes the tools we build for LA as being “akin to telescopes or microscopes that will hopefully allow us to see deeper, wider into student learning” thus optimising the student experience (Chalex, Essa, & Norris, 2012) .
The literature provides various examples of HEI applying data mining techniques. A numberof HEI in the United States are amongst the earliest leader adopters of LA. These include the Signals Project in Purdue University (Arnold, 2010), the establishment of the Performance Analysis Framework (Wagner & Ice, 2012) ,the early warning systems adopted by Paul Smith’s College in the US (Taylor & McAleese, 2012) and University of Michigan (Lonn, Krumm, Waddington, & Teasley, 2012). To narrow the scope of this study, the criteria for literature review focused on research papers adopting mining techniques on LMS data.
Case studies conducted by HEI have focused on student retention and identifing at-risk students. In an effort to address increasing demands for student success and institutional accountability HEI have witnessed a surge of interest in data mining and learning analytic technology in order to identify at-risk students and increase student retention rates(Campbell & Oblinger, 2007, pp. 3-6 ;DeBlois, Campbell, & Oblinger, 2007, p. 42).
During my literature review, I found research conducted at the University of British Columbia to be a significant research project (Macfadyen & Dawson, 2010). Similarites exist between the research carried out in their project and the research proposed in this document. They mined Blackboard data and utilised the SNAPP tool to carry out social network analysis to identify student/student and student/lecturer communication. They investigated which student online activities accurately predict academic achievement and which LMS data variables represented student effort or activity. Their research concluded strong correlation exists between LMS activity and course outcome particularly with discussion forum activity. This is further evidenced by Smith, Lange, & Huston (2012, p. 60) ;Dawson, Mc William, & Tan (2008, p. 227) whose findings suggest active site engagement with LMS can serve as an effective predictor of course outcomes. It was interesting to note fromDawson et al. (2008, p. 224), when referring to web 2.0 technologies, how mining of LMS data can provide valuable insight about lecturers pedagogical practice as they contend that data gleaned from LMS can be used to “guide and inform the diffusion of technology and integration into learning and teaching activities.” Such findings enlighten lecturers of the technology tools most effective in their teaching practice.
The readings highlight a concern relating to the utilisation of LMS that may have implications for this proposed study. In studies carried out by(Black, Dawson, & Priem, 2008 ;Dawson, Mc William, & Tan, 2008 ;Macfadyen & Dawson, 2010)LMS tools i.e. forums, wikis and discussion forums were used by students and lecturers. Since commencing research on LA, I have asked academic staff to what extent Blackboard is used and for what purpose. Feedback from informal discussions suggest LMS is regarded as a data repository. Previous research exploring LMS provided further evidence of this finding (Dyckhoff, Zielke, Bültmann, Chatti, & Schroeder, 2012, p. 58). Criteria for selection of DIT class codes to apply this research to will be based on the usage of web 2.0 tools within Blackboard in order to circumvent the practice.
Notably (Educause, 2011, p. 2) claimsstudents and instructors may feel that LA “takes assessment out of the realm of human judgment and reduces it to numbers and statistics”. The literature review uncovered similar findings withCampbell & Oblinger (2007, p. 16) raising the issue when highlighting concerns for HEI implementing analytic projects, contending such projects may be perceived as “dehumanising the educational process”. Diaz & Brown (2012) claim despite the benefits of LA technology, it should not be used by academics to abrogate their responsibilities in decision making as educators. Its role is to support sensemaking not supplant it (Brown & Diaz, 2012, p. 3). My thoughts on mining data are to inform the necessary stakeholders.It should provide intelligent data to enable better decision making. Whilst I am keen to explore this area, the proposal may be perceived as advocating the adoption of LA in HEI. I stress the research merely proposes an exploratory study to see beyond the “fog” of LA and determine to what extentits findings can be utilised. In line with thisSiemens contends
in spite of the potential of LA , it is important to emphasise that LA is not the end goal, it serves to provide stakeholders with better information and deep insight into the factors within the learning process that contribute to learning success. (Siemens, et al., 2011, p. 5)
Whilst LA offers much promise, the literature review highlightsa major concern namely data privacy. This has been raised by a number of experts in the field. Understandably students mayperceive LA as an invasion of privacy as the monitoring and tracking of student online engagement raises the “specter of a digital big brother” (Educause, 2011, p. 2). The concern was highlighted at the first LAK conference in 2011 at which attendees were in agreement that LA “raises deep and complex privacy issues” (Brown M. , 2013, p. 3) . This is one of the major challenges faced by LA that I identify with in the context of theproposal. I need to tactfullystrike a balance between learner privacy and the value of data collection for improving learning.
Research Methods
The purpose of this research is to conduct an exploratory case study to determine effectiveness of mining LMS data to monitor student engagement and progression with course modules. Cousins (2005, p. 421) states that “case study research aims to explore and depict a setting with a view to advancing understanding of it”. In line with this, it is envisaged that analysis of data and reports extracted from LMS, together with understanding lecturers’ perceptions of LA will shed light on the research question.
The study will adopt a sequential mixed method approach with a qualitative follow-up phase building on the initial quantitative phase (Creswell, 2009, p. 211) . The first phase will involve the measurement and analysis of data gleaned from LMS reporting to identify key statistical information regarding student activity. This will provide the quantitative data for the study. Tracking of LMS data variables will include login times, grades, frequency of logins, user activity in forums and discussion boards, etc. Agreement on what LMS data variables to track and monitor will be sought with each of the lecturers before the commencement of the study. Ongoing tracking and monitoring of student activity will be carried out during the first phase of the study.
In order to gauge lecturers’ perceptions on LMS data and reporting and to evaluate the proposed resource, one to one interviews will be held with course tutors during the second phase. Information from the first phase will be explored and discussed further during the interviews to evaluate the merits of adopting LA techniques in the modules. The interviews with course tutors will be the primary source of qualitative data during the study and will aim to address some of the following:-
o Evaluation of resource
o Tutors pre/post perception of LA
o Will course tutors adopt data mining techniques in future programmes?
Due to time constraints the case study will be targeted at a small number of course modules commencing at the start of the 2013/2014 academic year in DIT. One module is Strategic Management which will involve a cohort of 30 students in 4thYear BSc. Business Computing Programme, course code DT354. It is envisaged that an additional three/four modules will be selected from the College of Sciences & Health in DIT. Criteria for selection of modules will focus on the most frequent users of LMS and its web 2.0 tools.
The study will require authorisation and participation from each of the lecturers running the modules. Initial informal discussions had indicated that lecturers are interested in participating in this study but this will require further dialogue over the coming months. The Strategic Management tutor has already confirmed his participation.
Timescale
Table 1 Project Timeline gives a broad outline of the project timescale, which is a proposal, and entries and dates may be subject to change. In addition the timescale may be subject to change based on feedback on proposals in June 2013 when the scope of the project may need reviewing.
Limitations
The limitations of this research proposal can be summarised under the following headings;
Data Protection
Data protection laws that govern the use of certain data may prohibit access to sensitive data. Initial discussions suggest I may be able to gain access to such data. This will require further discussion with DITEthicsCommittee, academic staff and students.
Confinement to LMS
One limitation of confining data analysis to LMS is it does not capture face to face dialogue between lecturer and student nor does it capture student engagement with social media tools that reside outside LMS. This has been raised by Siemens & Long (2011) who assert that analytic models do not capture library use, access to learning support or face to face discussions with academia. While addressing the problem that arises from capturing the dialogue between lecturer and student may prove difficult, I will endeavour to promote the use of web 2.0 tools such as wikis and discussion forums within LMS during the course of research rather than the alternatives that exist outside Blackboard.
Time Constraints
The time and effort required for this research proposal will be balanced with full time work commitments at DIT. The exploratory phase will commence at the beginning of the 2013 academicyear. Prior to commencementa requirement to build a resource that demonstrates the reporting features of Blackboard is necessary. Identification and confirmation of DIT course codes to apply this research needs to be sought by the first week in July 2013. Blackboard reporting will be closely monitored from October to December 2013. Findings and follow up interviews will be conducted and documented in January and February 2014 prior to writing the final research paper in the second quarter of 2014.
Academia Participation
The research proposal is dependent on academic participation and buy in. It will necessitate lecturers’ participation to conduct an exploratory case study next year. This may require a change in their teaching practices that they may be reluctant to undertake.
Ethical Considerations
Initiating any analytics project requiring access to student records raises data privacy and confidentiality issues. Approval from the admissions office, academic staff, students and the DIT Ethics Committee will be a pre-requisite. Compliance with DIT Ethics Committee’s guidelines will be a requirement. In order to safeguard the security and confidentiality, student data will be encrypted using McAfee Endpoint encryption software.
In line with Creswell (2009, pp. 89-90) and in order to protect participants’ rights during the data collection an informed consent form will be provided for participants outlining elements of:-
- Purpose of the research
- How the participants were selected
- Level of participant involvement
- Risks involved
- Guarantee of confidentiality
- Assurance that the participant can withdraw at any time
- Provision of names to contact if questions arise
Summary
LA is still in its infancy stage. It is an emerging tool which needs to evolve in terms of sophistication, popularity and effectiveness, particularly with Irish HEI. Studies such as those mentioned aforesaid have suggested that HEI can harness the data that LMS provides us to gain better insight into our learners. Anaylsis of the data trails that students leave behind in their online engagment with LMS provides a basis for exploratory study on LA.
Siemens, et al. (2011) cite the broad goals of LA to include the improvement of completion rates and providing decision makers with needed information regarding learners. Whilst the scope of this project is much narrower, resources more limited and time constraints more restrictive, it will be interesting to observe what direct benefit we can draw from the analysis of LMS data, if any, andto determine its value in the context of student engagement, learning progression and access the impact of lecturer’s pedological approach.It is envisaged that the completion of this project and adoption of resource will shed new insight as to what role, if any, analytics can play in DIT.
References
Arnold, K. (2010, March 3). Signals: Applying Academic Analytics. Educause .
Bakharia, A., Heathcote, E., & Dawson, S. (2009). Social networks adapting pedagogical practice: SNAPP. In Same places, different spaces. (pp. 49-51). Auckland: Australian Society for Computers in Learning in Tertiary Education.
Black, E. W., Dawson, K., & Priem, J. (2008, March). Data for free: Using LMS activity logs to measure community in online courses. Internet and Higher Education, v11 n2 , 65-70.
Brown, M. (2013). LEARNING ANALYTICS: THE COMING THIRD WAVE. EDUCAUSE Learning Initiative (ELI) , 1-4.
Brown, M., & Diaz, V. (2012). Learning Analytics: A Report on the ELI Focus Session. Educause.
Campbell, J. P., & Oblinger, D. G. (2007, October). Academic Analytics. 20.
Campbell, J. P., DeBlois, P. B., & Oblinger, D. G. (2007). Academic Analytics A New Tool for A New Era. Educause.
Chalex, J., Essa, A. H., & Norris, D. M. (2012, April 12th). ELI Focus Session The LMS Perspective. (M. Brown, Interviewer) ELI (Educase Learning Initiative).
Cousins, G. (2005). Case Study Research. Journal of Geography in Higher Education , 29 (3), 421.
Creswell, J. W. (2009). Research design: Qualitative quantitative, and mixed methods approaches (3rd ed.). Thousand Oaks, CA: SAGE.
Cummins, M., Johnson, L., & Adams, S. (2012). 2012 Horizon Report. Austin: The New Media Consortium.
Dawson, S. P., & Bakharia, A. (2011). SNAPP -a bird's-eye view of temporal participant interaction. Learning Analytics and Knowledge 2011 (LAK2011) (pp. 168-173). New York: ACM.
Dawson, S., Mc William, E., & Tan, J. P.-L. (2008). Teaching Smarter: How mining ICT data can inform and improve learning and teaching practice. In Hello! Where are you in the landscape of educational technology? (p. 222). Melbourne: ASCILITE.
DEHub. (2012, May). Learning analytics: following the trail of evidence in digitised education. (11, Ed.) DEQuarterly-Winter-2012-Edition-No-11 , 8-10.
Diaz, V., & Brown, M. (2012, July 24). Navigating the waters of learning analytics. Retrieved from Educause Learning Initiative.
Dyckhoff, A. L., Zielke, D., Bültmann, M., Chatti, M. A., & Schroeder, U. (2012). Design and Implementation of a Learning Analytics Toolkit for Teachers. International Forum of Educational Technology & Society , 58-76.
Educause. (2011, December 6). 7 THINGS YOU SHOULD KNOW ABOUT FIRST-GENERATION LEARNING ANALYTICS. Educause , 1-2.
Fowler, S., & Diaz, V. (2012, November). Leadership and Learning Analytics. Educause Learning Initiative , 1-4.
Johnson, L., Adams Becker, S., Cummins, M., Estrada, V., Freeman, A., & Ludgate, H. (2013). Horizon Report 2013.
Johnson, L., Smith, R., Willis, H., Levine, A., & Haywood, K. (2011). The 2011 Horizon Report. Austin: The New Media Consortium.
Lonn, S., Krumm, A. E., Waddington, R. J., & Teasley, S. D. (2012). Bridging the gap from knowledge to action: putting analytics in the hands of academic advisors. Proceedings of the 2nd International Conference on Learning Analytics and Knowledge, (pp. 184-187). New York.
Macfadyen, L. P., & Dawson, S. (2010). Mining LMS data to develop an ‘‘early warning system” for educators: A proof of concept. Elsevier , 588-599.
O’Brien, C. (2013, April 29th). How making sense of big data could create jobs in Ireland. Irish Times .
O’Rourke, K., & Russell, M. (2013, May 1ST). Perspectives on Education Seminars 2013 - focus on retention. Using technology to enhance retention .
Punch, K. F. (2006). Developing Effective Research Proposals (2nd Edition ed.). London: SAGE.
Siemens, G. (2012, April 11). Sensemaking: Beyond Analytics as a Technical Activity. Retrieved May 2013, from Educause: http://educause.adobeconnect.com/p682d7bpzqd/
Siemens, G., & Long, P. (2011). Penetrating the Fog: Analytics in Learning and Education. 46 (5), 36-38.
Siemens, G., Gašević, D., Haythornthwaite, C., Dawson, S., Buckingham, S., Ferguson, R., et al. (2011). Open Learning Analytics: an integrated & modularized platform. (SOLAR)Society for Learning Analytics and Research.
Smith, V. C., Lange, A., & Huston, D. R. (2012). Predictive Modeling to Forecast Student Outcomes and Drive Effective Interventions in Online Community College Courses. Journal of Asynchronous Learning Networks , 16 (3), 51-61.
SOLAR. (2013). Society for Learning Analytics and Research. Retrieved April 2nd, 2013, from Society for Learning Analytics and Research: http://www.solaresearch.org/mission/about/
Taylor, L., & McAleese, V. (2012, July 18). Beyond Retention: Using Targeted Analytics to Improve Student Success. Educause Review Online , 1-9.
Wagner, E., & Ice, P. (2012). Data Changes Everything: Delivering on the Promise of Learning Analytics in Higher Education. Educause Review , 47 (4), 36-40.
|
|
