Manuscript title:

Modeling Students’ Performances in Activity-Based e-Learning from a Learning Analytics

Submitted to the journal:

Abstract:
Researchers have recently been directing significant resources to demonstrate how learning

analytics can help in better course design, improve student success and completion rate, and

to improve performances. In this research, descriptive and predictive learning analytics are

applied to available data for two different cohorts of students who followed an activity-based

online module in two subsequent academic years. The objectives of the research were to

understand what learning analytics could contribute in terms of improving our understanding

of the student performances, and whether a predictive approach to modeling of the students’

performances can help to improve the learning design and experiences of students. Students’

performances were related to the skills and competencies acquired during the course by

developing artefacts. The findings of this study combined with other findings as reported in

the literature, demonstrate that the learning design is an important factor to consider with

respect to application of learning analytics to improve teaching interventions and students’

experiences. Furthermore, to maximize the efficiency of learning analytics in e-learning

environments, there is a need to review the way offline activities are to be pedagogically

conceived so as to ensure that the engagement of the learner throughout the duration of the

activity is effectively monitored. However, the learning design can be reviewed in such a way

to make learning analytics appropriate to measure student engagement at various stages in the

different learning activities so that tutors can provide better support and scaffolding to

students who are having difficulties in the learning processes.

1
Keywords: predictive analytics, performance modeling, learning design, activity-based

learning, online learning, learning analytics

Introduction

A number of researchers, practitioners and e-learning developers have recently been

directing significant resources to demonstrate how learning analytics can help in better course

design, improve student success and completion rate, and to improve performances. These

are achieved through systematic and in-depth analysis of student data that is generated and/or

stored in e-learning environments and platforms. Such data are mainly related to platform

login frequencies, online content browsing, completion of drill and practice activities,

frequency of online interaction with peers and tutors, and submission of coursework online

(Retalis, Papasalouros, Psaromiligkos, Siscos and Kargidis, 2006; Johnson, Smith, Willis,

Levine and Haywood, 2011). There are also other variables that may be stored depending on

the learning design approaches such as learning styles and preferences, cultural factors,

student feedback and satisfaction surveys. There are four types of analytics as reported in the

literature, namely descriptive, diagnostic, predictive and prescriptive analytics (Gartner,

2012).

Literature suggests that many reported studies on learning analytics focus on classic

elements such as frequency of access, time spent online, and number of online posts as key

variables to be used as predictors of performance, student success and also the identification

of students who are at risk, and most recommendations regarding learning design would be in

line with what is normally referred to as the classic e-learning design model. The classic

model is referred to as a course which is centred around the dissemination of online content,

such as on-screen text, images, and videos where the maximum of the learning process

2
happens online, and on the e-learning platform (Santally, Rajabalee and Cooshna-Naik,

2012). On the other hand, in line with Nichols (2003), the Web can also be seen as a way to

transform the teaching and learning process. Schneider, Synteta, Frété, Girardin and Morand

(2003) argued that traditional e-learning courses as described above, are not designed to

achieve the development and acquisition of skills and competencies, and the same could be

achieved, through online project-based learning, along with a redefinition of the role of the

teacher. As such, an online course may be composed of a set of learning activities, which

may or may not be completed online, depending on the intrinsic learning design and expected

outcomes of each activity, and the learning domains might span and overlap different spheres

of the cognitive, psychomotor and affective domains depending on the subject being taught.

In such settings, applying learning analytics on the variables such as frequency of access, and

online interaction alone may not suffice.

In this research, the aim was to explore the potential, constraints and impacts of a

learning analytics approach to an online course that does not follow the conventional learning

design approach, and that has been offered to first year undergraduate students as a General

Education Module at the University of Mauritius for two consecutive academic years. There

were 217 students in the first cohort and 844 students in the second cohort. The module title

was “Education Technologies” and was destined to those having an interest to later take up a

teaching career focusing on the integration of Information and Communication Technologies

in Education. The module was designed using the activity-based approach as advocated by

Schneider et al. (2003) and the learning design followed the hybrid three phased-model of

knowledge acquisition, knowledge application and knowledge construction and reflection

(Santally and Senteni, 2006; Santally and Raverdy, 2006; Santally et al, 2012). This model

lays emphasis on the need to conceptualise learner-centred educational experience through

3
activity-based learning, which leads to authentic learning outcomes and acquisition of

competencies (Fernando, 2018; Reid, 2017; Morcke, Dornan and Eika, 2013). Literature in

learning analytics studies do not really cover modules or courses that employ a learning

designs similar to the module that is the subject of the current research.

There was a total of nine learning activities in all. The first eight were considered as

the continuous assessment activities, while the ninth learning activity was considered as the

final assessment of the module. For each learning activity, there was a kind of artefact to be

produced, which was assessed and marked by the tutor. As such descriptive and diagnostic

analytics was applied to the information that could be retrieved from the e-learning platform,

to understand and compare the performances of the students of the two cohorts from different

angles and to try to explain these observations. The key question that the researchers tried to

answer here, was ‘what can be learnt from such information?’.

The second phase of the work looked at the possibility of modeling the student

performances in this specific type of activity-based learning design using predictive learning

analytics. How could such an approach help academics and learning designers in better

predict student performances or identify at risk students to allow just-in-time learning

support, in an activity-based learning context? Would such an approach of modelling student

performances, be really helpful and accurate enough, taking into consideration that each

learner is different and may have varying needs? These were the other key questions that the

researchers tried to investigate in this research. To address these questions, data related to the

two cohorts who followed the module “Education Technologies” were retrieved from the e-

learning platform and tutor records regarding their marks in continuous assessment and the

final learning activity.

4
Literature Review

Learning Analytics to Predict Students’ Performances

Learning analytics, is the means to measure, collect and analyze volumes of data

about learners’ progress in order to understand, optimize and improve their learning

experiences (Siemens and Long, 2011; MacFadyen and Dawson, 2012). According to Gartner

(2012), there are four main forms of Learning Analytics namely (i) descriptive, (ii)

diagnostic, (iii) predictive and (iv) prescriptive analytics. Descriptive analytics refers to the

technique used to examine data that has been used in the past and try to report on what has

occurred. It presents archived data in an understandable way to provide meaningful

information and identify what has happened. Diagnostic analytics brings descriptive analytics

one step further, as once it is established about what has happened, there is a need to identify

why it has happened. In some situations, diagnostic analytics help to gain an in-depth

understanding as to why new trends are emerging and what are the roots and influential

factors causing such trends (Banerjee, Bandyopadhyay and Acharya, 2013). Predictive

analytics focuses on using estimates obtained from descriptive and diagnostic analytics to

look for patterns and trends to predict what is the likely outcome in similar situations in

future. It is a means to achieve foresight and forecasting in educational contexts through

statistical modelling techniques to identify at-risk students or prediction of students’

achievement based on significant parameters (Liñán & Pérez, 2015). Finally, prescriptive

analytics uses both descriptive and predictive analytics to understand what has happened and

to predict a number of possible outcomes and seeks the best approach or model to achieve the

desired outcome (Gröger, Schwarz and Mitschang, 2014).

5
There are different learning-related variables that actually contribute and impact on

learners’ learning experiences. Some of these variables are directly measurable while others

are not. Tempelaar, Heck, Cuypers, van der Kooij and van de Vrie (2013) argued that many

factors such as cultural differences, learning styles, learning motivations and learning

emotions might be influential on students’ academic performances in terms of how well they

fare in assessment activities. The choice of what is to be measured is important for accurate

predictions of performances (Gašević, Dawson, Rogers & Gasevic, 2016). Smith, Lange and

Huston (2012) found that learners' pace of learning and their engagement with their learning

materials can be used to predict their performance and eventually result in more conducive

learning environments. In studies on learner retention, researchers support the fact that

frequently learners’ performance can rationally be predicted, in terms of social, psycho-

emotional and demographic factors, by using learning analytics (Credé & Niehorster, 2012;

Richardson, 2012). Similarly, Macfadyen and Dawson (2010) found that variables such as

discussion forum posts and completed assignments, can be used as practical predictors to

measure learner performance, and thus can be used to help in learners' retention and in

improving their learning experiences. However, the determination of the right variables to be

included in a learning analytics approach is important in determining the success of such

approaches.

Pardo, Han and Ellis (2017) used self-reported data and observed data in a combined analysis

to investigate whether such an approach can predict academic performance and provide

comprehensive insights why some students tend to perform better. Their case study

represented a blended learning course where self-reported data was based on students’

motivational, affective and cognitive aspects while observed data was related to students’

engagement captured from activities and interactions on the learning management system. It

6
was found that students’ who adopted a positive self-regulated strategy tend to participate

more frequently in online events, which could explain why some students perform better than

others.

Learning analytics can be used to create a predictive model to predict learners’

academic performance. Lykourentzou, Giannoukos, Nikolopoulos, Mpardis and Loumos

(2009) applied machine learning techniques as prediction method, to study students’ profiles

on the LMS and to identify at-risk students in order to predict dropouts at an early stage.

Tempelaar et al. (2013) used learning analytics through the measurement of learning related

data such as demographic, entry test, learning dispositions, learning management system and

quiz data to predict students’ performances in blended courses. On the other hand, Agudo-

Peregrina, Iglesias-Pradas, Conde-González and Hernández-García (2014) did not find any

consistent patterns in using learning analytics for prediction of performances in blended

courses, precisely due to the lack of information that can be collected from the face-to-face

components of such courses.

Tempelaar, Rienties and Giesbers (2015) used learning analytics to investigate

whether learning dispositions, continuous formative assessments and LMS (Learning

Management Systems) data have the potential to model and predict students’ academic

performance by generating informative feedback for students and teachers. They reported that

formative assessments have better predictive power over LMS data and learning dispositions

as proposed by Shum and Crick (2012) can be used as a means for generating feedback.

Feedback which are related to learning dispositions, whereby signaling underperformance,

are generally used to improve learning processes (Lehmann, Hähnlein & Ifenthaler, 2014).

Therefore, feedback has the potential to derive or even empower prediction models.

Tempelaar, Rienties, Mittelmeier and Nguyen (2018) derived prediction models built on rich

7
data in the form of learning dispositions combined with learning analytics trace data to

identify students who are at-risk of failure. They found that providing students with formative

feedback have strong predictive power to support learner to succeed.

Viberg, Hatakka, Bälter and Mavroudi (2018) carried out a review of papers on

learning analytics research and its evidence for teaching and learning in Higher Education.

They argue that there is little evidence that learning analytics improves learning outcomes.

However, there is evidence (35%) based on the analysis of studies that fall under the years

2012-2018, that it improves learning support and teaching in higher education. Viberg et al.

(2018) further revealed that most of the learning analytics studies that were reviewed adopted

a descriptive approach, and there has a been a decline in the use of predictive analytics in

recent years. This is explained through the fact that there is a shift towards in-depth

understanding of the learning processes and students’ experiences. Ferguson, Brasher, Clow,

Cooper, Hillaire, Mittelmeier, Rienties, Ullmann and Vuorikari (2016) argued along the same

lines, that learning analytics as a tool, emphasizes more on visualizing of data to determine

engagement and activity developing systems, rather than identifying what are the actions that

need to be taken to improve teaching and learning.

Activity-based e-Learning Designs

Activity-based learning is an approach to engage learners actively by exposing them

to a variety of activities where they can learn-by-doing, develop their knowledge in terms of

ownership and exchange, conceptual understanding, competencies and practical skills

(Biswas, Das and Ganguly, 2018). Ranganath (2012) and Tan, Chong, Subramaniam and

Wong (2018) explained how activity-based learning is successful as a teaching model in

8
management, medicine, engineering and science field of study. Activity-based learning, as a

pedagogical choice, is therefore an educational method focused of the development of

competencies, skills and artefacts. It emphasizes on the learners’ application of skills as in

contextualized and authentic settings (Morcke et al, 2013; Frank, Snell, Cate, Holmboe,

Carraccio, Swing, Harris, Glasgow, Campbell, Dath & Harden, 2010). Tan et al. (2018)

postulated that activity-based learning within a competency development training model has

been effective in nursing education. It was also used in teacher education to empower

educators and to enhance their professional development (Leppert & Kingman, 2017;

Santally, 2012). Similarly, the concept of LDL which means learning by teaching also

emphasizes on shifting the responsibility on the learner to encourage their engagement and

participation in developing an explorative behaviour, disciplinary virtues, competencies and

skills (Grzega, 2006). Therefore, the effectiveness of activity-based learning approach can

allow learners to adopt tutoring experiences to train themselves to acquire “hard and soft

skills”.

Similarly, Priyono, Wena and Rahardjo (2017) also supported that such a pedagogical

approach encourages students to be involved in activities where they can acquire and apply

their knowledge through active experience. Such experiences promote the potential of

improving their competencies in critical thinking. The effectiveness of activity-based learning

for encouraging the application of theoretical knowledge in problem-solving was also

reflected from students’ feedback in Patil, Budihal, Siddamal and Mudenagudi (2016).

However, activity-based learning which was applied in teacher training, and which was found

to be effective, did not show any difference in terms of students’ achievement, when

compared to traditional teaching (Çakmak, 2016). In contrast, it also found that the

9
implementation of activity-based learning helped in skills development and enhancement of

specific competencies for both teachers and students (Fernando, 2018; Reid, 2017).

The hybrid model is essentially an integrated learning environment designed in such

way that it incorporates both the content-based and activity-based learning models to

complete the phases of (i) knowledge acquisition, (2) knowledge application and (3)

knowledge construction through reflective practices (Santally et al., 2012). Two models have

been defined by Santally et al. (2012) to blend content-based approaches with activity-based

learning. The first one is mainly outcomes based, centred around the topic and integrating

learning activities at the end of each topic (that spans over a specific time period). The second

model places the activity at the heart of the educational transaction. For example, a main

learning activity, can be broken down into a sequence of 3-4 subsidiary activities, which,

when completed, would result in an artefact demonstrating that the student has acquired and

applied certain skills and competencies. To carry out such learning activities, the student may

need to engage with contents as pedagogical scaffolds to the learning process (Schneider et

al., 2003). In line with this, Nuninger (2017) also proposed the Integrated Learning

Environment (ILE) as a pedagogical means that focusses on learner-centred pedagogy to

drive the use of a mix of pedagogical solutions. The objective was targeting competences that

are to be put into practice based on knowledge ownership. This was also highlighted by

Jones, Baran and Cosgrove (2018), about how the importance of outcome-based strategies in

learning can help students towards their goals and future professions.

Similarly, Conole (2012) has developed the activity type classification instrument which

essentially classifies student activity that focuses on what the student is actually engaged in a

learning-by-doing approach. The activity type classification of Conole (2012) proposes the

10
different rubrics to model learner activity within a course namely (i) Assimilative, (ii)

Finding & Handling Information, (iii) Communication, (iv) Productive, (v) Experiential, (vi)

Interactive/Adaptive, (vi) Assessment. This classification which is at micro-level fits well

within the macro-level framework proposed by Santally (2012) for activity-based learning

designs and their models of hybrid / model of learning design as follows:

Table 1. Mapping of the two models of activity-based learning.

Methodology

Context

The “Educational Technology” module was designed and developed following a

pedagogical model that is centered around the development of skills and competencies

through three distinct phases of knowledge acquisition, application and co-construction

phases (Santally and Raverdy, 2006). The learning designers used the activity-based learning

model, to promote and inculcate (i) a self-learning culture among students, (ii) development

and acquisition of practical competencies, and (iii) higher-order cognitive skills. Its main

objectives were to allow learners gain a good understanding of the different concepts and

11
theories of ICT in education and help them to develop their competencies in using

technological tools.

The Module was offered to two consecutive cohorts of first year students as an online

module over one academic year (two semesters of 13 weeks) duration. The module was

hosted on the MOODLE e-learning platform. It consisted of 9 different learning activities and

was examined mainly by coursework rather than through supervised written examinations.

The module was structured as highlighted in figure 1.

Figure 1. Module structure at run-time over a 26-week period.

Each unit had a descriptive overview and specific learning objectives to allow the

learners to know which competencies they were expected to develop before they progress to

the next unit. Each unit also had a learning activity and the criteria for assessment of that

activity, to enable the learners to understand how they would be assessed. Moreover, at the

start of the module, only the first unit was made available to the learners, so that they do not

get overloaded and confused with all the units at one go. Each unit of the module was

released, that is, made available to the learners, at a specific described time and this was done

by the tutor. The approach that was used for learners of both cohorts to have access to the

module, was by assigning a specific ‘enrolment key’ in the module, which was then sent by

email to all the learners, who expressed their interest for this module. The ‘enrolment key’ is

a password to access the module that was created to allow students who opted for the module.

Only students enrolled on the course are given access till the end of the course and this was

12
applied for both cohorts that were offered. Figure 2 provides a description of the learning

activities, their respective importance (cognitive level and competencies required) with

respect to the course learning outcomes and nature of the activities, in terms of whether

online presence was required or not.

Table 2. Description of Learning Activities.

Learner support was dealt as an essential component of this online module. Every unit

that was made available to the learners, had a ‘Question & Answer’ forum, which learners

could use to post their queries or share their knowledge with their tutors and peers, pertaining

to the specific unit. In this forum the tutors encouraged the learners to interact among

themselves and to engage in peer support. It is worth mentioning that throughout the module

student-student interactions generally took over tutor-student interactions.

13
Research Questions

To achieve the objectives of this research as highlighted in the introductory sections, the

following research questions are addressed for both cohorts:

● Are there significant differences in performances (marks achieved) of the students

within and across the two cohorts with respect to continuous assessment and the final

learning activity?

● Which of the learning activities are more significant predictors of performance (marks

achieved) for each cohort in the final learning activity of students in the module?

Profile of Participants

The students on the module came from different disciplines namely Science,

Engineering, Law and Management, Agriculture and Social Studies and Humanities. The

module needed a minimum level of digital literacy from the students. All students who did

the module have followed the basic Information Technology introductory course as well as

the national IC3 (Internet and Core Computing Certification) course at Secondary Level. The

module was facilitated by 7 tutors and students were divided into groups ranging from 100 –

130 per group. The role of the tutors was mainly to act as a facilitator for the learning

process, and to mark learning activities and to provide feedback to the students. All the tutors,

who form the tutoring team agreed on a peer teaching standard for the delivery of the module.

This agreement was encouraged by the pedagogical design decision made in line with the

educational objective that guided the development of such a module.

14
Table 3. Number of students per discipline in Table 4. Number of students per discipline in
Cohort 1. Cohort 2.

Procedure and Methods

The module was hosted and delivered on Moodle where the students’ activities such as forum

posts, submission of assignments and input of marks and feedback by tutors are kept. The

data on Moodle, was retrieved through querying the Moodle Database and from student

achievement records exported to Excel Formats. These were the data available at that point in

time. Once the raw data was retrieved, grouping and classification was carried out as per the

requirements of the study. Using raw data can be misleading as they may contain errors,

redundant or irrelevant information. Outliers were identified for further analysis. Incomplete

data sets were further refined through further mining of platform data where applicable, and

irrelevant data sets or inaccurate ones were discarded. This was done by mainly pre-

processing the data using both manual and automated data mining to verify imperfections.

Out of the 1105 records (for both cohorts), 44 have been discarded due to the fact that the

students had completed less than 3 activities and did not do the final assessment activity.

During the course of this research, it was ensured that there is compliance with the

appropriate data protection laws and regulations. No personal data was processed in such a

way that it would reveal the identity of the students or which would be in contradiction with

legal and ethical provisions. We have ensured that all data have been anonymized,

depersonalized and used under modified attribution. The main approach was quantitative

15
data-gathering and through correlational studies and comparison of means. Student

performances were measured by assessing and quantifying their level of proficiency for skills

as per established marking criteria for each activity, knowledge and understanding through

direct measures collection, such as formal assessment of coursework, report writing, online

quiz, collaborative discussion and indirect measures collection such as their perception and

reflection on their learning experience. All the learning activities were each marked on a total

of 10 marks. Each student mark is moderated by another tutor in an independent manner as

per the regulations of the University. The pass mark of the module is 40% as per the

regulations of the University. For the sake of the correlational studies of this research the

final learning activity is counted as a separate ‘examinations’ activity that will count as an

end of module assessment on its own. However for an in-depth understanding of the situation

within the authentic setting, in some of the analysis, we also include the overall performances

(a weighted aggregate of activities 1-9) as it was in reality computed for this module by the

course designers with the knowledge that in such situations, the multicollinearity element has

to be looked into. To generate the predictive models of performances, the RapidMiner Studio

9.3 was used. Variables that were included in the training datasets are marks of each activity

(1-8), gender, and discipline. Performance for the purpose of this specific research is defined

to be a linear function of the marks achieved in each learning activity (referred to as the

continuous learning activities), gender and the faculty/discipline from which students come

from. Each of the models that is generated is based on in-built machine learning algorithms of

the software tool namely Deep learning, Generalized Learning Models and Random Forest

amongst others.

Assumptions and Limitations of the Research

16
In this research, the data that was used were related to learning activities students carried out

in one online module at the University. These activities were pedagogically designed using a

different approach. Although there was a high number of students who enrolled on this

module rendering the study to be contextually relevant, it was also restrained within the

institutional level only. The student population consists of local students, coming from

different fields of study, but who are also elements of the same secondary schooling system.

Consequently, while this ensures homogeneity to a good extent within the population under

study, there is a need to be cautious with respect to generalization of the findings. Data in this

study was not an experimental design but real data collected for analysis. Although the study

of data was done in an unplanned setting, researchers in this study could only study and use

data that were available to them. Data related to frequency of access, logins and clicks which

would have been helpful to analyse from an additional perspective, were not available.

However, this emphasis of this stressed on modeling students’ performances in the final

learning activity, with respect to how they progressed in the different types of learning

activities of the module. Given that there were marking schemes and criterion, it was

assumed that tutors’ marking styles were consistent.

Results and Observations

Marks Distribution for Cohort 1 & 2

It can be observed from the figures, that the continuous learning assessment marks

histograms for both cohorts are left skewed and is not likely to follow a normal distribution.

A test of normality to calculate the Skewness and the Kurtosis value, revealed that the

distribution cannot be assumed to be normal (Ghasemi & Zahediasl, 2012). Therefore, for

continuous learning assessment marks, non-parametric tests were conducted using Kendall’

17
and Spearman Rank method (Grech & Calleja, 2018). On the other hand, the normality test

revealed that the variable final assessment mark (Activity 9) marks could be assumed to be

normally distributed.

Figure 4. Distribution of Marks for Cohort 1.

Figure 5. Distribution of Marks for Cohort 2.

The box plot in figure 6 reveals the marks distribution in all the learning activities for

Cohort 1. It is not surprising and can clearly be noted from the figure that activities 2 and 4

which were MCQs marked by the system focusing mainly on basic knowledge that students

across board performed very well.

18
Figure 6. Distribution of marks in all learning activities (Cohort 1).

It can further be observed that students performed quite average on activities 1 (forum

discussion) and 7 (YouTube video reflective activity) while the range was bigger for activity

6 (concept-mapping). Overall in the module, it can be initially deduced that the students in

general, scored higher than average in most of the learning activities. The box plot in figure 7

reveals the marks distribution in all the learning activities for Cohort 2. The observation is

coherent overall with those for Cohort 1.

Figure 7. Distribution of marks in all learning activities (Cohort 2).

Overall, students performed between 50 – 70% as illustrated by the interquartile

ranges of the other activities. They performed slightly better in Activity 3, an observation

which is coherent with Cohort 1.

19
Comparison of Means

For cohort 1, through the Friedman test, it could be observed that the X2r statistic is

144.872 (3, N = 217). Since, the p-value is <.00001, there is significant difference between

the means at 1% level of significance. Similarly, for cohort 2, there was a significant

difference in mean continuous assessment marks as compared to the final assessment marks

(p<0.01). Overall, the students performed better in the continuous assessment mark than the

final assessment mark (X2r =117, 1, N = 844). The box-and-whisker plot in figure 8 illustrates

the data distribution for the five disciplines in the continuous learning activities (1-8) and

then for the final learning activity (Activity 9).

Figure 8. Box plot showing data distribution for 5 disciplines in Cumulative

From the continuous assessment marks, students from Sciences, Law & Management,

Engineering and Agricultural sciences in both cohorts performed in comparable ways, except

for the humanities with a lower median and wider interquartile range. For the final

assessment, the Sciences and the Humanities looked comparable while the Law &

Management disciplines had a higher median and wider interquartile range. However, in the

Engineering discipline, it can be observed that the median mark for the final assessment was

quite lower than for the continuous assessment mark. The highest mark in the final

20
assessment was from the Law and Management discipline. As mentioned earlier, there were

only 26 students in the Agriculture discipline. With respect to the final performances of the

two cohorts, there were no significant differences (p>0.01; α=0.01) observed in the mean

marks scored by the students. However, the mean marks of the continuous learning activities

(total aggregated marks of Activities 1-8) of the two cohorts were significantly different

(p<0.01; α=0.01).

Final Performances t-test

Table 5. t-test for final performance.

Continuous Learning Activities

Table 6. t-test for cumulative assessments.

Prediction of Final Performances

Data from Cohort 1

The system generated 58 models, among which the Random Forest Model (RFM) and

the Deep Learning Model (DLM) had better predictive power as presented in figure 9. The

correlation coefficient of both models was very close with practically equal variances.

21
Figure 9. First Predictive Model Generation using RapidMiner.

For the DLM, the Activities 8, 6, 3 and 2 are considered as significant to the

predictive ability of the model while Discipline (Faculty), Activities 1 and 4 are considered to

be contradicting prediction. Activities 5, 7 and the Gender element was not included as

significant factors for prediction.

Figure 10. Deep Learning Model (DLM).

Figure 11. Random Forest Model (RFM).

As regards to the Random Forest Model, Activities 8, 6, 3 and 2 were significant

factors to the predictive power of the model. On the other hand, discipline (faculty), Gender,

and Activity 1 were contradicting. Activities 4, 5 and 7 were however excluded as significant

factors for the prediction of final performances. Furthermore, a multiple regression analysis

for the performance variable with respect to each of the learning activities was performed

22
(R2=0.197; F=5.06). It was observed that the predicted performance P can be defined as

follows:

P = 5.95 + 0.97*Activity6 + 0.161*Activity8

Table 7. Multiple regression analysis 1 for performance variable.

From the model, Activity 6 emerged as a major predictor in the performances of cohort 1

students, with Activity 8 with a lesser influence. It can be argued that in the case the module

was offered again, the formula above could be used to predict student performances quite

accurately after activity 6 was over. This does not however imply that only activities 6 and 8

have pedagogical relevance, but it is mainly an observation that is obtained after applying

machine learning algorithms to the data set.

Data from Cohort 2

The system generated 195 models, among which the Generalized Linear Model

(GLM) and the Deep Learning Model (DLM) had strongest predictive power as presented in

figure 12. The correlation coefficient of both models was very close with practically equal

variances.

23
Figure 12. Second Predictive Model Generation using RapidMiner.

For cohort 2 the models had higher correlations and stronger predictive ability than

for Cohort 1. This is because the training dataset in Cohort 2 was about 4 times the dataset for

Cohort 1. This may also explain why the Generalized Linear Model (GLM) was a better fit in

terms of the predictive power. In both models, Activity 6 (concept-mapping) was the major

predictor, and it was considered as an important activity in this module, followed by activity

8.While Activity 6 would require them to build knowledge maps to demonstrate

understanding, they still had to demonstrate skills related to the use of ICT tools. Similarly,

Activity 8 was a practical activity where they had to demonstrate skills related to the use of

ICT for educational purposes. Gender was found not to have any significant predictive ability

at this stage while discipline (faculty) was a positive predictor for both models for this cohort.

The salient observation is that both models do not consider the same elements based on the

specific machine learning algorithms/regression modeling that they use. For the GLM,

Activity 4 was found to be contradicting prediction, most probably because most students

scored high marks in that activity, given that they could attempt it more than once. Activities

1 and 5 were not retained by the model as predictors. Activities 1 and 5 were pitched at

different cognitive levels whereby activity 1 was mainly at knowledge level while activity 5

required critical thinking. However, their nature was similar as they had to make posts on

forum and comments on blog posts respectively.

24
Figure 13. Generalised Linear Model (GLM).

Figure 14. Deep Learning Model (DLM).

For the DLM, Activities 4, 5 and 1 are found to be counterproductive to prediction

processes while activities 3, 6 and 8 which were including discipline were influential in the

prediction process. Furthermore, a multiple regression analysis for the performance variable

with respect to each of the learning activities was performed (R 2=0.151; F=14.8). It was

observed that the predicted performance P can be defined as follows:

P = 3.77 + 0.07*Activity2 + 0.1*Activity6 + 0.1*Activity 8 + 0.138* Discipline *

25
Table 8. Multiple regression analysis 2 for performance variable.

Discussion

The main observation from this research is that application and outcomes of

predictive analytics in e-learning is highly context-dependent and is mainly influenced by the

learning design approach adopted. This is in line with the findings of related research in the

area as reported by Smith et al. (2012) and Tempelaar et al. (2015). The outcomes of

predictive analytics also depend on the accuracy of models generated, which in fact, depends

on the amount and type of existing data available. In this research, although descriptive

analytics could provide some insights into the available data in terms of distributions and

comparative observations, the modeling of the performances did not seem to be of significant

help given the current learning design of the module. When using learning analytics to model

and predict student performances, there are a number of different variables that can be

important to consider. Some researchers have emphasized on the engagement data in e-

learning environments, also referred to as virtual learning environments (VLE) such as

frequency of access, links visited and number of clicks, navigational pattern and other related

actions of the student. In this research, it is observed that by the very nature of the learning

design, frequency of access and number of clicks would have necessarily been of value to the

performance modeling process. Others have argued that VLE information alone was not

26
sufficient for an accurate prediction of students’ performances as other learning dispositions

were important to be considered. Such learning dispositions according to Shum and Crick

(2012) would need to include self-reported data from students such as learning preferences

(or learning styles for some) and student feedback amongst others. This argument of Shum

and Crick (2012) is very relevant taking into consideration that VLE engagement data is only

relevant to the extent that a student has to spend time online when he or she is engaged in an

e-learning course. However, self-reported data from students, have also been subject to

criticisms with respect to their reliability and accuracy, especially in the context of learner

feedback and student learning styles preferences.

It can reasonably be argued that not all e-learning courses follow the same

instructional and/or learning design model that would make adoption of learning analytics in

its classic application useful from a pedagogical perspective, as evidenced by the current

module ‘Education Technologies’. This module was offered fully online using an activity-

based approach as described above. Out of the 9 activities, there were at 4 activities which

did not require full presence on the VLE for successful completion. The important element

about these activities were the instructions and supporting materials which could be

downloaded for offline browsing. While doing these activities, the students had the freedom

to look for additional resources online such as step-by-step tutorials of how to use a specific

software or YouTube tutorials. This issue has been highlighted by researchers (Bienkowski,

Feng & Means, 2012; Ferguson et al., 2016; Sharples, de Roock, Ferguson, Gaved,

Herodotou, Koh, Kukulska-Hulme, Looi, McAndrew, Rienties & Weller, 2016) and is a

challenge from an analytics perspective when it comes to limiting the definition of

engagement to VLE data alone.

27
Student performances, more precisely in the case of the education technology module,

academic performance in the final learning activity, is often looked at from different

perspectives based on the stakeholders’ lens. For instance, a student may be either concerned

with the specific mark he or she will score in a module, or sometimes they are mostly

interested to pass the module. From a teacher’s perspective, a classification might be more

appropriate such as a trend as to how many high, moderate or low performers could emerge,

and for those who are low or borderline fail, what can be done to minimize that risk of

actually happening. The key question about prediction of performances (continuous or final)

in an educational process is whether a specific mark has to be predicted, and if yes, what is

the pedagogical importance of it, or whether the aim is to better assist students in difficulty,

through prediction of the categories (high risk students, high performers or satisfactory

progress) they might likely be classified into. It is obvious that in the case of the present

research, and in line with Nuninger, Picardi, Goy and Petrone (2019), that if the goal is to

develop learner’s autonomy and development of skills and competencies, which needs to be

demonstrated by the student in the assessment process, prediction models of student

performances should not focus on the mark only.

As highlighted by Herodotou, Rienties, Boroowa, Zdrahal, Hlosta and Naydenova

(2017), predictive analytics could be applied to identify students who could be at risk of not

submitting their next assignments in an online course so that remedial cost-effective measures

could be applied. However, the module ‘Education Technologies’ was designed in such a

way that it would not require students to report gradually on the progress as long as the

submission dates were not close. Although some students would engage on forums or via

email with tutors, but this would not be enough for a close monitoring of each student. For

these types of learning activities, it would be only after assessment that one can make up an

28
idea of the involvement of the student in carrying out the activity. At that point in time,

although corrective measures can be taken by the tutors for the forthcoming activity, no

specific scaffold can be put in place within the occurrence of the activity to maximise the

chances of the student, unless the student himself or herself seeks for such support.

Furthermore, contrary to the argument of Herodotou et al. (2017), in the current situation,

only a non-submission of a particular activity could be a potential signal for a student who

would be at risk of not submitting the next assignment.

To address the above observation and concern, the findings of Rienties, Lewis,

McFarlane, Nguyen and Toetenel (2018) with respect to the fact that learning design

decisions have direct effect on student online behavior, if the learning module requires

significant time to be spent on the VLE, in particular for assessment and productive activities.

Such activities that require mainly offline work, can be redesigned in such a way that students

have to actually get back to the VLE, for e.g. to post an update on progress, or to submit sub-

tasks that ultimately lead to completion of the main tasks. In so doing, there is also a need to

revise the way the assessment criteria for such activities are conceived, as if such sub-tasks

are not included in the marking, students may not give the due importance to them.

Traditional universities are more and more adopting technology-enabled learning

environments. In the context of the current work, and specific to a such a module, real-time

learning analytics can help to address concerns about tutor commitment, learner scaffolding

and provide management at institutional level to monitor in real-time overall student

experiences to address problems and probable issues before they escalate. This might be a

more viable approach than trying to predict specific student mark in an online course. In line

with this, Viberg et al. (2018) tried to explain the decrease of predictive analytics studies and

29
applications over the recent years, with the preference going towards descriptive analytics.

Such argument is furthermore supported by Ferguson et al. (2016) highlighting the shift of

focus towards in-depth understanding of the learning processes and students’ experiences

rather than on prediction of what would be their specific performance.

Conclusion

In this study, the predictive influence of continuous learning activities on the final

learning activity mark was investigated for an online module for two different cohorts. The

findings showed that higher order cognitive and practical learning activities have a high

predictive power, given that the learning design of those activities closely match that of the

final learning activity. However, lower order activities are an important part of the knowledge

acquisition phase, but they carried less weight in terms of marking ratio. There may be a need

to rethink the learning design process. The findings of the result cannot be generalized given

the specific context that the research has been conducted. There are also limitations in terms

of applying learning analytics given the activity-based learning design approach adopted in

the module “Education Technologies”. Finally, predictive analytics is not appropriate in

modules conceived as per the learning design model adopted in the Education Technologies

module to predict specific marks or likely final performances of students. However, the

learning design can be reviewed in such a way to make learning analytics appropriate to

measure student engagement at various stages in the different learning activities so that tutors

can provide better support and scaffolding to students who are having difficulties in the

learning processes.

30
References

Agudo-Peregrina, Á. F., Iglesias-Pradas, S., Conde-González, M. Á., & Hernández-García,

Á. (2014). Can we predict success from log data in VLEs? Classification of

interactions for learning analytics and their relation with performance in VLE-

supported F2F and online learning. Computers in Human Behavior, 31, 542–

550. https://doi.org/10.1016/j.chb.2013.05.031

Banerjee, A., Bandyopadhyay, T., & Acharya, P. (2013). Data analytics: Hyped up

aspirations or true potential?. Vikalpa, 38(4), 1-12.

https://doi.org/10.1177/0256090920130401

Bienkowski, M., Feng, M., & Means, B., 2012. Enhancing teaching and learning through

educational data mining and learning analytics: An issue brief. In M. Campbell (Ed.),

Proceedings of conference on advanced technology for education, 1, (pp1-57).

Retrieved from http://www.ed.gov/technology

Biswas, A., Das, S. and Ganguly, S., 2018. Activity-Based Learning (ABL) for Engaging

Engineering Students. In Industry Interactive Innovations in Science, Engineering

and Technology (pp. 601-607). Springer, Singapore. https://doi.org/10.1007/978-981-

10-3953-9_58

31
Çakmak, M., 2016. Study in Teacher Education: Effects of Activity-Based Instruction on

Student Teachers’ Achievement. İlköğretim Online,15(3). doi:

http://dx.doi.org/10.17051/io.2016.29020

Conole, G. (2012). Designing for learning in an open world (Vol. 4). Springer Science &

Business Media; https://doi.org/10.1007/978-1-4419-8517-0

Credé, M., & Niehorster, S. (2012). Adjustment to college as measured by the student

adaptation to college questionnaire: A quantitative review of its structure and

relationships with correlates and consequences. Educational Psychology

Review, 24(1), 133-165. https://doi.org/10.1007/s10648-011-9184-5

Ferguson, R., Brasher, A., Clow, D., Cooper, A., Hillaire, G., Mittelmeier, J., Rienties, B.,

Ullmann, T., & Vuorikari, R., (2016). Research evidence on the use of learning

analytics: Implications for education policy. http://dx.doi.org/doi:10.2791/955210

Fernando, M. (2018). Pedagogical and E-Learning Techniques for Quality Improvement of

ICT Education. Advanced Learning and Teaching Environments: Innovation (p. 1).

Contents and Methods; https://doi.org/10.5772/intechopen.72203

Frank, J. R., Snell, L. S., Cate, O. T., Holmboe, E. S., Carraccio, C., Swing, S. R., . . . Harris,

K. A. (2010). Competency-based medical education: Theory to practice. Medical

Teacher, 32(8), 638–645. https://doi.org/10.3109/0142159X.2010.501190

32
Gartner Report (2012). Forecast: Enterprise software markets. Worldwide, 2011-2016, 4Q12

Update. Available from http://www.gartner.com/resId=2054422

Gašević, D., Dawson, S., Rogers, T., & Gasevic, D. (2016). Learning analytics should not

promote one size fits all: The effects of instructional conditions in predicting

academic success. The Internet and Higher Education, 28, 68–

84. https://doi.org/10.1016/j.iheduc.2015.10.002

Ghasemi, A., & Zahediasl, S. (2012). Normality tests for statistical analysis: a guide for non-

statisticians. International journal of endocrinology and metabolism, 10(2), 486. doi:

10.5812/ijem.3505

Grech, V., & Calleja, N. (2018). WASP (Write a Scientific Paper): Parametric vs. non-

parametric tests. Early human development, 123, 48-49.

https://doi.org/10.1016/j.earlhumdev.2018.04.014

Gröger, C., Schwarz, H., & Mitschang, B. (2014). Prescriptive analytics for recommendation-

based business process optimization. In International Conference on Business

Information Systems (pp. 25-37). Springer, Cham. https://doi.org/10.1007/978-3-

319-06695-0_3

Grzega, J. (2006). Developing more than just linguistic competence. The model LdL for

teaching foreign languages with a note on basic global English. Humanising language

teaching, 8(5), 1-7. Available from http://citeseerx.ist.psu.edu/viewdoc/download?

doi=10.1.1.473.2544&rep=rep1&type=pdf

33
Herodotou, C., Rienties, B., Boroowa, A., Zdrahal, Z., Hlosta, M., & Naydenova, G., 2017,

March. Implementing predictive learning analytics on a large scale: the teacher's

perspective. In Proceedings of the seventh international learning analytics &

knowledge conference (pp. 267-271). ACM. doi:

http://dx.doi.org/10.1145/3027385.3027397

Johnson, L., Smith, R., Willis, H., Levine, A., & Haywood, K. (2011). The horizon report

2011. The New Media Consortium, Austin. Retrieved from

https://files.eric.ed.gov/fulltext/ED515956.pdf

Jones, J. E., Baran, M. L., & Cosgrove, P. B. (Eds.). (2018). Outcome-based Strategies for

Adult Learning. IGI Global. DOI: 10.4018/978-1-5225-5712-8

Lehmann, T., Hähnlein, I., & Ifenthaler, D. (2014). Cognitive, metacognitive and

motivational perspectives on preflection in self-regulated online learning. Computers

in Human Behavior, 32, 313–323. https://doi.org/10.1016/j.chb.2013.07.051

Leppert, K., & Kingman, L. (2017). The rise of competency-based education. AORN

Journal, 106(6), P18–P19. https://doi.org/10.1016/S0001-2092(17)31052-9

Liñán, L. C., & Pérez, Á. A. J. (2015). Educational Data Mining and Learning Analytics:

differences, similarities, and time evolution. International Journal of Educational

Technology in Higher Education, 12(3), 98-112.

https://doi.org/10.7238/rusc.v12i3.2515

34
Lykourentzou, I., Giannoukos, I., Nikolopoulos, V., Mpardis, G., & Loumos, V. (2009).

Dropout prediction in e-learning courses through the combination of machine learning

techniques. Computers & Education, 53(3), 950-965.

https://doi.org/10.1016/j.compedu.2009.05.010

Macfadyen, L. P., & Dawson, S. (2010). Mining LMS data to develop an “early warning

system” for educators: A proof of concept. Computers & Education, 54(2), 588–

599. https://doi.org/10.1016/j.compedu.2009.09.008

Macfadyen, L. P., & Dawson, S. (2012). Numbers are not enough. Why e-learning analytics

failed to inform an institutional strategic plan. Journal of Educational Technology &

Society, 15(3), 149-163. Available from

https://www.jstor.org/stable/pdf/jeductechsoci.15.3.149.pdf

Morcke, A. M., Dornan, T., & Eika, B. (2013). Outcome (competency) based education: An

exploration of its origins, theoretical basis, and empirical evidence. Advances in

Health Sciences Education : Theory and Practice, 18(4), 851–

863. https://doi.org/10.1007/s10459-012-9405-9

Nichols, M. (2003). A theory for eLearning. Journal of Educational Technology &

Society, 6(2), 1-10. Available from https://www.jstor.org/stable/jeductechsoci.6.2.1

35
Nuninger, W. (2017, July). Integrated Learning Environment for Blended Oriented Course.

In International Conference on Learning and Collaboration Technologies (pp. 137-

157). Springer, Cham. https://doi.org/10.1007/978-3-319-58509-3_13

Nuninger, W., Picardi, C., Goy, A., & Petrone, G. (2019). Multi-Perspective Concept

Mapping in a Digital Integrated Learning Environment: Promote Active Learning

Through Shared Perspectives. In L. Bailey (Ed.), Educational Technology and the

New World of Persistent Learning (pp. 114-144). Hershey, PA: IGI Global. doi:

10.4018/978-1-5225-6361-7.ch007

Pardo, A., Han, F., & Ellis, R. A. (2017). Combining university student self-regulated

learning indicators and engagement with online learning events to predict academic

performance. IEEE Transactions on Learning Technologies, 10(1), 82–

92. https://doi.org/10.1109/TLT.2016.2639508

Patil, U., Budihal, S. V., Siddamal, S. V., & Mudenagudi, U. K. (2016). Activity based

teaching learning: An experience. In 3rd International Conference on

Transformations in Engineering Education ICTIEE. Doi:

10.16920/jeet/2016/v0i0/85433

Priyono, Wena, M., & Rahardjo, B., 2017. Using activity-based learning approach to enhance

the quality of instruction in civil engineering in Indonesian universities. In AIP

Conference Proceedings (Vol. 1887, No.1, p. 020008). AIP Publishing.

https://doi.org/10.1063/1.5003491

36
Ranganath, N. S. (2012). Activity based learning: An effective model for business

schools. The Journal of Commerce, 4(1), 17. Retrieved

from http://joc.hcc.edu.pk/articlepdf/joc201256_17_23.pdf

Reid, D.H., 2017. Competency-Based Staff Training. In Applied Behavior Analysis

Advanced Guidebook (pp.21-40). Academic Press. https://doi.org/10.1016/B978-0-12-

811122-2.00002-4

Retalis, S., Papasalouros, A., Psaromiligkos, Y., Siscos, S., & Kargidis, T. (2006, April).

Towards networked learning analytics–A concept and a tool. In Proceedings of the

fifth international conference on networked learning (pp. 1-8). Retrieved from:

https://www.lancaster.ac.uk/fss/organisations/netlc/past/nlc2006/abstracts/pdfs/

P41%20Retalis.pdf

Richardson, J. T. E. (2012). The attainment of White and ethnic minority students in distance

education. Assessment & Evaluation in Higher Education, 37(4), 393–

408. https://doi.org/10.1080/02602938.2010.534767

Rienties, B., Lewis, T., McFarlane, R., Nguyen, Q., & Toetenel, L. (2018). Analytics in

online and offline language learning environments: The role of learning design to

understand student online engagement. Computer Assisted Language Learning, 31(3),

273–293. Retrieved

from http://dx.doi.org/doi:10.1080/09588221.2017.1401548 https://doi.org/

10.1080/09588221.2017.1401548

37
Santally, M. (2012). Training of in-service educators through online activity-based

learning. International Journal of Technologies in Learning, 19(2), 1–

19. https://doi.org/10.18848/2327-0144/CGP/v19i02/49077

Santally, M. I., Rajabalee, Y., & Cooshna-Naik, D. (2012). Learning design implementation

for distance e-learning: Blending rapid e-learning techniques with activity-based

pedagogies to design and implement a socio-constructivist environment. European

Journal of Open, Distance and E-learning, 15(2). Retrieved

from https://www.eurodl.org/index.php?

p=archives&year=2012&halfyear=2&article=521

Santally, M. I., & Raverdy, J. (2006). The master’s program in computer-mediated computer

communications: A comparative study of two cohorts of students. Educational

Technology Research and Development, 54(3), 312–

326. https://doi.org/10.1007/s11423-006-8812-0

Santally, M. I., & Alain, S. (2006). Personalisation in web-based learning

environments. International Journal of Distance Education Technologies

(IJDET), 4(4), 15-35. doi: 10.4018/jdet.2006100103

Schneider, D.K., Synteta, P., Frété, C., Girardin, S., & Morand, S., 2003, December.

Conception and implementation of rich pedagogical scenarios through collaborative

portal sites: clear focus and fuzzy edges. In Proceedings of the International

Conference on Open & Online Learning,

ICOOL. http://tecfa.unige.ch/proj/seed/catalog/docs/icool03-schneider.pdf

38
Sharples, M., de Roock, R., Ferguson, R., Gaved, M., Herodotou, C., Koh, E., Kukulska-

Hulme, A., Looi, C.K., McAndrew, P., Rienties, B., & Weller, M., 2016. Innovating

pedagogy 2016: Open University innovation report 5. Retrieved

from http://hdl.handle.net/10497/18319

Shum, S.B., & Crick, R.D., 2012, April. Learning dispositions and transferable competencies:

pedagogy, modelling and learning analytics. In Proceedings of the 2nd international

conference on learning analytics and knowledge (pp. 92-101).

ACM. https://doi.org/10.1145/2330601.2330629

Siemens, G., & Long, P. (2011). Penetrating the fog: Analytics in learning and

education. EDUCAUSE review, 46(5), 30. Available from https://eric.ed.gov/?

id=EJ950794

Smith, V. C., Lange, A., & Huston, D. R. (2012). Predictive modeling to forecast student

outcomes and drive effective interventions in online community college

courses. Journal of Asynchronous Learning Networks, 16(3), 51–61. Retrieved

from https://files.eric.ed.gov/fulltext/EJ982673.pdf https://doi.org/10.24059/

olj.v16i3.275

Tan, K., Chong, M. C., Subramaniam, P., & Wong, L. P. (2018). The effectiveness of

outcome-based education on the competencies of nursing students: A systematic

review. Nurse Education Today, 64, 180–

189. https://doi.org/10.1016/j.nedt.2017.12.030

39
Tempelaar, D. T., Heck, A., Cuypers, H., van der Kooij, H., & van de Vrie, E. (2013).

Formative Assessment and Learning Analytics. In D. Suthers & K. Verbert

(Eds.), Proceedings of the 3rd International Conference on Learning Analytics and

Knowledge (pp. 205–209). ACM. https://doi.org/10.1145/2460296.2460337

Tempelaar, D. T., Rienties, B., & Giesbers, B. (2015). In search for the most informative data

for feedback generation: Learning Analytics in a data-rich context. Computers in

Human Behavior, 47, 157–167. Retrieved

from http://dx.doi.org/doi:10.1016/j.chb.2014.05.038 https://doi.org/10.1016/

j.chb.2014.05.038

Tempelaar, D., Rienties, B., Mittelmeier, J., & Nguyen, Q. (2018). Student profiling in a

dispositional learning analytics application using formative assessment. Computers in

Human Behavior, 78, 408–420. https://doi.org/10.1016/j.chb.2017.08.010

Viberg, O., Hatakka, M., Bälter, O., & Mavroudi, A. (2018). The current landscape of

learning analytics in higher education. Computers in Human Behavior, 89, 98–

110. https://doi.org/10.1016/j.chb.2018.07.027

40