INCORPORATING KAHOOT! IN CORE ENGINEERING COURSES: STUDENT ENGAGEMENT AND PERFORMANCE

Technology-enriched lessons can contribute to improving student engagement and learning in engineering courses. In this study we systematically incorporated Kahoot!, a game-based response system, in two mandatory content-intensive undergraduate engineering courses. In both courses, short quizzes were incorporated regularly at the beginning of the lesson. The questions were mostly about concepts, and students earned bonus points. At the end of the course, the students filled in an anonymous survey regarding their perceptions of the Kahoot! part of the lessons. In addition, the students’ grades in the final test were compared to their gains in the Kahoot! quizzes. We found that student satisfaction with the systematic incorporation of the Kahoot! quizzes was high in both courses. Most students were motivated to review the course content before class and about half were motivated to attend class. Most students perceived Kahoot! quizzes as contributing to their understanding of the course content. We also found a positive association between student success in the quizzes and their performance in the final exam in both courses. The frequent Kahoot! quizzes supplied continuous feedback to the students and lecturers, and the students became more engaged in the course content. Our findings indicate that technologyenriched strategies, specifically gamification techniques, contribute to increased student engagement, motivation, and performance in content-intensive engineering courses.

Harshman, Barker, Chasteen, Cole, DeChenne-Peters et al., 2018). There is evidence of the positive effect of active learning on student gains in STEM disciplines (Freeman, Eddy, McDonough, Smith, Okoroafor, Jordt et al., 2014). Yet, most lectures in engineering courses still lack active learning experiences and the continuous feedback required for enhancing students' learning and developing professional competencies needed for their future employment.
Technological tools involving gamification may enrich STEM lessons and enhance students' engagement and learning. Gamification is "the use of game design elements in non-game contexts" (Deterding, Dixon, Khaled & Nacke, 2011). Gamification techniques are a means to increase student engagement and motivation (Goshevski, Veljanoska & Hatziapostolou, 2017;Leaning, 2015) by integrating game elements in educational activities. For example, students' perceptions of motivation, attention, and learning performances were more positive when their lectures included gamified student response systems activities (Barrio, Muñoz-Organero & Soriano, 2016). Excitement and enjoyment of game-like activity act as a leverage for student engagement and motivation. A review of gamified learning in higher education (Subhash & Cudney, 2018) indicated improved student engagement and performance as the most significant benefits, and revealed the lack of research of gamification in engineering education.
Kahoot! is a game-based response system that is designed for promoting motivation in the classroom setting (Wang & Lieberoth, 2016). The students use their mobile devices to answer short quizzes that incorporate participation-enhancing elements, such as on-screen questions and distribution of answers and points, limited response time, immediate feedback, and music. Previous studies demonstrate that Kahoot! quizzes engage students in learning and provide them with immediate feedback, thereby improving the quality of learning (Barrio et al., 2016;Licorish, Owen, Daniel & George, 2018). The Kahoot! answers convey the level of students' knowledge and understanding and immediately draw the lecturer's attention to shortcomings. The immediate feedback to the students offers formative ongoing assessment.
Student engagement is an evolving concept taking many forms (Groccia, 2018;Mandernach, 2015). Earlier definitions highlighted different aspects of engagement, while recent works composed an integrated definition of engagement as a multidimensional concept encompassing behavioral, emotional, and cognitive components (Groccia, 2018;Mandernach, 2015). Specifically, Groccia (2018) described student engagement as participation, effort and persistence (behavioral), experiencing interest, enjoyment, and motivation (affective), and processing the experience and linking it to previous experiences (cognitive). He indicated that it is possible for a student to be positively engaged on one or more levels, but negatively engaged on other dimensions. Hodges (2020) suggested that behavioral and emotional engagement promotes motivation, therefore is necessary for cognitive engagement. Kahoot! quizzes may engage students behaviorally, affectively, and cognitively.
To the best of our knowledge, there is limited research on students' engagement and learning in technology-enriched gamification teaching in general and specifically in engineering courses. In this study we systematically incorporated Kahoot! in two content-intensive engineering courses. Two main research questions were examined: A) How do engineering students perceive the influence of the quizzes on their learning experience?
In particular, how do the students view aspects of their motivation, engagement, and understanding? B) How do participation and success in the quizzes relate to students' performance in the final course tests?

Course Description
Kahoot! quizzes were introduced in two mandatory content-intensive thermodynamics undergraduate (B.Sc.) courses at ORT Braude College of Engineering. The semester was 14 weeks long, and attendance was not mandatory for either course. The final grades in both courses were on a 100-point scale, with 55 being the passing grade. The first course was fourth-semester Thermodynamics at the Department of Mechanical Engineering (Thermo-M). The course consisted of two weekly sessions: a lecture of three academic hours (where an academic hour is 50 minutes long) and a tutorial of two academic hours. Eighty-nine students were enrolled and could choose between two lecture sessions and three tutorial sessions. The second course was third-semester Thermodynamics and Kinetics at the Department of Biotechnology Engineering (Thermo-B). The course consisted of two weekly sessions: a lecture of two academic hours and a tutorial of two academic hours. Twenty-eight students were enrolled (Table 1).

Kahoot! Quiz Incorporation
The courses had different lecturers, each of whom incorporated Kahoot! in their own way. In both courses, the short quizzes were incorporated regularly at the beginning of the lesson. The questions were mostly about concepts, and students could gain bonus points by answering correctly.

Thermo-M.
Kahoot! quizzes consisted of three questions and were incorporated twice a week at the beginning of each lesson (both lectures and tutorials). Two questions addressed the content of the previous lesson and one addressed an earlier topic. The questions were mostly about concepts, but occasionally an easy calculation was required (several examples are provided). A quiz winner was the student who received the highest score amongst all players, i.e., answered all questions correctly and quickly. A quiz winner was granted a bonus of one point added to the final grade. Additionally, in order to reduce student speed as a major success factor, the number of correct answers given by each student was calculated at the end of the semester, and the top ten students were granted five bonus points to their final grade. A student was eligible for up to ten bonus points.
Examples of the Kahoot! questions: • When can the (gravitational) potential energy of a gas be neglected?

Never 2. For high temperatures
Thermo-B. The Kahoot! quizzes consisted of four to six questions and were incorporated typically once a week at the beginning of the lecture. The questions addressed concepts and definitions discussed in the previous lesson. All the students who answered all the questions in a quiz correctly were granted one bonus point for their final grade up to a maximum of five points. Examples of the questions: • A reversible process is a process 1. In which the system is restored to its initial state 2. In which no changes occur in the surroundings Since success in the Kahoot! quizzes added to the final grade in both courses, the students had to identify themselves either by their true name or by a consistent nickname. The quizzes usually introduced an "action" atmosphere to the lesson. The students requested further clarifications in the case of a distribution of answers to a question, thus provoking discussion in class that clarified difficult topics or concepts.

Data Sources and Analysis
At the end of the semester the students were asked to answer an anonymous survey regarding the Kahoot! part of the course. This survey included six questions: five closed-ended and one optional openended. The survey questions were: 1. On a scale between 1 and 10, how satisfied were you with Kahoot! quizzes, where 1 is extremely dissatisfied and 10 is extremely satisfied.
2. Did you review the course content before the lesson to increase your chances of success in the Kahoot! quiz?
Only one answer could be chosen. The possible answers were: • Yes, before every class, except perhaps once or twice • I made an effort to review the course content before each class, but didn't always have time • Occasionally • No 3. Was the Kahoot! quiz a consideration in your decision whether to attend the class?
Only one answer could be selected. The answers were: • Yes, it was the primary consideration • Yes, it was a secondary consideration • No 4. Has your participation in the quizzes made you understand the course content better?
Only one answer could be selected. The answers were: • Yes, and it was the primary factor in understanding • Yes, and it was a secondary factor in understanding Students could choose as many answers as they saw fit. This question tried to assess the reaction of the students to the quizzes in a more detailed way.
The options were: •

Findings and Discussion
This section is divided into two subsections, each addressing a research question posed in the introduction.

Student Perceptions of the Kahoot! Quizzes' Influence on their Learning Experience
This section presents findings regarding the first research question: How do engineering students perceive the influence of the quizzes on their learning experience? In particular, how do the students view aspects of their motivation, engagement, and understanding?
At the end of the semester students completed a survey of five questions regarding the Kahoot! part of the course. The survey was voluntary, and the participation rate was 28% (25 students) and 43% (12 students) for Thermo-M and Thermo-B respectively.

Students' General Satisfaction
Students' answers to the survey question #1 indicated their general positive view of the Kahoot! quizzes. The average ranking in both courses was high. For Thermo-M the average ranking was 8.36, with 14 students ranking the quizzes at 10. Two graded this question with 1 and one student with 5. For Thermo-B, the average ranking was 8.58 with the lowest being 7. The high average rankings in both courses reflect the positive appreciation of the Kahoot! in the courses. These findings are similar to findings in a recent study of gamification in a problem-based learning software engineering course, in which most of the students rated the use of games in the course at the highest possible rank (Ivanova, Kozov & Zlatarov, 2019).
Students had the option to add remarks about the incorporation of Kahoot! in their course (survey question #6). Several students who had negative views of the Kahoot! quizzes pointed out two issues. The first was that they experienced this activity as stressful. They felt there was pressure on them to perform well in the quizzes and felt uncomfortable about this. The second issue regarded the timing aspect of the quizzes. The timing aspect was twofold. The first was that the winner in Thermo-M was determined by giving correct answers faster than other participants. The slower students were the reason for the five bonus points given to the best 10 students at the end of the course (Thermo-M), regardless of their speed. We assume that this remark was probably an expression of frustration by students who were less successful in the Kahoot! quizzes. The second aspect is that every question had a time limit (for most questions thirty seconds, although some questions were allotted two minutes). Some students regarded this as stressful and/or unfair. While this frustration is understandable, the time limit was inherent to the game. A conscious effort was made by the lecturers to allocate sufficient time for everyone to answer. An example of a positive remark is: "Some of the Kahoot! questions indicated possible mistakes I would have done in a stressful situation". This remark demonstrates a positive aspect in giving the students ongoing feedback during their studies. The repetitious Kahoot! quizzes provided a continuous formative assessment.

Students' Motivation to Review the Content and Attend Classes
Survey questions #2 and #3 indicated students' motivation to review the course content prior to the lesson and to attend the lessons (attendance was optional) in relation to the Kahoot! quizzes. Figure 1 presents the answers to survey question #2 by Thermo-M and Thermo-B students. As can be seen, most of the students made at least some effort to review the course content before the class (76% of the students in Thermo-M, 66% of the students in Thermo-B). The systematic Kahoot! quizzes motivated the students to devote time to content reviewing, thus they came to class better prepared. One of the students remarked in answering survey question #6: "It's a great way to encourage reviewing the course content before class. Kahoot! at the beginning of every class is a brain warm-up towards the lesson".
The students' answers to survey question #3 showed that in both the Thermo-M and Thermo-B courses the quizzes were a secondary consideration in the decision to attend class for about half the students (Thermo-M: 40%, Thermo-B: 50%). For the rest, it was not a consideration at all. No one considered it as the primary reason to attend the class. Since class attendance in the courses was optional, it was not recorded or compared with previous years, but the personal impression of the lecturers was that many students made an effort to attend on time in order to not miss the quiz.
The quizzes seemed to influence the students' behavior, according to the lecturers' observations. Some students declared that they came to class because of the quizzes. In addition, there were very few late arrivals (the quiz began precisely at the start of class). Furthermore, students would arrive at class with their phones open and ready to enter the game. Success in answering a quiz question was, in many cases, celebrated. Occasionally, the winning student took a picture of the winning result to post on social media. Some students even asked (and received) a selfie with the lecturer and the scores.
Students' answers to survey question #5 indicated mostly motivation and enjoyment, hence supporting the data gathered from survey questions #2 and #3. The answer chosen most by Thermo-M students was "Enjoyment" (chosen by 13 students), followed by "Motivation to study" (11 students) and "Competition between students" (10 students). Thermo-B students gave similar answers; the leader was "Motivation to study" (7 students), followed by "Enjoyment" (6 students) and "Competition between students" (4 students). Hodges (2020) stressed the importance of behavioral and emotional engagement as a pre-requisite for cognitive engagement. Our findings demonstrate that the consistent use of Kahoot! makes it a beneficial tool for promoting behavioral and emotional engagement. It is important to mention that we have not stated whether the competition between students was positive or negative. It appears that most of the students viewed the competition inherent in Kahoot! as positive, since the "Stress" and "I would rather not have bonus points at all" options were only chosen by a few students (6 in Thermo-M and 2 in Thermo-B).

Student Perception of Kahoot! Contribution to Course Content Understanding
Students' answers to survey question #4 are presented in Figure 2. The findings show that, by and large, students in both courses felt that Kahoot! quizzes improved their understanding of the course content. In both courses most students perceived the influence of Kahoot! as a secondary factor. However, in Thermo-B more positive answers were given (83%), compared to Thermo-M (60%). This finding differs from previous research that reported findings about Socrative, a similarly designed GSRS to Kahoot!. They found that students agreed less about whether Socrative enhanced their understanding, and practice procedures, although they appreciated Socrative as improving classroom dynamics, involvement and knowledge awareness (Coca-Méndez & Sliško, 2013). The difference in the findings may be related to several factors, one of which is the type of quiz questions. The Kahoot! questions in both courses of our study were designed to address the major course concepts and highlight important emphases. One of the students wrote (survey question #6): "I wish we would have had more questions, because I understood the principals of all the topics by answering the quizzes". The frequent Kahoot! quizzes provided immediate feedback that promoted learning, as well as an opportunity to develop understanding of important concepts. The lecturers reported that some of the quiz questions invoked discussion about topics that needed more clarification.

Kahoot! Quiz Success and Final Test Performance
This section presents findings regarding the second research question: How do participation and success in the quizzes relate to students' performance in the final course test?
Thermo-M. Twenty-four Kahoot! quizzes were given during the course at the Department of Mechanical Engineering. Practically all the enrolled students participated in a quiz at least once. Forty-three percent of the students (38 of 89) earned at least one bonus point, with one student receiving the full ten maximum bonus points. Those who received points were granted, on average, a bonus of 2.8 points. Most of the students who gained points (22 of 38) earned one bonus point.
Several correlations between receiving Kahoot! bonus points and success in the course were found. The overall failure rate in the course (i.e., final grade below 55) was 38%; but among those who received at least one bonus point, the failure rate was lower (27%).
Another finding was that the participation rate in the quizzes declined by up to 30% with the progression of the semester. This was in accordance to reduced class attendance, which is typical of this course. Thermodynamics is an intensive class, and some students opt for "marathon" type studying before the exam, while other students prefer watching video lessons in the convenience of their homes.
Thermo-B. Ten Kahoot! quizzes were given during the course at the Department of Biotechnology Engineering. Here, too, almost all students participated in a quiz at least once. Seventy-nine percent of the students (22 of 28) earned at least one bonus point, with two students receiving the full five maximum bonus points. In this course, as well, there was an association between success in Kahoot! and success in the course. Five students failed the course, and these are the same five who did not receive even a single bonus point.
The courses' final test grade averages were similar: Thermo-M mean: 63.6 (SD = 21.5), Thermo-B mean: 68.1 (SD=16.2) ( Table 2). Therefore, the ANOVA tests were performed on the combined group of 117 students from both courses.
The final test mean of the students (n=19) that gained a higher level of Kahoot! points (60% of maximum points and above) was significantly higher (mean=81.5 SD=14.5) than the final test mean of the students (n=41) that gained fewer Kahoot! points (mean=66.1 SD=20.2).
The differences presented between students may be partially attributed to the fact that successful students also performed well during the quizzes. However, with more than a third of the class earning bonus points in Thermo-M and more than two-thirds in Thermo-B, it is not likely to be the only explanation. From the lecturers' observations and the survey answers, it seems that students felt the urge to participate and compete with their peers, which encouraged many of them to invest at least minimal effort in preparing for the quizzes. One of the students wrote (survey question #6): "Some Kahoot! questions pointed to certain content that I need to learn better. The combination of the Kahoot! quiz format and the atypical questions creates motivation to delve into the content". It seems that students were faced with formative assessment that encouraged them to take responsibility for their learning. We suggest that consistent Kahoot! participation influenced, to a certain extent, the quality of their performance in the final exam.
In many cases the quiz encouraged question-related discussion. This usually happened for Kahoot! questions where there was a distribution of answers. In many cases the students who did not choose the correct answer wanted to know why their answer was incorrect. Such questions provided an opportunity for discussion that clarified important issues. The quizzes also allowed the teacher to assess the level of student comprehension, pointing to common challenges and shared understandings. Sometimes a question about a topic that was expected to be understood would show less than the satisfactory number of correct answers, indicating a weak point that required more attention. In parallel, a question that many students answered correctly was usually a good indication of satisfactory understanding of a specific concept.

Concluding Remarks
Technological tools such as Kahoot! provide opportunities for active learning in engineering classes. In this research student satisfaction with the systematic incorporation of the Kahoot! quizzes during class in two thermodynamics courses was high. The possibility to obtain bonus points was perceived by many students as positive and viewed as a motivational and/or an enjoyable part of the course. Only a small minority viewed this activity as negative. The difference between the courses' maximum bonus points did not seem to matter. Both lecturers described a positive influence on students' behavior and engagement during classes. The students in both courses became more active learners. Most students were motivated to review the course content before class and to attend class at least to some degree. These findings are in accordance with former studies of Kahoot! (Leaning, 2015;Licorish et al., 2018) that noted an extended student engagement with their peers, the lecturer, and course content.
A substantial number of students earned Kahoot! bonus points in both courses. Most students suggested that Kahoot! quizzes improved their understanding of the course content, albeit their perception of the Kahoot! contribution was as a secondary factor. There is a positive association between student success in the quizzes and performance in the final exam in both courses. This finding is in accordance with former research that found a match between students' high final grades and good Kahoot! scores (Aleksic-Maslac, Sinkovic & Vranesic, 2017). A comparison of students' final exam grades of these courses with results from previous years was not possible in this study. Further research is needed to explore the influence of consistent Kahoot! quizzes on students' learning process and achievements.
We believe that the systematic incorporation of the gamification platform Kahoot! improved the learning process in both courses. The frequent Kahoot! quizzes provided formative assessment. The students and lecturers received continuous and immediate feedback and the students became more engaged with learning the course content. Previous studies suggested that behavioral and emotional engagement promote motivation, therefore is necessary for cognitive engagement (Hodges, 2020). Gamification techniques are a means to increase student enjoyment and motivation to participate and prepare for a game-like activity. Our findings suggest that Kahoot! quizzes systematically used in mandatory intensive-content undergraduate engineering courses may engage students behaviorally, affectively, and cognitively.
Our findings support the claim that technology-enriched strategies, specifically gamification techniques, contribute to increased student engagement, motivation (Goshevski et al., 2017;Leaning, 2015), and performance (Fabregat-Sanjuan, Pàmies-Vilà, Piera & de la Flor López, 2017; Subhash & Cudney, 2018). This claim is demonstrated in mandatory content-intensive thermodynamics courses in two different disciplines of engineering education. Engagement and motivation are inherent to students' learning and performance. Therefore, we call for other researchers in STEM courses at large, and engineering courses specifically, to incorporate Kahoot! or similar technological tools involving gamification to further develop our knowledge and understanding of these tools' usage and impact in a specific context.