ENHANCEMENT IN EVALUATING SMALL GROUP WORK IN COURSES WITH LARGE NUMBER OF STUDENTS . MACHINE THEORY AT INDUSTRIAL ENGINEERING DEGREES

This article examines new tutoring evaluation methods to be adopted in the course, Machine Theory, in the Escola Tècnica Superior d’Enginyeria Industrial de Barcelona (ETSEIB, Universitat Politècnica de Catalunya). These new methods have been developed in order to facilitate teaching staff work and include students in the evaluation process. Machine Theory is a required course with a large number of students. These students are divided into groups of three, and required to carry out a supervised work constituting 20% of their final mark. These new evaluation methods were proposed in response to the significant increase of students in spring semester of 2010-2011, and were pilot tested during fall semester of academic year 2011-2012, in the previous Industrial Engineering degree program. Pilot test results were highly satisfactory for students and teachers, alike, and met proposed educational objectives. For this reason, the new evaluation methodology was adopted in spring semester of 2011-2012, in the current bachelor’s degree program in Industrial Technology (Grau en Enginyeria en Tecnologies Industrials, GETI), where it has also achieved highly satisfactory results.


INTRODUCTION
Machine Theory is a required course, originally taken during the fourth semester of the former Industrial Engineering degree program at ETSEIB.The course is taught twice a year (during the fall semester -FS-and spring semester -SS-), and has a 6 credit teaching load (5 ECTS).4,5 credits are taught in a classroom with 60 students, and the remaining 1,5 credits taught in laboratory sessions with 15 to 20 students.Lab sessions include hands-on assignments, and a supervised, small group work, as specified in the Machine Theory Informational sheet (2010( -2011 SS) SS).The supervised work is carried out in groups of three or four students that have three tutored sessions of 1,5 hours, over the course of the semester.The supervised work must be written up in a 10-page report, and orally defended at the end of the semester before a jury of two professors from the program.The oral defence is a maximum of 30 minutes long.The work grade constitutes 20% of the studentsf inal grade in the course.If a student repeats the course, the work grade can be maintained for 2 consecutive semesters, if the student wishes.
Successful completion of this work involves specific competencies in the subject, as well as generic competencies, such as team work, efficient oral and writing communication.
Since academic year 2000-2001, Machine Theory has always included a supervised, small-group work developed as previously explained.This work has always achieved teaching staff objectives, and a fixed for teachers' staff.The percentage of students receiving a passing mark over this 10 year period is 94,3% (Figure 1).During spring semester of 2011, the number of enrolled students increased from an average of 230 students (between new and repeating students) to 350 students (285 new and 65 repeating).While in the previous years there had been 60 groups with an average of 3 students in each group, in spring of 2011 there were now 102 groups, with an average of 4 students in each group.This resulted in a greatly increased teaching load, and greater difficulties in supervising work reports.This in turn led to the need for improved coordination among professors and more precise and objective criteria for evaluation.
The new evaluation methodology was used during fall semester of 2011-2012 for students in the former Industrial Engineering degree program, and was planned as a pilot test for the new course, Mechanism and Machine Theory (6 ECTS), offered in the fourth semester of GETI (Grau en Enginyeria en Tecnologies Industrialsbachelor's degree program in Industrial Technology), which started in spring 2012.The new course substitutes the former course, yet is taught in a similar environment, with similar content and objectives.

Actions
In order to achieve work-related competences and objectives, as well as facilitate coordination among teachers, new strategies for carrying out and evaluating student tasks were developed.These strategies included: • Changing the number and the content of tutored sessions.
• Adjusting the content and extension of the work.
• Changing the evaluation method.The strategies were designed with the idea that neither the quality nor number of passing reports would be diminished, and that student motivation, dedication, and responsibility would improve.
For student feedback on the new methodology, a Student Evaluation of Educational Quality Questionnaire (SEEQ) was handed out to students during the last laboratory session, and a meeting set up with teaching staff.

Description
On July 2011, Machine Theory teaching staff decided to increase the number of tutored sessions from 3 to 6 in order that students do the bulk of their work during tutoring rather than as homework.Although an increased number of tutored sessions meant a heavier teaching load, it was believed that there would be a decrease in orientation tutorial effort on the part of teachers, and fewer appointments outside the tutored sessions.As a consequence, teachers would experience fewer interruptions to their other teaching responsibilities, and could dedicate less hours to evaluation juries.
An increased number of tutored sessions meant that student work may be more closely supervised.According to Del Canto et al. (2011), mistakes can thus be detected and solved early on before delivering the final report.
In order that students clearly know their duties, and in order to set the contents of the tutored sessions, the document Work tasks (Table 1) was written.This document contains an explicit plan with all the tasks that students must do during the tutored sessions and for homework.

P1
The mechanisms of a sewing machine.

T3
• Revise, if it is necessary, the PAM simulations.
• Begin the PAM dynamic simulations.
• Print the graphs and the revised schematic mechanism diagram.
• Check that all the variables used appear in the revised schematic mechanism diagram.

T4
• Deliver the graphs and the revised schematic mechanism diagram.• Continue with the PAM dynamic simulations.
• Present the completed work to classmates.
• Begin the writing report.
• Read the written report peer-assessment rubric.
• Write the report.
• Print the report first version.13 14 T5 • Deliver first draft of report.
• Evaluate the classmate reports.
• Upload on ATENEA in pdf format the final report and the PAM file.• Print the final report.
• Prepare the oral presentation.

T6
• Deliver a printed copy of the final report.
• Give oral presentation of work.
• Evaluate classmates' oral presentation.The document Guidelines for the writing report (Table 2) was also written, inspired in Cardona and Jordi (2003) and also in Singhose and Donnel (2009).This document explains the required chapters that the final written report must include with a brief explanation.Also the extension of the written report was fixed in 6 pages as maximum.All the documents are available as of the first day of the course on the Digital Campus.
The Machine Theory written report has a maximum length of 6 pages (including cover Reflect clearly and in orderly manner the deductions done with the obtained results.
Table 2. Guidelines for the written report.
The regular evaluation methodology was changed in order to include the learning process of the students as suggested in ICE-UPC work (2008a) and Andrade (2005) but moreover considering the teaching resources during the last week of the course.As mentioned previously, the work represents 20% of the final mark for the course.The evaluation method used since 2010-2011, and the new proposal for the fall term of 2011-2012, are summarised in Table 3.
In the new evaluation system, the two first deliveries are maintained, as they are effective in providing timely feedback to students.The significance of these deliveries is not the grade, but rather the comments received from the tutor and fellow classmates.For this reason, some weeks later (week 12), the first delivery is given again with corrections.This action search a manner to give feedback to the students about their learning process as Gibbs and Simpson (2009) suggest as an effective action.
According to ICE-UPC reports (2008b) that recommend that each learning objective must be evaluated, the new proposal assigns different weights to written report and oral presentation (see Table 3).Therefore, every specific objective is assigned a specific weight in the final grade, depending on its relative importance, according to teachers' criteria.
Peer assessment is done using two rubrics: a Written Report Peer-Assessment Rubric and an Oral Presentation Peer-Assessment Rubric, available on the Digital Campus as of the first day of the course.These rubrics may also be used for self-evaluation.In addition, students may write explicit comments when evaluating classmates´ reports by using the Writing report rubric complement.As Álvarez-Méndez ( 2001) points out, the use of rubrics ensures reflection, content-specific learning, and requires students to explain and defend their evaluation scoring.The rubrics evaluate specific aspects of the report as either "very good" (a score of 8-10); "satisfactory" (a score of 5-7); or "unsatisfactory" (a score of 0-4).The rubric Written report peer-assessment (Table 4) (and its complement) evaluates the schematic diagram, the presentation, the objectives set, the obtained results, and the conclusions reached.The Oral presentation peer-assessment rubric (Table 5), on the other hand, is focused on presentation content and organization, and on verbal and non-verbal skills of the students.

Schematic diagram
All the schematization criteria are followed.The schematic diagram presents all the generalized coordinates used in the graphs, and dimensions define completely the mechanism.Some of the schematization criteria are not followed.Some missing parameters do not allow the complete definition of the mechanism.
The schematic diagram does not follow much of the schematization criteria and some missing dimensions do not allow the complete definition of the mechanism.

Writing presentation
The report follows the required structure and the typography is consistent throughout.Magnitudes and units are presented as Magnitude [unit] in the graphical axis.There are neither orthographic nor syntactic errors.
The written presentation is not sufficiently precise and the typography is not consistent.In some graphics the magnitude and/or the units are not correctly presented.There are some spelling and/or syntactical errors.
The report does not follow the required structure.The data are not clearly presented in the graphics.The units do not belong to the International System.Some contents are missing.There are frequent spelling and/or syntactical errors.

Objectives
The objectives are clear and concise, and relate to the results.
The objectives do not clearly define the work done.The objectives are a bit ambiguous and could be more specific.
The work does not present objectives or the objectives do not match with the work carried along the report.

Results
The simulation conditions appear.The graphics clearly show the results and support the relations and the data obtained.The results are related with topics from Machine Theory.There is no relationship between the conclusions and the objectives.The conclusions are not supported by the results of the work.

Not all the simulation conditions appear. The graphic analysis is wrong and leads to incorrect arguments. There is a lack of topics related with
Order the reports according to their global quality (from the best to the worst) including the own one.
Before announcing the results, the students are asked to evaluate their own work, using the same rubric, and to then order the reports (including their own) according to quality.The evaluation and the classmates' comments are given anonymously to each group.Each group then critiques its work, and then compares the grade they assigned themselves, with the marks they received from their classmates.The complementary document to the Written Report Peer-Assessment Rubric allows students to know the reasons for the marks received, what areas of the work were well done, and what can be improved.This evaluation approach provides each group direct and specific observations about their work.The mistakes can be corrected easily by the students.
The process of self evaluation and peer-assessment is carried out during the next to the last tutored session (Table 1).This allows students an opportunity to improve their reports before turning in them in, and potentially raise their final mark.
Using a rubric for the oral presentation has two main purposes: 1) that students know the aspects they are going to be evaluated on during their oral defence, in order to be able to better prepare their presentation (it is worth noting that this is the one of the first oral presentations students enrolled in this degree program must give); 2) that students succeed in holding the attention of the rest of the class during their presentations, because the students have to evaluate the presentation of the others.These objectives are in agreement with Blanco (2008).

Content
The ideas presented are related to the topic and are clearly presented.The objectives, the main ideas, and the results are clearly highlighted.
In some parts of the presentation the structure does not clearly related to the objectives.The conclusions do not highlight some of the main ideas of the work.Some results are not clearly presented.
The ideas presented have little relationship to the topic, are unclear, briefly explained, and subjective.Many ideas are repeated.The main ideas and results are not highlighted.

Presentation sequence and organization
The details appear in a logical sequence.The presentation is coherent and the transition between ideas is fluid.
The presentation is not well organized.Ideas are presented isolated without fluency along the presentation.
The details lack organization.The presentation is incoherent.

Verbal skills
The language is technical and appropriate.The information is clear, precise and vocabulary is used.
There are some mistakes in the language.The vocabulary is not rich and lacks precision.
The technical language does not suit the presentation.The group has little knowledge of the topic.The presentation has not been tested previously.

Non verbal skills
Group attitude is appropriate.The group shows confidence in its work.The presentation has been tested previously.Order the presentations according to their overall quality (from the best to the worst) including the own one.
Once the new methodology was implemented, the teaching staff evaluated it in order to identify weak points and introduce the following actions.
• Insist on the need for students to be organized into groups of 3 and to attend the first tutored session with a chosen mechanism.This has been done during the two first laboratory sessions.
• Move the two last tasks of the 2 nd tutored sessions to the 3 rd session.
• Incorporate audiovisual tools for the oral presentation, using a maximum of 4 slides.
These actions were applied to spring semester 2011-2012 for the course, Machine and Mechanism Theory of GETI.
At the end of the semester, a short survey was filled out by GETI students in order to get feedback on the new implemented actions.These opinions were useful for perfecting methodology.

RESULTS AND DISCUSSION
In this section, the results obtained with the traditional methodology (data from SS of the academic year 2009-2010), the pilot test (data from FS of the academic year 2011-2012), and the consolidation phase for the GETI course (data from SS of the academic year 2011-2012) are discussed.
Figure 2 shows the marks obtained with the traditional methodology (an average grade of 7,29) and Figure 3 shows marks on the pilot test (an average mark of 7,83).It is observed that failing marks for the supervised work disappear, and the distribution grade is substantially modified, although the mean grade is fairly similar.Figure 4 shows the results obtained during the consolidation phase.The comparison between Figure 3 and 4 shows an improvement in marks and continued elimination of failing marks.Moreover, the mean mark in Figure 4 is 8,70, and the mode is 9,0 (substantially higher than any other semester).
The fact that student tasks (with the new methodology) are more closely supervised has resulted in reports having similar formats to one another, and has caused some loss of the creativity and originality present with it with the traditional methodology.This drawback is tolerated since the course is given in the early stages of the degree program (fourth semester), where the autonomous learning should be guided.Later in the degree program, when learning is less guided, students can develop greater skills in originality and creativity.The autonomous learning phase can be initiated when students undertake their bachelor´s final work.The new methodology has the advantage to remove reports too large or too short and has helped to harmonize the criteria of corrections.
Peer-assessment of the first draft report has proven to be an effective tool for early detection of errors.This fact is directly related to the elimination of failing marks, which has meant that all reports have met the minimum requirements for each part.
Next, the results obtained during the pilot test and consolidation phase (two consecutive semesters) are discussed.
The results of the written report evaluations show that the marks given by teachers for the final reports are higher than those given by classmates.For the pilot test phase, 74% of the reports received higher final marks than the one given by the classmates, while during the consolidation phase the percentage was 73% (Figure 5).
It should be highlighted that during the pilot test, 35% of the reports have a final mark 1 or 2 points higher than the mark given by the classmates.During the consolidation phase, this percentage increases until 41%.This fact demonstrates that the initial report submitted to peer-assessment had been modified and improved before being evaluated by the teachers.Figure 5 also shows that in the pilot test, 26% of reports received a final grade that was lower than the grade given by classmates.In the consolidation phase, 27% of reports received a final grade that was lower than that assigned by classmates.
Figure 6 shows the teacher´s mark (TM), and the classmates´ mark (CM) for each report.It can be seen that reports with a TM higher than 8,5 normally obtain lower CM, while reports with TM lower than 7 obtain better CM.This fact reinforces the idea that it is hard for students to give extremely high or low marks on a scale of 0 to 10, though they can distinguish between correct and incorrect work.Furthermore, groups that obtain the highest mark from classmates, also obtain the highest mark from the teacher.These groups are the most critical to the classmates' work and they fill the Writing report rubric complement with a lot of observations.
The number of students instructed with the traditional methodology and pilot test methodology is very low, as few students repeat the work.Some of these students were required to fill out a SEEQ questionnaire.Student comments were: • New methodology facilitates the realisation of the work, although it reduces originality and freedom in preparing the written report.
• Evaluation criteria are more objective and known since the first day of the course.Rubrics clearly show what is evaluated in every section of the report.
• Peer-assessment provides students an idea regarding the quality of their work, and allows students to correct common mistakes such as spelling errors, incorrect usage of the decimal separator or given magnitude without the units.
• Mistakes detected by the classmates can be easily solved and the work improved.The modifications required affect mainly format and orthography but not the basis of the work.
• Being able to turn in the report twice increases chances of obtaining better marks.In order to collect more feedback about the new methodology, the GETI students filled out a longer questionnaire with open answers.Some of their comments were: • The number of tutored sessions and their content are adequate.
• The documents available on the Digital Campus are useful and help to know what is going to be done during the tutored sessions and what the homework is.
• The rubrics are good templates for the written report but also for the oral defence.
• The peer-assessment process is positive.It allows students to detect mistakes and improve the quality of the final report.
• The peer-assessment of the oral presentation is not so useful as it does not help to improve the final presentation.
• The peer-assessment process is not always objective.The affinity between groups is reflected on the marks given in the rubrics.However the objectivity is maintained in the Writing report rubric complement.
• The slides are mandatory for a good oral exposition and facilitate comprehension of classmates' work.Responses to the rubrics and the questionnaire show that students feel motivated, and dedicated to the peerassessment process.
The obtained marks during the pilot test are excellent.Nevertheless, it must be pointed out that GETI students in the consolidation phase have better average marks than students in the pilot test.The mean during the previous 10 years was 73% for passing students (including new and old students) while the mean for GETI students is 87%.
For teachers, the reform has allowed them to split the work during the semester.Before the reform, each teacher taught 4,5 hours of tutored sessions, divided in 3 sessions, and attended a mean of 10 hours of oral defences, leaving aside appointments outside the tutored sessions.With the reform, the teachers teach 6 tutored sessions of 1,5 h hours each (the oral defence is done in the last tutored session).Also the hours devoted to reading the reports have been reduced with the new methodology thanks to the 6-page limit.The reform also considers that two teachers are involved in the evaluation of the oral and written report.

CONCLUSIONS
On the basis of only two consecutive semesters adopting the new methodology in tutored sessions, authors conclude that the new methodology has been received positively by both students and teachers.The results show the benefits of working with greater guidance: 1) hours devoted to the report are spread out better for both students and teachers; 2) required objectives are clearer to the students; 3) the teacher evaluation process is more objective; 4) rubrics are a good tool for identifying evaluation criteria; and 5) the peerassessment process helps students recognize the quality of their work, and be able to critique classmates' work.
It should also be pointed out that: • Students working under greater guidance know what is expected of them at each step of the report process, and are better able to correctly complete their homework.
• While students feel the new methodology eliminates some of the freedom enjoyed under the old system, they consider the new methodology to be better overall, as it allows students to know the evaluation criteria from the first day of the course, and to turn in two deliveries of the written report.
• Though it was not one of the aims of the new methodology, the authors conclude that students prefer working with templates, even if it implies a loss of creativity.
• The number of passing students rose to 100% (in both semesters).The marks are more even within groups: it is harder to obtain a 10 but it there are fewer lower marks.
• Peer-assessment and self-evaluation have proven to be effective tools, and they have demonstrated students' dedication and motivation, as well as their critiquing skills, particularly in the Writing Report Rubric Complement.
• The students have learned that critiquing classmates' work is a duty in the learning process, and that it increases the group´s sense of responsibility.
From the teachers' standpoint: • The teachers have a more balanced teaching load during the semester, while maintaining teaching quality.
• The results suggest that with the new methodology, report objectives were achieved as well or better than with the traditional methodology.
• Lastly, considering GETI student comments, possible improvements could include greater precision in certain aspects of the rubrics, and providing templates for slides used during the oral presentation.

Figure 4 .
Figure 4. Distribution of work marks during spring semester of the academic year 2011-2012 (consolidation phase).

Table 1 .
Work tasks plan for the FS 2011-2012.
Vol.3(1), 2013, pp 13 Describe the chosen mechanism for the report and explain, if required, the simplifications done.2.Objectives (half page approximately)Mark clearly and precisely the objectives to study.3.Schematic diagram and mechanism dimensions (maximum 1 page)Present the schematic mechanism diagram using a CAD program or similar.The Machine Theory schematization criteria must be used.Also the Graphical Expression basic criteria must be used ).It must include: Cover.The minimum information that has to appear on the cover is: report title, course title, students name, tutor name, group number, academic year and semester.Parts: 1. Mechanism description.Adaptations and simplifications (half page approximately) (line's width, avoid the crossing of dimensions...).Represent in the schematic diagram all the variables (generalized coordinates, forces, torques...) that appear along the work.Include all the required values of the parameters (dimensions, masses, forces, springs' stiffness or dampers...) to define only the mechanism.4. Results (maximum 2 pages) Present the graph results (kinematic and dynamic variables) indicating in each case the actuators' characteristics (for each actuator: controlled coordinate, type of function and its parameters...), type of study realised and, if it is required, time of simulation.Pay attention to the visual aspect of the graphs (line's width, grid...) and indicate the magnitude and the units represented as follows magnitude [units]. 5. Discussion (approximately half page) Include, if it is considered, this part.From the graphs it is possible to observe dead points, tendencies, plausible or impossible results, validity of the results... 6. Conclusions (approximately half page)

Table 3 .
Weights of each delivery in work evaluation.