EVALUATION OF SOCIAL COMPETENCIES IN CHEMICAL ENGINEERING: APPLICATION AND RESULTS OF THE PILOT TEST (ACADEMIC YEAR 2012-2013)

The Escola Tècnica Superior d’Enginyeria Química has a long traditon in the deployment of social competencies in engineering curricula through Integrated Projects (IP) carried out in structured teams. Social competencies are taught and practced during the development of the IPs. We conceptually introduce a methodology for a 360 assessment of the students’ social competencies, as a tool to foster the improvement of their competency levels. In this artcle we analyze the results of the pilot test where the aforementoned methodology has been implemented in the Bachelor studies of Chemical Engineering. The results indicate that it is possible to objectvely obtain the student’s competency level discriminatng among diferent social competencies, as well as among diferent students in the same team. The applicaton of this tool fosters the development of specifc educatve actons to help the students with low competency profle, to reach acceptable levels for a successful inserton in the labor market.

Although Spencer's defniton is neither unique nor probably the best, it indicates that the individuals that excel in their functon are not distnguished from the average only by their knowledge or their experience. Other capacites, many of them of social nature, also play a very important role.
Hence, in order to respond to the new paradigm, one has to incorporate social competencies in the engineering curricula. A consequence of this fact is the need of providing an adequate feedback to the students. Therefore, it is essental to objectvely evaluate their competency level during the studies, to push them towards becoming outstanding professionals in their future jobs.
The educatonal model of the Escola Tècnica Superior d'Enginyeria Química (ETSEQ) is based on Integrated Projects (IP) in which the students work in structured teams on open design projects related to Chemical Engineering and Food Engineering (Wit, Alabart, Giralt, Herrero, Vernís & Medir, 2006). In this context, students receive specifc training about social competencies and interact in an adequate environment to practce them. Social competencies are refected to the individual behavior (Suñé & Bonet Avalos, 2012). The most signifcant behaviors will take place in quasi professional situatons that occur during the IP. Hence, it is essental to observe the behavior of the individuals while they are actng in their roles, in order to determine the competency level. Therefore, the intrinsic difculty of assessing social competencies lies in the need of an objectve and reliable procedure based on direct observatons of the behaviors.
In this artcle we discuss the procedure implemented in the ETSEQ to observe and assess these social competencies among the students. We also discuss the results of the pilot test carried out in the ETSEQ during the second semester of the academic year 2012-2013 of the Bachelor studies of Chemical Engineering (GEQ) and Food Engineering (GEA), which show that an objectve assessment of social competencies is possible.

METHODOLOGY 2.1 Background
From the beginning of 2012, the ETSEQ has developed and implemented an evaluaton methodology to assess the fve social competencies included in its competency dictonary. These fve social competencies are described in the memoranda of the Bachelor studies (GEQ, 2009;GEA, 2009), and are listed in Table 1. However, these competencies, as they stand, are meaningless if no additonal explanaton is provided. In Table  2 we give the defniton of competency B1 in terms of behavioral descriptors, which are further deployed through a list of observable elementary behaviors related to the later. We will return to this point in secton 2.3. In summary, the set of social competencies B1-B5 need 22 behavioral descriptors, with 37 elementary behaviors, which are ultmately evaluated one by one.

Context
The competency assessment has been implemented in the subjects where the IP's, mentoned in the Introducton, are carried out. The IP implies building balanced teams using the principal profles of the students. Such profles can be obtained from questonnaires of the type of Belbin (Belbin, 2014), together with relevant informaton about them like grades, geographic constraints, etc. Within these teams, we diferentate two roles: leader and team member. Professors of the frst year play diferent roles. On the one hand, a professor can be a consultant and address the questons posed by the students on a given subject. On the other hand, the same professor acts as a client because he/she will grade the fnal report at the end of the IP. Another professor plays the role of coach to support the team during the development of the IP, without interventon in the evaluaton. The goal of the IP is proposed by the coordinators, which act as sponsors.
The IP of the frst academic year involve students of the frst year and students of the fourth year, enrolled in the electve subject Team Leadership Practce (PLE). In this partcular IP, the leader of the team is one of these PLE students. He/she should lead from 5 to 7 students of the frst year who act as team members (Wit et al., 2006). The professors of PLE coach the leaders during the IP, grade their performance in the subject, and also assess their social competencies. A scheme of the organizaton of the IP of the frst year is shown in Figure 1. In the IP of the second and third year, the leader is a student belonging to the same academic year, who also plays a signifcant role as a team member.
The team has to set a Team Charter in order to determine the objectves, the team rules, the planning of the tasks, the deadlines, the deliverables, etc. The leader and team members have to meet at least once a week according to a schedule established at the beginning of the academic year.

Implementaton of the methodology of evaluaton of competencies
According to Postulate 2, one has to defne a standard for each role. The standard has to be related to the competency dictonary given in Table 1. Such a standard is deployed as a collecton of rubrics, bound to the behavioral descriptors, which defne each competency, as in Table 2. Hence, the standard materializes in a questonnaire to be used for all the observers to assess all the individuals actng within the same role. In our case, we have chosen these rubrics to be descriptons of elementary behaviors, also known as pinpointed behaviors, easily identfable in the student's behavior during the IP. When the workload is high she/he reviews the work plan with the team and resets priorites.

Displays resilience.
Is able to cope with setbacks and disappointments while focusing on moving the team forward.
In the case of a confict, she/he faces the situaton by fnding a soluton with win-win scenarios. Understand confict resoluton methods and applies them efectvely. The students are provided with forms containing the questonnaires with the complete list of competencies, behavioral descriptors, and the associated pinpointed behaviors for each role. In these questonnaires the pinpointed behaviors have to be evaluated one by one by the observer, using a scale with three values: 0, 1, 2, depending on whether a given elementary behavior has been seldom observed (0), sometmes (1), or ofen (2), if it denotes frequency, or low (0), medium (1) or high (2) if it denotes intensity. If circumstances are not appropriate for the observaton of a specifc elementary behavior, the observer leaves the evaluaton of the item blank. At the end of the year, the average of the observatons for each elementary behavior is determined. Collectng these average values of the elementary behaviors for a given competency one obtains the competency level from their mean on a scale from 0 to 2. The level is fnally scaled up to a range between 0 and 10 for the ease of interpretaton. The student fnally receives a qualitatve assessment of her/his competency profle in terms of superior performance (8.25-10), average performance (6-8.25), needs improvement (4-6), and needs serious improvement (0-4).
The process of evaluaton starts with a training session with the students in order to explain to them the methodology step by step. From this moment on, the students have access to the rubrics and a detailed guide with all the informaton required.
The dynamics of the evaluaton then proceeds as follows. The team members have to evaluate once a week his/her teammates (cross-evaluaton), perform the self-evaluaton, and also evaluate the leader. To reduce the evaluaton burden, however, the competencies to be evaluated are distributed among the team members so that each team member evaluates all the other team members with respect to one single competency (maybe two, depending on the size of the team), while another team member evaluates the leader. These roles rotate among the team members according to a calendar that the leader establishes at the beginning of the project. The leader has to evaluate individually her/his team members once a week, and do the self-evaluaton with the same periodicity. The leader is furthermore evaluated twice a month by the PLE professor. In the example of the team member, the relatve weight of each type of evaluaton is approximately the same (1/3).

RESULTS
To check the feasibility of the procedure, we conducted a pilot test during the 2nd semester of the academic year 2012-2013. The pilot test involved one team of the integrated project (GEQ/GEA) of the frst year (the leader was a student of the fourth year), one team of the fourth year (involving several students of PLE of the GEQ) and one team of the second year (GEA). A total of 17 students, 3 of them as leaders, and the rest as team members, were involved. The procedure allowed us to collect 70 observatons related to the role of leaders and 232 observatons for the role of team members; that is, 23 observatons/leader and 17 observatons/team member.
Using the data collected we did an analysis to validate the methodology and the rubrics, namely, what we refer to as the instrument. The objectve was to determine whether the instrument is a) statstcally signifcant, b) consistent with the ability to identfy the superior performance, c) discriminates between individuals and competencies within the same individual, and d) feasible in our educatonal model. We have done the statstcal analysis (a) using the matrix of correlaton coefcients (Pearson, 1920;Pearson, 2015) and the Cronbach's alpha (Cronbach, 1951), widely used in psychology and social sciences. The consistency (b) of the instrument was determined by comparing the results of the instrument with the qualitatve appreciaton of the students at the end of the process. The qualitatve appreciaton is based on the subjectve impression that the observers obtained during the year. The observers were asked to separately evaluate only the competencies (not the elementary behaviors) on a scale from 0 to 10, considering all the period of the pilot test. Additonally, we demanded an external evaluaton of the leaders from the supervisor of the internship in industry. We have used this evaluaton to compare the results with those obtained in the pilot test and the qualitatve appreciaton. These comparisons allow us to say whether these projectons of the evaluatons are consistent with the results obtained from the instrument and the qualitatve evaluaton.
Firstly, the partcipants measured the tme spent in completng the corresponding questonnaires (within the forms) afer the weekly meetng. We found that each team member spent about 10 minutes to evaluate 2 competencies of her/his remaining 4-5 teammates together with the self-evaluaton. The leader spent on the order of 20 minutes to evaluate the 5 competencies of her/his 5-6 team members and to complete the selfevaluaton. In turn, the professors of PLE spent 5 minutes to evaluate one leader for the 5 competencies.
In Figures 2 a) and b) we show the competency level of two members of the fourth year team (GEQ). In blue, Vol. 5(2), 2015, pp 145 we indicate the results obtained directly from the instrument. In red, we show the qualitatve appreciaton and, in green, the evaluaton of the internship (summer 2013, ranging from 2 to 6 months). In this later case, we established a correspondence between the items evaluated in the internship (initatve, communicaton, responsibility, cooperaton, etc.) and our social competencies, in order to extract a projecton as an estmate of the competency level with regards to the dictonary given in Table 1 (Suñé, 2015).
As we observe in both Figures 2 a) and b), the three estmatons of the competency level cast a very similar profle. Considering the case of Figure 1a), the qualitatve evaluaton is numerically equal for the competencies B1, B2, and B5, but shows deviatons in B3 and B4. However, taking into account its nature, this evaluaton tends not to sharply discriminate between the competencies. Rather, it responds to an overall impression of the observers about the individual evaluated at the end of the IP. Regarding the internship, the competencies with higher scores are B3 and B4, together with B2. Instead, B1 and B5 are lower rated. Hence, we can say that the internship evaluaton casts a similar profle as the other measures. In Figure 1 b), the evaluatons are qualitatvely coincident.

Figure 2. Competency level of two members, a) and b), of the team of fourth year (GEQ). Comparison between the results of the pilot test, the qualitatve evaluaton and the internship evaluaton
Generally, we observe a good correspondence between the evaluatons. However, in some cases, the evaluaton from the supervisor of the internship overestmates the competencies (not shown), mostly because the person flling in the evaluaton form is not the person that has directly worked with the student during the internship.
In Table 3 we indicate the results of the evaluaton of one team that presents high variatons between the evaluatons of its team members. The code of colors used in Table 3 corresponds to: red, for a competency level Vol. 5(2), 2015, pp 146 clearly to be improved (lower than 6); yellow, that the competency level is average (between 6 and 8.25), and, green, that the competency level is excellent (higher than 8.25). These are the qualitatve levels used in the assessment. As we can observe in Table 3, TM2 has the worst results of the team and TM4 has the best results, partcularly in the B3 competency. The tutor of the team suggested, according to his/her experience, that TM4 was a hard worker and he/she interacted a lot with the teammates. Actually, he/she was actng as a leader among the team members in the absence of the formal leader, or during the ongoing tasks. Furthermore, the tutor also indicated that TM2 had no commitment with the team and he/she did not deliver, for example, the outcomes of the tasks on tme. Competency  TM1  TM2  TM3  TM4  TM5 Table 3. Results of the competency evaluatons of a given team. Red indicates that the competency level is lower than 6; yellow indicates that the competency level is between 6 and 8.25, and green indicates that the competency level is higher than 8.25

Team members Leader
The interview with the tutor revealed that this team had some difcultes to carry out the project. The evaluatons of other teams show diferent results whose detailed analysis, however, go along the same lines as the case exposed here. Therefore, from this analysis we can conclude that the instrument is able to respond to the set objectves.
From a statstcal point of view, the matrix of correlaton coefcients (Pearson, 2015) is defned as where x  and y b are two observaton of given pinpointed behaviors related to competency X and Y, respectvely, of the same individual. The bars indicate an average over the populaton and x  stands for the deviaton with respect to the mean. In Table 4 we present the correlatons between the elementary behaviors related with the B4 competency, which is part of the 37 x 37 team members' correlaton matrix. The negatve correlatons appear in magenta. If the element is higher than 0.5, it appears in red. The negatve correlatons indicate grosso modo, that, the two correlated behaviors on average occur in oppositon. If the coefcient is higher than 0.5 (and positve), it means that both correlated behaviors are simultaneously over the mean. High correlaton coefcient may indicate redundancies in the questonnaire. As an example, the coefcient 0.577 that appears in Table 4, is referred to the following pair rubrics:

CONCLUSIONS
From the analysis of the results we see that the instrument is statstcally consistent and permits one to identfy competency levels appearing in professional situatons. At the same tme, it allows one to discriminate not only between members of the same team, but also between competencies of the same individual. The predictve capability of the instrument for superior performance, in the sense of our quotaton of Spencer and Spencer (Spencer & Spencer, 1993), has been checked only from the internship of the fourth year students. The consolidaton of the instrument within the educatonal model of the ETSEQ should allow carrying out long-term studies about the professional evoluton in the labor market over 5 or 10 years.
The positve results of the pilot test allowed the implementaton of the methodology of evaluaton of social competencies to all years in the Bachelors of Chemical Engineering and Food Engineering, during the academic year 2013-2014. This implementaton demanded the partcipaton of more than 300 students, 7 coordinators and more than 15 tutors. The treatment of the data collected, the maintenance of confdentality, and the necessity of coordinatng a large number of individuals are the main obstacles to efectvely implement the instrument and, more importantly, make the methodology sustainable. These problems can be solved using specifc tailor-made sofware to enable remote access, keep the data secure, treat the informaton automatcally, and manage confdentality. On the other hand, our experience suggests that a reducton of the items to be evaluated is advisable, and that the range of values per item should be increased. Both actons aim at simplifying and systemizing the procedure in order to guarantee the sustainability of the methodology.
Secondly, the implementaton of this methodology of assessing social competencies reveals new informaton about our students. This knowledge is useful for the students since they are confronted with what the others see in them, in the context of the IP: γνwθι σεαυτόν, know thyself. Such knowledge should trigger by itself behavioral changes in the students. In the frst place, the exposure to the rubrics, when actng as evaluators, causes behavioral changes by imitaton of actons suggested by the standard, as we observed during the pilot test. In the second place, the assessment places the students face-to-face with what others see in them. If they dislike the image returned by this mirror they will have motvaton to change. Furthermore, the existence of this objectve knowledge about the students introduces a new educatonal necessity in the staf: how to help our students to improve their competency level before they are ready to enter the labor marked. Therefore, including improvement plans in our educatonal model will be a subject of major research in the future.