ASSESSMENT OF OUTCOME-BASED INTEGRATED DESIGN PROJECT

Outcome-Based Education (OBE) implementation is one of the qualifying requirements for engineering programme accreditation in Malaysia. Implementation of the OBE in Integrated Design Project (IDP) is essential in producing high quality engineering graduates that are able to meet the challenges especially in the era of Industry 4.0 in Malaysia. IDP is the course offered to the final year students of undergraduate programme in Faculty of Civil Engineering, University Teknologi MARA. This paper presents the mapping of the addressing Course Outcomes (CO) and Programme Outcomes (PO) to the scopes of assessment in a capstone project (CP) for IDP. The explicitness in the measurement of CO and PO in the course is demonstrated. The rubrics for the assessment of CP are also presented. Finally, evaluation of student performance in IDP particularly CP is discussed through a sample analysis of the CO and PO attainments.


Introduction
The Board of Engineers Malaysia (BEM) registers graduate engineers that are qualified from a recognised engineering programme under the Registration of Engineers Act 1967 (Laws of Malaysia, 1967(Revised 2015)).The Engineering Accreditation Council (EAC) is the only recognised body delegated by BEM for accreditation of engineering degree programmes offered in Malaysia.EAC comprises of five stakeholders namely BEM, The Institution of Engineers, Malaysia (IEM), Industry Employers, Malaysian Qualification Agency (MQA) and the Public Service Department (Jabatan Perkhidmatan Awam Malaysia (JPA)).BEM is one of the signatories of Washington Accord (WA) since 2009.The accredited engineering degree programmes by EAC are given full recognition as equivalent qualifications by other signatories of WA including United State, United Kingdom, Australia, Hong Kong, Japan and Singapore.
The implementation of Outcome-based Education (OBE) which is promoted by WA becomes one of the mandatory requirements in an accreditation of engineering degree programmes outlined by EAC.Specific outcomes taking inputs from the stakeholders must be designed in one curriculum to produce high quality graduates that meet the needs of the students and stakeholders.Specifically, the teaching, learning and assessment methods shall be conducted consistently accordance to the appropriate graduate attributes listed in the Programme Educational Objectives (PEO), Program Outcomes (PO), Course Outcomes (CO) of the education content (BEM, 2017).
International Engineering Alliance presented a total of twelve differentiating characteristics for WA graduates attributes for engineer track as shown in Table 1 (Accord, 2013).The characteristics described with respective knowledge profiles were adopted as the POs in an undergraduate programme EC220 in Faculty of Civil Engineering (FCE), Universiti Teknologi MARA (UiTM).Integrated Design Project (IDP) is one the courses offered to the final year students of undergraduate programme in FCE, UiTM.This paper presents the assessment of a capstone project (CP) for IDP which is implemented in outcomebased manner.The mapping of the addressing COs and POs to the scopes of assessment is also demonstrated.Demonstrate knowledge and understanding of engineering management principles and economic decision-making and apply these to one's own work, as a member and leader in a team, to manage projects in multidisciplinary environments; 12 Life Long Learning Recognise the need for, and have the preparation and ability to engage in independent and life-long learning in the broadest context of technological change.Note: WK denotes the knowledge profile Table 1.Graduate attribute profiles for WA.(Accord, 2013) This paper is organized as follows.Section 2 introduces background of the implementation of OBE.Section 3 presents the implementation of OBE in the course Integrated Design Project (IDP).Section 4 discusses the measurement of the POs and COs through a sample analysis for a capstone project in IDP.Finally, our works of this paper are summarized in the last section.

Outcome-Based Education
The implementation of OBE focuses on the achievement of graduate attributes in the teaching and learning processes.OBE incorporates innovative teaching methods such as problem-based leaning, capstone projects, oral presentations, professional talks and site visits.OBE approach at an institutional level always deals with generic graduate attributes whereas at the program level focuses on the specific discipline-oriented goals of the program such as the POs and COs.Continuous Quality Improvement (CQI) of an education could be then judged by government and accreditation agencies based on the assessment of student attributes through the implementation of OBE (El-Maaddawy & Deneen, 2017).Effectiveness of OBE implementation in promoting teaching and learning process relies on strong interconnection between the proper teaching, learning and assessment activities and the constructive mapping of the outcomes in terms of graduate attributes (Taras, 2015).An example of implementation model of OBE practices for the civil engineering students was demonstrated by Alias and Bhkari (2017).
Assessment is an essential process in OBE implementation.With clear and measureable outcomes, the results from the assessment could be used for the CQI, specifically in the teaching and learning process.Assessment methods for student performance are generally classified into direct methods and indirect methods (Easa, 2013).A set of indicators are used in a direct assessment method to examine and observe student knowledge or skills in a direct manner.Popular direct methods are final examination, test, quiz, viva, rubrics, presentation, project, and behavioral observations.An example of rubrics to evaluate students attributes for design courses and the capstone courses was developed (Platanitis & Pop-Iliev, 2010).Alternatively, indirect methods indicate student performance through perceptions and self-assessment.Common indirect methods are entrance-exit surveys, alumni surveys and feedback forms.For example, Henry Samueli School of Engineering and Applied Science at the University of California, Los Angeles applied a qualitative assessment of The High School Summer Research Program through exit and alumni survey to determine the growth of the program, benefits and long-term impact of the program to the students (Kittur, Shaw & Herrera, 2017).A model with the method of peer assessment has been developed in the assessment cycle.This approach discussed the activities in peer assessment and proposed the peer assessment for the learning opportunities rather than grade analysis (Reinholz, 2016).

Integrated Design Project
Integrated Design Project (IDP) is the course offered to the final year students of a 4-year undergraduate programme in Faculty of Civil Engineering, University Teknologi MARA.IDP simulates a real world integrated design environment through a capstone project (CP).CP generally involves a process of solving civil engineering related design problems with considerations on safety; accessibility, constructability and sustainability; impact on the aesthetics, societal and environmental.CP also involves the study of cost-effectiveness of the proposed design solutions.The conduct of the CP promotes the students to enroll as role-playing planners, managers, engineers, operators and maintenance community, fostering coordination and cooperation among team members.This section presents the scopes of assessment in the CP and the mapping of the associating COs and POs to each of the scopes of assessment.

Scopes of Project
The overall scopes of CP are relying on two main parts: (1) a final report and (2) a presentation of the design solutions.Figure 1 summarizes the scopes under the main components of the CP.
The integrated design components in the final report of CP are categorized as follows: (a) Superstructures and substructures design Structural analysis and design of a superstructure (i.e.building) by using modern tools, verified by manual calculations with the relevant design code of practices are expected.Structural key plans and detailings for the superstructure shall be prepared.Additionally, a proposal of processes or systems or materials that are environmental responsible (green building related) should also be included.
A design proposal for an appropriate substructure (i.e.foundation system) for the superstructure is also expected.Justification of the proposal could be based on a good interpretation of soil investigation data and geotechnical analysis of the given site.
(b) Infrastructures design CP also includes design of infrastructure works namely road and earthwork, sewerage, water supply and drainage systems.The design of the infrastructures is carried out with the considerations of the authority requirements, study of the site and neighborhood topography, impacts to the societal and environmental aspects, accessibility and availability of local suppliers.

(c) Construction management
Cost-effectiveness study of the design in (a) and (b) and the planning of the construction phases for the design are incorporated in CP.The students are required to prepare the taking off, estimation, bill of quantities and the construction schedule for the project.
At the end of the IDP course, students are required to present their design solutions to a numbers of evaluators specifically professional engineers from construction industry.The presentation of CP is evaluated based on the students' communication skill, sequential and organization of the presentation, ability to discuss and teamwork during the presentation session especially in answering the questions.

Mapping of CO and PO
A total of four COs and seven POs are mapped for CP in the course IDP.The COs and POs reflect the potential and ideal graduate attributes derived from general contents of the course.It is noted that these outcomes are to be achieved by the students who have completed the course.Table 2 shows the descriptions written in a form of objectives of the four COs in IDP course.The COs appropriately emphasize the aims of the course in producing students with the abilities to solve complex design problems under constraints, while demonstrating effective communication and leadership skills.
Table 3 shows the CO-PO matrix and the addressing domains in CP.Mapping of the COs and POs is developed in a parallel way.One or more POs are mapped to a CO according to their relevancy to the particular CO.POs that are mapped under COs should be able to clearly explain the specific outcomes after completing CP.

CO
Descriptions CO1 Define and formulate solutions to complex design problem.CO2 Use the design standards with consideration of real world constraints in particular the environmental responsibilities of professional engineer and matters related to economic, legislation, lifespan, ethical, social etc. CO3 Demonstrate effective communication skill through presentation and defense of design project.CO4 Demonstrate leadership skills, ability to work independently and in a team through project design, presentation and defense of project findings.

Evaluation of Capstone Project
This section presents evaluation method for report and presentation in CP.Analysis results of student attainment in COs and POs for two semesters are also presented.

Assessment Rubrics
Evaluation of the graduate attributes is made through a standard assessment form with descriptive rubrics.The assessment form with rubrics is given to the students during the assignment of CP.Awareness on the outcomes of CP specifically based on CO-PO by both lecturers and students is essentially needed.
Table 4 shows the assessment of CP, inclusive the scopes (see also Figure 1), the mapping of CO-PO to the scopes and the associated marks.The assessment form is divided into two parts: report and viva.
It is noted that CO1 and CO2, with the addressing PO2, PO3, PO5, PO11 and PO12 are the expected outcomes after completing the design report.The students attain the skills of analysis and design, application of modern tools, understanding of fundamental in management and appreciation of life-long learning in proposing design solutions via report.Although PO3 and PO5 are associated with CO2, the evaluation for each of the scopes is carried out explicitly.Specifically, independent rubrics are available for the evaluation for CO2-PO3 and CO2-PO5 for the scopes 1(a-d) and 1(g-j).For instance, for scope 1(b), assessments on the student ability to design with constraints through CO2-PO3 whereas the ability to use the modern tool for the design through CO2-PO5 are shown.Meanwhile, for the viva session, the students are measured on their communication skills, CO3-PO10 and ability to work as a team via CO4-PO9 are expected.The simplified rubrics for the assessment of CP are presented in Table 5. Figure 2 shows the percentage of attainment of COs and POs specially in completing CP for two semesters.There are 159 and 215 final year students who have attended the course for the semester March to September 2016 and semester September to January 2017, respectively.The assessment of the COs and POs for the course was conducted by the authors through the assessment form (Table 4) and constructive rubrics (Table 5).
The results in Figure 2 reveal that all students adequately achieved all COs and POs (i.e.greater than 50%).The attainment of all COs and POs, based on IDP student performance, was on average 82%, with a standard deviation of 4%.The highest attainment of about 86% was recorded for CO2-PO5 (student ability in the application of modern tools) for both semesters.Attainment CO3-PO10, pertaining to student communication skills, exhibited low attainment of about 77% for both semesters.Attainment CO1-PO2 (student ability in interpretation and analysis of results) was recorded as the lowest of approximately 76% for semester March to September 2016.This suggests that the activities such as presentation for project could be conducted more frequent throughout the semester before the final viva session to the panels.

Conclusions
This paper shows an evidence of the implementation of OBE in IDP.This paper also presents the guidelines for assessing graduate attributes assessment in civil engineering education particularly through a capstone project.Rubrics for making the assessment more efficient and judicious are presented.The mapping of CO-PO, assessment method and constructed rubrics have successfully measured student performance in completing their project.Results of an effective assessment of student attainment in COs and POs are important in the CQI for future teaching and learning processes.It is hoped that this paper will provide useful guidelines on assessment of graduate attributes for engineering programs.

Figure 1 .
Figure 1.Scopes of the capstone project in IDP pronunciation and present content of design project in logical and interesting sequence and attractive slides within allocated time.4-5Discussion(Q&A) (CO4-PO9) Teamwork in answering questions correctly and clear roles, constant transition between team members, effective cross reference each other.

Table 2 .
COs in IDP course

Table 3 .
CO-PO Matrix for CP Both direct assessment (i.e.projects, test, and presentation) and indirect assessment (i.e.exit-entrance surveys) are implemented in IDP course.Only the evidence of student achievement of CO-PO particularly on CP taken from assessment of IDP students is presented.The percentage of students attainment in COs and POs particularly after completing the course IDP is analyzed by using excel template at course level.The results of COs and POs for every course in the undergraduate programme EC220 are analyzed through a system MyCOPO at programme level.MyCOPO is a system utilized by FCE University Teknologi MARA to analyze individual student outcomes.Actions for CQI could be then planned if any students failed to achieve any POs.

Table 5 .
Simplified rubrics for CP Figure 2. CO-PO attainment for CP