• Home
• About
• Login
• Archives
• Submissions
• Publication Fee
• Indexing & Statistics

Home > Vol 3, No 2 (2013) > Yin

An Analytical Hierarchy Process Model for the Evaluation of College Experimental Teaching Quality

Qingli Yin Department of Laboratory and Facility Management, Shandong Jianzhu University China yql69@yahoo.com.cn

Received December 2012

Accepted July 2013

Abstract

Taking into account the characteristics of college experimental teaching, through investigation and analysis, evaluation indices and an Analytical Hierarchy Process (AHP) model of experimental teaching quality have been established following the analytical hierarchy process method, and the evaluation indices have been given reasonable weights. An example is given, and the evaluation results show that the evaluation indices proposed in this paper are capable of reflecting objectively, exactly and reasonably experimental teaching quality, and of effectively promoting the quality of experimental teaching.

Keywords – experimental teaching quality, evaluation, weight, AHP model, consistency

----------

1 Introduction

College experimental teaching plays an irreplaceable role in the cultivation of innovative talents, and experimental teaching quality has a direct impact on teaching quality as a whole. It is therefore necessary to introduce evaluations of experimental teaching quality. The key point of evaluating experimental teaching quality concerns how to improve it. According to the characteristics of experimental teaching, we have identified the core elements of experimental teaching in this paper, and provide a reasonable evaluation index system based on Analytical Hierarchy Process (AHP). Indices at all levels have been given reasonable weights based on a mathematical model, and the degree of influence each evaluation index has on experimental teaching quality has also been determined. Finally, the evaluation results of selected teachers are examined, based on a sample of evaluation data obtained using a mathematical model, which demonstrates the rationality and credibility of the evaluation results.

2 An AHP model for evaluation of experimental teaching quality

Although the evaluation of college experimental teaching quality has been previously discussed (Chen, 2009; Feng, Shi & Du, 2010; Ma, Liu & Lv, 2009; Qin & Shi, 2010; Zhang, Zhou, Han & Huang, 2011), every university has its own features, and therefore, the evaluation index system or certain weights may not be same. This highlights the importance of creating an evaluation system that is suitable for each university. This paper takes into account the students’ points of view to establish an AHP model of evaluating experimental teaching quality, according to the actual situation at our university, focusing on the position of the students in the evaluations.

2.1 Construction of evaluation system assessing experimental teaching quality

Based on the characteristics of experimental teaching, taking into account the recommendations of experts, instructional supervisors, teachers and students, this evaluation system can be summarized as four aspects consisting of 13 factors, the hierarchical structure of which is shown in Table 1.

Goal A	Criterion B	Alternative C
Evaluation of experimental teaching quality	Teaching attitude: B1	C11: Teaches and cultivates people，sets strict demands and is worthy of being called a teacher. C12: Engages in experimental teaching, is well-prepared and lectures seriously and fluently. C13: Corrects lab reports in timely, serious manner and patiently gives guidance.
	Teaching contents: B2	C21: Is familiar with the experimental contents and use of instruments; provides guidance materials C22: Contents evidence a reasonable design and are explained clearly and accurately. The theoretical course and the experimental course can be organically linked. The most important topics related to the subject are given an appropriate description. C23: The emphasis on and difficulty of the experimental teaching process is prominent, the amount of contents is suitable for students to master and the level of difficulty is appropriate for students to understand. C24: Comprehensively designed experiment contents are incorporated in the course and scientific research is introduced through experimental teaching.
	Teaching methods: B3	C31: Is good at inspiring students to think, stimulates the students' intellectual curiosity through timely guidance during experiments, encourages them to participate in the discussion of experiments and express different views. C32: Is good at guiding students as they analyze experimental phenomena and results, incorporating learned knowledge. C33: Teaching is organized in a flexible and effective manner, students are taught according to their aptitudes and instruction follows a logical order.
	Teaching results: B4	C41: Contribute to the consolidation of related theoretical knowledge by the students. C42: Increased knowledge, developed thinking and improvement in the students' practical skills through experimental activities. C43: Promote innovation by students and their ability to develop and design comprehensive experiments.

Table 1. Evaluation system assessing experimental teaching quality

2.2 Design of comparison matrices

Comparison matrices are the basis for weight sorting, and they have a decisive influence on the final overall sort. Therefore, the design of comparison matrices is a very important aspect of AHP. To accurately design comparison matrices at all levels, they must be carefully and objectively analyzed, researched and corrected until consistency verifications produce satisfactory results. To make comparisons, we need a scale of numbers that indicates how many times more important or dominant one element is over another with respect to the criterion. One common scale (Saaty, 2008) is shown in Table 2.

Intensity of importance	Definition	Explanation
1	Equal importance	Two factors contribute equally to the objective.
3	Somewhat more important	Experience and judgment slightly favor one over another.
5	Much more important	Experience and judgment strongly favor one over another.
7	Very much more important	Experience and judgment very strongly favor one over another. Its importance is demonstrated in practice.
9	Absolutely more important	The evidence favoring one over the other is of the highest possible validity.
2, 4, 6, 8	Intermediate values	Compromise is needed
Reciprocals of the above	If activity i has one of the above non-zero numbers assigned to it when compared to activity j, then j has the reciprocal value when compared to i.

Table 2. The fundamental scale of absolute numbers

The matrix of pairwise comparisons A = (a_ij) represents the intensities of the expert’s preference between individual pairs of criteria (alternatives) (B_i versus B_j, for all i, j = 1, 2, …, n). They are usually chosen according to a given scale (1/9, 1/8, …, 8, 9). Given n criteria (alternatives) {B₁,B₂,…, B_n}, a decision-maker compares pairs of criteria (alternatives) for all the possible pairs, and a comparison matrix A is thus obtained, where the element a_ij shows the preference weight of B_i obtained by comparison with B_j.

Using the scale of relative importance shown in Table 2, a set of pairwise comparison matrices is created by synthesizing the recommendations of experts, instructional supervisors, teachers and students, as shown in Tables 3-7, where the comparison matrix A is the criteria matrix for the criteria with respect to the goal, the comparison matrices B₁, B₂, B₃, B₄ are alternative matrices for alternatives with respect to each criterion. For example, we see criteria matrix A, in which the principal diagonal contains entries of 1, as each factor is of equal importance. The experts decide that B₄, teaching results, is somewhat more important than teaching attitude, which is rated as 3 in the cell B₄, B₁ and 1/3 in B₁, B₄. They also decide that teaching contents is slight more important than teaching methods, assigning a score of 2 in the cell B₂, B₃ and 1/2 in B₃, B₂. The other elements of criteria matrix A are obtained in a similar manner, as shown below.

A	B1	B2	B3	B4	Priority vector WA	Consistency check indicators
B1	1	1/4	1/2	1/3	b1 = 0.0994	lmax = 4.02062
B2	4	1	2	2	b2 = 0.4379	CI = 0.00687
B3	2	1/2	1	1	b3 = 0.2190	CRA = 0.00764 < 0.1
B4	3	1/2	1	1	b4 = 0.2437

Table 3. Criteria matrix A and its consistency check

B1	C11	C12	C13	Priority vector WB1	Consistency check indicators
C11	1	1/2	1/4	0.1429	lmax = 3
C12	2	1	1/2	0.2857	CI1 = 0
C13	4	2	1	0.5714	CR1 = 0

Table 4. Alternative matrix B₁ and its consistency check

B2	C21	C22	C23	C24	Priority vector WB2	Consistency check indicators
C21	1	1/2	1/4	1/2	0.1111	Lmax = 4
C22	2	1	1/2	1	0.2222	CI2 = 0
C23	4	2	1	2	0.4445	CR2 = 0
C24	2	1	1/2	1	0.2222

Table 5. Alternative matrix B₂ and its consistency check

B3	C31	C32	C33	Priority vector WB3	Consistency check indicators
C31	1	4	3	0.6337	lmax = 3. 0092
C32	1/4	1	1	0.1744	CI3 = 0.0046
C33	1/3	1	1	0.1919	CR3 = 0.0079

Table 6. Alternative matrix B₃ and its consistency check

B4	C41	C42	C43	Priority vector WB4	Consistency check indicators
C41	1	1/3	1/5	0.10945	lmax = 3.0037
C42	3	1	1/2	0.3090	CI4 = 0.0018
C43	5	2	1	0.58155	CR4 = 0.0032

Table 7. Alternative matrix B₄ and its consistency check

2.3 Relative weights and consistency check

It is important to note that AHP does not demand perfect consistency. Some inconsistency is allowed in random judgments. An inconsistency ratio of about 10 percent or less is usually considered “acceptable”. The consistency index (CI) is calculated according to the following equation CI = (l_max-n)/(n-1), where l_max is the largest eigenvalue of the comparison matrix, and n is the number of criteria. The Consistency Ratio (CR) is calculated using the equation CR = CI/RI. The RI is the random index representing the consistency of a randomly generated pairwise comparison matrix. It was derived by Saaty (1980) as average random consistency index (Table 8) calculated from a sample of 500 randomly generated matrices based on the AHP scale (Table 2).

n	1	2	3	4	5	6	7	8	9
RI	0	0	0.58	0.90	1.12	1.24	1.32	1.41	1.45

The CR tells the decision-maker how consistent he has been when making the pair-wise comparisons. If CR<0.10, the decision-maker’s pair-wise comparisons are relatively consistent and the criterion is considered to have acceptable consistency. If CR > 0.10, the decision-maker should seriously consider re-evaluating his pair-wise comparisons – the sources of inconsistency must be identified and resolved and the matrix reanalyzed. The priority vector WA (weights of the criteria corresponding to the goal) is the normalizing eigenvector corresponding to l_max of matrix A, and the consistency ratio of matrix A is CR_A = 0.00764, as shown in last two columns of Table 3, respectively. Similarly, the priority vectors W_B1, W_B2, W_B3, W_B4 (weights of alternatives for each criterion) and their consistency check indicators are listed in last two columns of Tables 4-7, respectively.

2.4 Combinatorial weights and combinatorial consistency check

The priorities of each alternative for the goal are referred to as combinatorial weights, and the components of the combinatorial weight vector are calculated as described by Han (2005):

, where each value of bj and cij is listed in Table 9.

The combinatorial consistency ratio is calculated according to the formula:

CR = 0.00764 + (0.0994 × 0 + 0.4379 × 0 + 0.2190 × 0.0046 + 0.2437 × 0.0018)/(0.0994 × 0.58 + 0.4379 × 0.9 + 0.219 × 0.58 + 0.2437 × 0.58) = 0.0096 < 0.1; therefore, the combinatorial consistency is acceptable, and the result of the global ranking has satisfactory consistency. If CR > 0.10, the decision-maker should seriously reconsider the model or reconstruct the comparison matrices so that they have a higher consistency ratio.

Alternative C	Criterion B				Combinatorial weights wi
	B1	B2	B3	B4
	b1 = 0.0994	b2 = 0.4379	b3 = 0.2190	b4 = 0.2437
C11	c11 = 0.1429	c12 = 0	c13 = 0	c14 = 0	w1 = 0.0142
C12	0.2857	0	0	0	w2 = 0.0284
C13	0.5714	0	0	0	w3 = 0.0568
C21	0	0.1111	0	0	w4 = 0.0487
C22	0	0.2222	0	0	w5 = 0.0973
C23	0	0.4445	0	0	w6 = 0.1946
C24	0	0.2222	0	0	w7 = 0.0973
C31	0	0	0.6337	0	w8 = 0.1388
C32	0	0	0.1744	0	w9 = 0.0382
C33	0	0	0.1919	0	w10 = 0.0420
C41	0	0	0	0.10945	w11 = 0.0267
C42	0	0	0	0.3090	w12 = 0.0753
C43	0	0	0	0.58155	w13 = 0.1417

Table 9. Combinatorial weights

2.5 Model evaluation

As presented in Table 3, it was observed that for students, “teaching contents” represented the most important criterion, followed closely by “teaching results” and “teaching methods”. It can be seen that students rank teaching results higher than teaching methods. Lastly, “teaching attitude” does not seem to be particularly important to students. This model is in agreement with the current situation at engineering universities, and it comprehensively reflects student evaluations of experimental teaching quality. If a teacher does not have a certain amount of theoretical knowledge and practical experience, he can not reconcile theory with practice in the process of teaching, his instructional process is more boring, and is not very effective.

3 Application of the AHP model

We wrote a questionnaire that requires students to evaluate their teachers, assigning them one of five grades (A, B, C, D and E; equivalent to very satisfied, satisfied, generally, dissatisfied, very dissatisfied) for each of the 13 alternatives shown in Table 1. Table 10 shows the evaluation results of 105 students for three teachers.

First, the statistical data are quantified with the values 5, 4, 3, 2, 1, which correspond to the ranking grades A, B, C, D and E. Accordingly, taking into account the membership function of the Cauchy distribution:

(*)

Where a, b, a, b are constants to be determined. Supposing that the membership degree is 1 when the evaluation is grade A, this means that f(5) = 1; the membership degree is 0.70 when the evaluation is grade C ( f(3) = 0.70); and the membership degree is 0.10 when the evaluation is grade E (f(1) = 0.1). The values of a, b, a, b are then determined to be a = 2.8049, b = 0.4417, a = .5873 and b = 0.0548, respectively. The membership function is obtained by substituting the values of a, b, a, b in the formula (*):

The values of the membership function at x = 2, 4 are calculated as f(2) = 0.4640, f(4) = 0.8690, thus grades A, B, C, D and E are quantified with values {1, 0.869, 0.70, 0.464, 0.1}. The scores for each alternative for each teacher are calculated using the data in Table 10, listed in columns 3, 5 and 7. The total scores for each teacher are obtained by adding the products of each alternative score and its combinatorial weight w_i obtained from Table 9, shown in the last row of Table 10 as the evaluation result tallies for the current situation.

Alternative	Teacher 1	Score	Teacher 2	Score	Teacher 3	Score
C11	68A 30B 6C 1D	0.9403	58A 28B 15C 4D	0.9027	53A 38B 6C 7D 1E	0.8911
C12	67A 31B 2C 5D	0.9301	30A 45B 21C 5D 4E	0.8240	52A 40B 8C 5D	0.9017
C13	71A 17B 15C 2D	0.9257	43A 13B 25C 16D 8E	0.7621	52A 32B 16C 5D	0.8888
C21	60A 30B 12C 3D	0.9130	38A 27B 17C 14D 9E	0.7691	48A 43B 11C 3D	0.8996
C22	65A 36B 4C	0.9437	41A 36B 24C 4D	0.8661	40A 51B 9C 5D	0.8851
C23	69A 16B 9C 7D 4E	0.8843	33A 45B 22C 5D	0.8555	51A 33B 14C 7D	0.8831
C24	68A 33B 2C 2D	0.9429	36A 36B 25C 6D 2E	0.8359	35A 47B 17C 4D 2E	0.8552
C31	71A 23B 5C 6D	0.9264	43A 38B 17C 4D 3E	0.8579	65A 23B 13C 4D	0.9137
C32	63A 27B 8C 7D	0.9077	31A 47B 21C 6D	0.8507	40A 43B 10C 9D 3E	0.8461
C33	65A 21B 17C 2D	0.9150	34A 45B 17C 3D 6E	0.8285	45A 33B 18C 5D 4E	0.8476
C41	62A 26B 13C 4D	0.9100	40A 37B 21C 7D	0.8581	49A 28B 23C 5D	0.8738
C42	69A 23B 9C 4D	0.9252	46A 35B 18C 5D 1E	0.8708	43A 26B 28C 4D 4E	0.8329
C43	57A 28B 13C 5D 2E	0.8853	31A 33B 26C 10D 5E	0.7906	38A 30B 20C 12D 5E	0.8013
Total score		0.9136		0.8359		0.8681

4 Conclusion

The evaluation indicators and their weights of experimental teaching quality are developed according to the AHP method. As compared to other methods, the greatest advantage of the analytical hierarchy process (AHP) is that it is able to combine both qualitative and quantitative methods and to consider all influencing factors as fully as possible. The evaluation results of the AHP method are more objective, scientific and rational, and it has been proven that the AHP method is the most appropriate when surveys need to account for a high degree of intuition and subjectivity. The evaluation results for the 3 teachers in section 3 are only from a student perspective; if we administer the same questionnaire to experts, instructional supervisors and colleagues of the 3 teachers, requesting their evaluations of the 3 teachers according to the 13 alternatives in Table 1, it would be possible to obtain evaluation results for the 3 teachers from the experts’ points of view, using the same AHP model developed in this paper. This would provide more objective evaluations for the 3 teachers, synthesizing the students’ and the experts’ evaluation results. The model established in this paper can be extended to the comprehensive evaluation of textbooks, teaching management, quality courses, and other situations requiring comprehensive evaluation. The AHP method has already been used in many applications (Saaty, 2008; ISAHP, 2009).

References

Chen, H. (2009). Constructions and applications of the evaluation model of practical teaching quality in colleges [in Chinese]. Research and Exploration in Laboratory, 28, 159-161.

Feng, L.X., Shi, S.T., & Du, W.M. (2010). The model of teaching evaluation based on analytic hierarchy process [in Chinese]. Northwest Normal University, Natural Science Edition, 46, 19-22.

Han, Z.G. (2005). Mathematical modeling method and its application [in Chinese]. Beijing, China: Higher Education Press.

ISAHP (2009). Proceedings of the 10th International Symposium on the AHP. ISAHP Web Site. ISAHP. August, 2009. Retrieved 2011-01-05.

Ma, Z.L., Liu, H.D., & Lv, S.P. (2009). Building experiment teaching quality evaluation system in science and engineering high school [in Chinese]. Experimental Science and Technology, 7, 78-81.

Qin, C.M., & Shi, Z.G. (2010). Constructing and practicing of experiment teaching quality evaluation system in colleges [in Chinese]. Heilongjiang Education, 5, 22-23.

Saaty, T.L. (1980). The Analytic Hierarchy Process. New York, NY: McGraw Hill International.

Saaty, T.L. (2008). Decision making with the analytic hierarchy process. Int. J. Services Sciences, 1, 83-98. http://dx.doi.org/10.1504/IJSSCI.2008.017590

Zhang, Y.J., Zhou, Y., Han, Y., & Huang, D.X. (2011). Research and practice on quality evaluation of experimental teaching in common colleges and universities [in Chinese]. Experimental Technology and Management, 28, 270‑273.

Citation: Yin, Q. (2013). An Analytical Hierarchy Process Model for the Evaluation of College Experimental Teaching Quality. Journal of Technology and Science Education (JOTSE), 3(2), 59-65. http://dx.doi.org/10.3926/jotse.66 On-line ISSN: 2013-6374 – Print ISSN: 2014-5349 – DL: B-2000-2012

Author biography

Qingli Yin

---