Student Rating of Instruction: A Survey of Satisfaction and Uses

Rhonda Magel Charles McIntyre

Professor and Chair Associate Professor and Chair

Statistics Construction Management and Engineering

North Dakota State University North Dakota State University

Fargo, ND 58105 Fargo, ND 58105

rhonda.magel@ndsu.edu charles.mcintyre@ndsu.edu

Introduction

The use of student evaluation data has dramatically increased during the past 20 years (Langbein 1994; Wachtel 1998; Sheehan and DuPrey 1999). A fundamental concern of student evaluations focuses on the validity of the data (Greenwald 1997; Langbein 1994). However, many argue that student ratings of instruction provides some of the best measures of teaching effectiveness (Trout 2000; Theall and Franklin 2001). This debate continues at a time when increasing tuition and other related costs of higher education have put pressure on administration for accountability for student learning and improving assessment (Koslowski 2005). Accreditation agencies and other external sources have required a more rigorous assessment of teaching effectiveness and student learning outcomes. In effect, student rating of instruction will take on an expanded role (Quinlan 2002). The quality and validity of the collected assessment data will be under increasing scrutiny. These are not easily measured, however, reasonable starting points for determining quality and validity in student response data would be to determine if students, faculty and administration are satisfied with the current process and uses of the student evaluation data which could lead to a better understanding of what motivates students to participate in the evaluation process and the level of effort that they will exert in completing the evaluation forms.

Outline of the Study

This paper describes a study conducted at North Dakota State University (NDSU) in an effort to measure the level of satisfaction and uses of the student evaluation process, from now on referred to as the Student Rating of Instruction (SROI). This study consisted of the following five (5) phases:

Phase I - Input from the Students

Phase II - Input from Administrators

Phase III - Input from Faculty

Phase IV - Data Analysis and Evaluation

Phase V - Recommendations and Conclusions

Phase I - Input from the Students

During the 2005 Spring Semester, input was collected from undergraduate students concerning their attitudes and perceptions related to the current SROI process, as well as, what motivates them to participate an SROI process. Students from three different classes were invited to participate in the survey and included, Math 166 - Calculus II, Stat 330 - Introduction to Statistics, and CE/CME 489 - Capstone (Senior Design). Although engineering was the primary major, students from a variety of colleges and majors participated in the study in order to provide a more balanced cross-section of student response. Overall, 171 students completed the survey (122 men and 48 women). The breakdown by year and affiliated college is presented in Tables 1 and 2. Table 3 presents the current GPA of the survey participants. It should be noted that a few of the 171 students did not respond with a year, GPA, or major.

Table 1. Year in the Major

First	64
Second	38
Third	25
Fourth	18
Fifth	24
Total	169

Table 2. College of the Major

Arts, Humanities & Social Sciences	10
Agriculture	13
Business	16
Engineering	66
Human Development & Education	19
Pharmacy	4
Science & Math	25
Undecided	8
Total	161

Table 3. GPA Breakdown

< 1.50	1
1.50‑1.99	13
2.00‑2.49	36
2.50‑2.99	58
3.00-3.49	39
3.50‑4.00	14
Total	161

At NDSU, it is required that every student registered for all undergraduate courses must be given an opportunity to complete a course evaluation during the last three weeks of the semester. A standard SROI form is used. Within the context of this study, the surveyed students were asked their general impressions concerning the SROI process on a scale of 0 to 9 (“0”=Useless and “9”=Very Helpful). In addition, students were asked to rank the following four possible uses of an SROI process in order of importance: improving teaching; promotion, tenure, and merit pay raises; improving course content; and providing information to other students. Results from the student survey are given in Phase IV - Data Analysis and Evaluation.

Phase II - Input from Administrators

Administrators were surveyed at a University Chair’s meeting in October of 2005. Thirty‑one (31) Chairs representing all seven (7) Colleges within the University responded to the survey. Questions asked of the Chairs included information related to their duration at the university and as a Chair, college affiliation, and specific rating and use of the SROI process (using the same 0 to 9 scale used by students). In addition, the Chairs were asked to rank the following four possible uses of SROI data and were allowed to provide some text response related to additional ways to use an SROI process, as well as, any additional comments that they might have for improving and/or modifying the current SROI process. Results from the Chairs survey are given in Phase IV - Data Analysis and Evaluation.

Phase III - Input from Faculty

Faculty were surveyed at a university-wide pedagogical luncheon in November of 2005. Sixty‑two (62) faculty members representing all Colleges within the University participated in the survey. Faculty survey questions were structured to collect data concerning the following items: duration at NDSU, college affiliation, gender, academic status (rank), general impressions of the SROI process, and ranking the uses of SROI data. Similar to the Chair survey, faculty were asked to provide some text response related to additional ways to use an SROI process and any comments that they might have for improving the current SROI process. Results from the faculty survey are given in Phase IV - Data Analysis and Evaluation.

Phase IV - Data Analysis and Evaluation

Students

Students gave the current SROI process an average rating of 4.025 (the maximum score was 9.0). No significant differences were found between the responses of men and women. Significant differences were found between students in different years of their major with students in their fourth and fifth year rating the current SROI process significantly lower (p<0.05). The correlation between a student's GPA and the rating that the student gave the current SROI process was not significant (with a p‑value equal to 0.228).

Students then rated the attractiveness of participating in an SROI process based on probabilities that the process would be used for each of four different outcomes: 1.) improvement of teaching by the instructor; 2.) salary, promotion, and tenure decisions; 3.) course improvement; and 4.) releasing the information to other students. The survey instrument was based on the model used in Giesey, et al., 2004. Based on this model, students rated the most attractive reason for them to participate in an SROI process is that the process would lead to the “improvement of teaching”. This was followed closely by the reason that the process would lead to “course improvement”. There was actually only a marginally significant difference between these first two reasons (p=0.075). There was a significant difference between all other reasons (p≤0.05) with using the process for “promotion, salary, and tenure decisions” being the least attractive reason for students to participate in an SROI process. The results are presented in Table 4.

Table 4. Results for Students (n=171)

(how they would like to see SROI data used)

Uses of an SROI process	Average (Ranking)*
Improvement of Teaching	2.064 (1)
Improvement of Course Content	2.257 (2)
Promotion, Tenure, and Salary Decisions	3.000 (4)
Making Information Available to Other Students	2.515 (3)

*All average rankings are significantly different with p≤0.05 except for improvement of teaching and improvement of course content (p=0.075).

Administration

Administrators gave the current SROI process an average rating of 4.452 using the 0-9 scale. Correlation coefficients were calculated between the number of years a chair had been at NDSU and the overall rating they cave the current SROI process at NDSU. A marginally significant negative correlation was found (p=0.082) meaning that the longer the chair had been at NDSU the greater a tendency to give the system a lower rating. The results for administrators are presented in Table 5.

Table 5. Results for Administrators (n=31)

(how they would like to see SROI data used)

Uses of an SROI process	Average (Ranking)*
Improvement of Teaching	1.29 (1)
Improvement of Course Content	2.43 (2)
Promotion, Tenure, and Salary Decisions	2.77 (3)
Making Information Available to Other Students	3.87 (4)

* All average rankings are significantly different with p≤0.05.

Comments from administrators on improving or modifying the current SROI process included the following: 1.) have course instructors inform students that their evaluations are actually used and present, to the students, some of the uses; 2.) provide additional space on the evaluation forms for student comments and have faculty encourage student comments; 3.) an SROI process should only be one aspect of evaluating teaching; and 4.) concerns with comparing student evaluations, i.e., lower-level classes vs. upper-level classes and large class enrollment vs. small class enrollment.

Faculty

Faculty gave the current SROI process an average rating of 4.365 using the 0-9 scale. In analyzing the faculty data, a marginally significant difference was found between the number of years a faculty member had been at NDSU and their rating of their general impression about the current SROI process (p=0.07). No significant difference was found between the average rating given by women versus men (p=0.617). No significant difference was found between a person's academic status and their rating (p=0.247). No significant difference was found among colleges (p=0.611).

Since faculty were directly asked their opinions as to “how they would like to see the process used,” an average ranking can be established, as shown in Table 6. There is a statistically significant difference in all of the average rankings.

Table 6. Results for Faculty (n=63)

(how they would like to see SROI data used)

Uses of an SROI process	Average (Ranking)*
Improvement of Teaching	1.34 (1)
Improvement of Course Content	2.02 (2)
Promotion, Tenure, and Salary Decisions	3.16 (3)
Making Information Available to Other Students	3.67 (4)

* All are significantly different with p≤0.05.

The average rating for administrators, faculty, and students related to the overall SROI process was low. All were well below 5.0 using the 0-9 scale. None of the averages are significantly different at an alpha level of 0.05. A summary of the ratings of the SROI process for all three groups is given in Table 7.

Table 7. Group Ratings of the Current SROI Process

Average rating given by:	95% confidence interval
Students (n=171)	4.025±0.320 *
Administrators (n=31)	4.452±0.698 *
Faculty (n=63)	4.365±0.421 *

Scale: 0 to 9 (“0”=Useless and “9”=Very Helpful)

* Not significant at α=0.05.

After analyzing the responses from all three groups concerning the current SROI process, there was some indication that the longer an individual is exposed to the current SROI process, the lower their level of satisfaction in the process. The current SROI course evaluation system does not appear to be fulfilling the needs of any of the three groups.

Students, faculty, and administrators all indicated that the number one reason to use course evaluations was to improve teaching. This was closely followed by improving course content. Administrators and faculty agreed that the third use of course evaluations should be in salary, promotion and tenure decisions. This differed with students overall who selected making the information available to other students as their third highest reason. Students in their fourth and fifth years however, actually had put this reason as last and their rankings actually agree with faculty and administrators.

Phase V – Conclusions and Recommendations

The results of this survey indicate that corrective action is needed in order to revise the current SROI process. At NDSU, the Teaching and Professional Services Committee, appointed by the University Senate, was charged with developing a new survey form used in the SROI process. A new form has been developed by this committee which will allow higher quality feedback to instructors and guidance on how to, 1.) improve teaching and 2.) improve course content, however, this form has yet to be officially approved and implemented. Based on the results of this study, these two factors were the top two reasons given by all groups related to the value of an SROI process.

From a faculty viewpoint, data must be collected related to faculty satisfaction with the revised process. Faculty and student buy-in is critical. If student response is not perceived as valued by the faculty and if it is not conveyed to future students (i.e., complete the loop), then student motivation will decrease, as will the level of effort that students expend on the SROI process. In essence, the validity of the data may regress and the value of the overall process may lose credibility. Administrators also need to be satisfied with the process. They need to feel comfortable with using the results of the course evaluations for promotion, tenure, and salary decisions.

Follow-up measurements and continual evaluation must be included in the revised process. It is anticipated that one year after starting the revised SROI process, surveys will be given to a random sample of students, faculty, and administrators to collect data related to the revised process. Adjustments, if any, will be made based on the information obtained from these surveys. Approximately every three years additional follow-up surveys will be conducted.

 

References

Giesey, J.J., Chen, Y., and Hoshower, L.B. (2004). "Motivations of Engineering Students to Participate in Teaching Evaluations," Journal of Engineering Education, 93, 303‑312.

Greenwald, A.G (1997). Validity concerns and usefulness of student ratings of instruction, American Psychologist, 52, 1182‑1186.

Koslowski III, F. A. (2005). Expectations, challenges, and frustrations: faculty and administration quality and assessment, Academic Leadership, 3, 22 pages.

http://www.academicleadership.org/volume3/issue3/articles/16/article.html

Langbein, L.I. (1994). The validity of student evaluations of teaching. PS: Political Science & Politics, 27, 545‑552.