ASSESSMENT OF STUDENT LEARNING PRACTICES: THE PERCEPTIONS OF FACULTY MEMBERS AND ADMINISTRATORS AT A PUBLIC UNIVERSITY IN THE UNIVERSITY SYSTEM OF GEORGIA

James L. Williams, Ed.D .

Background

According to Banta (2002), the assessment of student learning movement in higher education began in response to public criticism of poor program and student performances in higher education. These criticisms lead to a reduction in funds from both federal and state government, the largest funding sources in public higher education (Deming, Doyle, & Woods, 1993). Federal and state sources of funding then began to establish mandates to promote accountability in student learning at public colleges and universities. These mandates caused accrediting agencies to place emphasis on assessment as a means of measuring student learning at public colleges and universities. Soon it was established that all institutions of higher education must be accredited to receive federal and state aid (Deming et al., 1993). As a result of the mandates, more than 80 percent of the U.S. colleges and universities began to operationalize assessment of student learning practices so that they could meet the demands of this reform movement in higher education. The assessment movement was unofficially founded at the very first national conference of assessment in higher education, held in the fall of 1985 in Columbia, South Carolina (Banta, 2002). This conference makes the scholarship of assessment a little more than 20 years old.

There is a discrepancy in the utilization of assessment practices. The Federal and state funding sources mandate that colleges and universities maintain accreditation status to receive government assistance and accrediting agencies require colleges and universities to include assessment practices at the individual institutions to maintain accreditation. However, educators remain divided on opinions and attitudes toward the definition, purpose, and importance of assessment practices in higher education (Ewell, 2002). The mixed views toward the practices included under the assessment movement have created much debate among educators.

Method

The primary purpose of this study was to determine the opinions of selected faculty members and administrators at a public university in the University System of Georgia regarding the most important assessment practices of undergraduate student learning. In addition it was to provide findings that would enhance the assessment plans of University System of Georgia member institutions and reflect the most important practices. Finally it was to provide a basis for further discussion for implementing standard practices of assessment of undergraduate student learning.

Research Questions

To establish a complete understanding of the assessment phenomenon at the participating school, a triangulated method was used to collect the data. All data collected was used to answer the following three research questions:

1. What are the opinions of faculty members at a public university

regarding the most important assessment practices?

2. What are the opinions of administrators at a public university

regarding the most important assessment practices?

3. Are there any differences in opinions between faculty members and administrators at a public university regarding the most important

assessment practices?

Participants

The University of Georgia (UGA) is the largest and most comprehensive public university in Georgia and for these reasons, it was selected to provide the participants for this study. There were a total of 810 faculty members and 179 administrators (academic department heads and directors) at the University of Georgia at the time of the study (Board of Regents, 2005). A sample of 230 faculty members and administrators in this study was selected from the total population of 989 and this amount was chosen because it would provide the researcher with a sampling error of five percent (Ravid, 2000).

Instrument

A survey instrument (Appendix A) was developed and piloted by the researcher prior to administering it to the faculty members and administrators at the University of Georgia. The instrument was reviewed by a panel of educational experts from Georgetown University, The University of the District of Columbia and from the Richmond County Georgia Public School District to determine its validity. The survey was designed with internal consistency, so that the instrument is reliable. The statistical procedure Pearson Product Moment was used to determine the survey instrument’s correlation coefficient and the item had substantially high (r = .63) correlation (Ravid, 2000). The Split-Half Method was used to demonstrate the instrument’s reliability which was .77. The Standard Error of Measurement (SEM) was 1.12. This measurement confirmed little error in the survey instrument. The reliability of an instrument affects its SEM and the lower the reliability, the higher the SEM (Ravid, 2000). When an instrument has a high level of reliability (i.e. r is closer to 1), the smaller the margin of error. Since this survey instrument had high reliability, it had a low SEM.

Procedures

A systematic random sampling technique (every fourth faculty member and administrator) was used to select the names of 230 potential participants to compete the survey. A cover letter and survey were sent to the survey participants on February 28, 2005. A consent form requesting permission to conduct a fifteen-minute interview was sent to 14 deans at the University of Georgia on February 28, 2005 as well. The data from the study were collected during between February and May 2005. Once the completed surveys began to return, the data were prepared for analysis. The total number of useable surveys were tallied and coded. Frequency counts, tables and percentages were used to organize and report the results of the descriptive data in section I of the survey instrument. The categorical groups created were as follows: 0-5 years, 6-10 years, 11-15 years, 16-20 years, 21-25 years, 26-30 years and 30+ years.

Results

Administrators and faculty members from 48 academic schools and

departments

(n = 63) participated in the study. The data revealed that

respondents from the department of mathematics education, the school of education and the animal science department responded with the greatest frequency. Section I of the survey instrument asked for demographic information about the respondents. The first question asked respondents to indicate their current classification as either a faculty member or an administrator.

The data revealed that 44.2 percent (n = 27) of the respondents were

administrators and 55.7 percent (n = 34) were faculty members. These amounts constitute 30.6 percent (n = 88) of administrators and 23.9 percent (n = 142) of faculty members from the total sample population. Thus, a greater percentage of administrators responded to the survey. In the second question, respondents were asked to indicate the number of years they have been employed at UGA. The responses were categorized by 0-5 years, 6-10 years, 11-15 years, 16-20 years, 21-25 years, 26-30 years and 30+ years.

The data revealed that 19.6 percent (n = 12) of the respondents were employed at UGA for 0-5 years, 18 percent (n = 11) 6-10 years, 9.8 percent (n = 6) 11-15 years, 16.3 percent (n = 10) 16-20 years, 13.1 percent (n = 8) 21-25 years, 16.3 percent (n = 10) 26-30 years and 6.5 percent (n = 4) 30+ years. Employees from the 0-5 and 6-10 years categories responded to the survey with the greatest frequency. The mean years of employment at UGA was 15.7 years (n = 61) and the median years of employment at UGA was 16.7 years (n = 61). The standard deviation was 9.7 years.

The third question asked respondents to indicate the number of years they have been employed in higher education. The responses were categorized by 0-5 years, 6-10 years, 11-15 years, 16-20 years, 21-25 years, 26-30 years and 30+ years. The data revealed that 3.2 percent of the respondents (n = 2) were employed in higher education for 0-5 years, 13.1 percent (n = 8) 6-10 years, 8.1 percent (n = 5) 11-15 years, 8.1 percent (n = 5) 16-20 years, 16.3 percent (n = 10) 21-25 years, 26.2 percent (n = 16) 26-30 years and 24.5 percent (n = 15) 30+ years. Employees from the 26-30 and the 30+ years categories responded to the survey with the greatest frequency. The mean years of employment in higher education was 22.1 years (n = 61) and the median years of employment was 25 years (n = 61). The standard deviation was 9.7 years.

Discussion

Findings for Research Question One

The analysis of Section II data revealed answers to research question I: What are the opinions of faculty members at a public university regarding the most important assessment practices? Answers to questions 1 – 4 of this section revealed strong support for the use of assessment practices at the university (M = 3.06147525) to include purposes of demonstrating accountability and obtaining accreditation. The data showed the respondents did not support the use of assessment to determine funding (M = 2.823529).

Answers to questions 5 – 25 revealed the assessment practices faculty members rated as most important for use at UGA. Thirteen assessment practices received ratings with a mean score that exceeded 3.0. The most supported assessment practices were the assignment of grades (M = 3.382353), followed closely by the research paper (M = 3.294118), and faculty developed tests (M = 3.235294). Other assessment practices in order of suitability included the written report (M = 3.235294), peer assessment (M =3.205882), service learning (M = 3.117647) and the student portfolio.

Findings for Research Question Two

The analysis of Section II data revealed answers to research question two: What are the opinions of administrators at a public university regarding the most important assessment practices? Answers to questions 5 – 25 revealed the assessment practices administrators rated as most important for use at UGA. Eleven assessment practices received ratings with mean scores exceeding 3.0. The most accepted assessment practices were the research paper and the tutor-led assessment (M = 3.518519), followed closely by the capstone project (M = 3.4444444) and the written report (M = 3.2222222). Other important assessment practices in order of suitability included assignment of grades, student administered surveys and the exit interview (m = 3.074074).

Findings for Research Question Three

Additional analyses of Section II data revealed answers to research question Three: Are there any differences in opinions between faculty members and administrators at a public university regarding the most important assessment practices? An independent t-test found there was a significant difference (t = 6.67, p < .05) between the opinions of administrators and faculty members, regarding the most important assessment practices. An analysis of the mean responses to the first four items on section II of the survey revealed both groups felt the use of assessment practices was important at UGA. Both groups also indicated assessment is important for demonstrating accountability at the institution.

The two groups also demonstrated little support for using assessment to obtain funding for the institution. However, the two groups differed in their opinions of the importance for using assessment to obtain accreditation. Administrators showed strong support for using assessment to obtain accreditation, while faculty members showed little support for using assessment to obtain accreditation. A possible explanation for this response is administrators are charged with the responsibility of providing responses to performance reports and they must cope with the consequences and reductions in funding.

Using chi square, an analysis of the two groups’ responses to the individual survey items revealed they differed significantly on the importance of 16 of the 25 survey items to include the use of various assessment practices for the following purposes: accreditation, student administered survey, tutor-led assessment, faculty-developed tests, service learning, student portfolio, the standardized test, the regular assignment of tasks and projects, the oral report, the exit interview, the group project, multi-source assessment and daily homework and quizzes. The mean scores of respondents according to the number of years they were employed at UGA were compared using a one-way ANOVA. No significant difference was found. Respondents from the seven groups (0-5, 6-10, 11-15, 16-20, 21-25, 26-30, 30+) did not differ significantly in their responses to the survey items based on the number of years employed at UGA.

An analysis of the mean scores of respondents according to their number of years in higher education was also compared using a one-way ANOVA. No significant difference was found. Respondents from the seven groups (0-5, 6-10, 11-15, 16-20, 21-25, 26-30, 30+) did not differ significantly in their responses to the survey items based on the number of years employed in higher education. The data suggest uniformity among employee responses, based on the number of years employed at UGA and in higher education. A possible explanation to this uniformity may be an overriding factor such as employee classification.

Section III of the survey allowed respondents to identify additional assessment practices not mentioned in the survey and to add comments regarding the survey. There was minimal response to this item. The common response to this item was the need to establish additional appropriate assessment practices.

Conclusions

The conclusions of this study are reported as they relate to the research questions presented earlier. This study focused on the opinions of two critical stakeholders in at a public university in the University System of Georgia, administrators and faculty members. A review of the literature revealed that there are many discrepancies between administrators and faculty members regarding their opinions of the most important assessment practices. One source of the discrepancies is that assessment practices are often chosen based upon preferences that are mandates from external organizations (i.e. government and accrediting agencies), rather than by administrators and faculty members collectively.

The purpose of this study was to determine the opinions of selected faculty members and administrators at a public university in the University System of Georgia regarding the most important assessment practices of undergraduate student learning. It was concluded that the research paper, assignment of grades, the oral report and the written report are regarded as the most important assessment practices for public universities in the University System of Georgia. The capstone project, service learning, the faculty developed test, self assessment and tutor-led assessment all received favorable ratings.

There were no significant differences in opinions based on respondents’ number of years employed at the University of Georgia and the number of years employed in higher education. Two respondents participated in the interview and the data collected revealed that the participants felt assessment should be appropriate to the academic discipline. Based on the findings from the survey and the interviews of the faculty members and administrators of a public university in the University System of Georgia, five recommendations were made for further research and practice. Regarding research, it was recommended that additional on-going research studies securing opinions of other public university stakeholders be conducted to complement this study. It was also recommended that additional on-going research studies securing opinions of other stakeholders for all universities in the University System of Georgia. Furthermore, it was recommended that additional research studies be conducted to determine the most important and appropriate assessment practices for all types of colleges and universities within the University System of Georgia.

Regarding practices, it was recommended that a system-wide consortium be

established to determine the most important and appropriate assessment practices for public universities in the University System of Georgia. It was also recommended that a system-wide committee representing all facets of higher education be organized for the purpose of identifying the most important assessment practices and assessment plans to use for institutions of higher education in the University System of Georgia, prior to linking college funding and accreditation to assessment.

References

Altbach, P., Arthurs, A., Boyers, E., Breneman, P., Callan, P., Cerych, L., Clark, B.,

Cross, K., Diamond, M., Edwards, P., Geiger, R., Glenny, L., Gordon, M.,

Kauffman, S., Koltai, L., Levine, A., McLaughlin, S., Perkins, S., Riesman,

P., Smelser, N., Smith, V., Stewart, D., Trow, M. 1993. Higher learning in

America 1980-2000. Baltimore, MD: The Johns Hopkins University Press.

Banta, T., 2002. Building a scholarship of assessment.

San Francisco, CA: Jossey Bass.

Banta, T., Palomba, C . 1999. Assessment essentials planning, implementing, and

improving assessment in higher education. San Francisco, CA: Jossey Bass.

Black, H., Duhon, D. 2003. Evaluating and improving student achievement in

business programs: The effective use of standardized assessment tests.

The Journal of Education for Business, pp. 90-98.

Board of Regents of the University System of Georgia. 2005. Information digest.

Accessed February 8, 2005, http://www.gsu.edu/.

Deming, M., Doyle, K., Woods, S. 1993. A comprehensive assessment plan for

professional preparation programs in health education at Eastern Illinois.

University. Journal of School Health, pp. 1-4.

Ewell, P. 2002. Building a scholarship of assessment: A brief history of

assessment. San Francisco, CA: Jossey Bass.

Heywood, J., 2000. Assessment in higher education student, learning, teaching,

programmes and institutions. United Kingdom: Jessica Kingsley Publishers, Ltd.

Kameoka, V., Lister, L. 1991. Evaluation of student outcomes in MSW Programs

Journal of Social Work Education, pp. 25-33.

Ravid, R. 2000. Practical statistics for educators. Lanham, MD; University Press

of America.