|
From Academic Leadership Empirical Research Fort Hays State University One of the most significant changes thrust upon higher education over the past 30 years is undoubtedly the increased focus on assessment of student learning outcomes. A variety of justifications have emerged to provide sound reason for this substantial change. For many, the link between improving student learning outcomes and enhanced graduates earnings is an assumption that has prima facie validity. Moreover, while many academics may find the nature of the relationship too applied for their tastes, the reality is that most students attend college to improve their life situation – earnings upon graduation are one among few significant indicators of their ability to succeed. Research in the field of learning outcomes efficacy is sparse and has not responded to this asserted relationship. Purpose The primary motivation for exploring the relationship between learning outcomes assessment and graduate earnings is to simply document the assumed link so that educators have some basis for claiming long-term benefits of outcomes assessment. Furthermore, the scope of the project was limited to exploring learning outcomes assessment in the state of Kansas as this was deemed a preliminary inquiry. Specifically, this article seeks to address the following three questions: 1. Have academic programs in Kansas adopted assessment plans? 2. Does student ability play a role in earnings differentiation upon graduation? 3. Is there a difference in earnings between the highest, moderate, and lowest performing graduates? Methods In early 2007 a survey method was designed to learn more about the relationship between assessment of student learning and graduates earnings. Sample. It was determined that about 221 undergraduate departments existed in public higher education in Kansas. A census of all undergraduate programs was selected as the sampling procedure so that coverage of the relationship was exhaustive and uniform. Departments were contacted through their academic chairs and all were given the “Assessment and Learning Outcomes Improvement Statewide ROI Benchmarking Study” survey and instructions. The first mailing occurred about the beginning of June 2007 with a follow-up survey sent to all department chairs not responding around August 2007. A total of 76 departments (34%) responded to the second request by the deadline. Instrument. A survey instrument, the “Assessment and Learning Outcomes Improvement Statewide ROI Benchmarking Study” was created by asking questions about the following basic areas: · Annual number of graduates · Classification of department · Source of SCH · Faculty interest in assessment · Estimated salary of highest 10%, middle 50%, and lowest 10% of graduates · Typical sectors for employment for highest 10%, middle 50%, and lowest 10%. Each department chair received a printed copy of the instrument with instructions. Procedure. A database of all departments with undergraduate programs at Kansas public universities was created. The survey instrument was sent to each department for completion. A follow-up mailing was sent if no response was received. The data was compiled for analysis. Results Data analysis of the 76 responses was conducted providing the following results. Table 1. Mean and Median of Selected Variables.
Departments were also asked to report the typical source of student credit hour production. General education was noted as the primary source for 20% of departments, major courses were noted as the primary source for 49% of respondents, and some departments (26%) noted that student credit hours were from both sources. Existence of assessment strategy. Departments were asked to report on the existence of an assessment plan. About 16% of departments reported having no assessment plan implemented, 11% reported that they just implemented an assessment plan, and most departments (70%) reported having an assessment plan in place and collecting outcomes data. Finally, departments were asked to gauge the level of faculty interest in assessment. Most programs reported that their faculty had more than pedestrian interest in assessment (67%), but 27% of departments reported that faculty had no more than minimal commitment, if any. In relation to the first research question, the conclusion that emerges is that faculty are generally interested in assessment and most programs report participating in assessment. Relationship between assessment interest and graduate earnings. A correlation analysis was conducted looking at the relation between faculty interest in assessment and the existence of an assessment plan with the six levels of estimated graduate earnings. There were no significant correlations emergent from the analysis (Table 2). Table 2. Correlation Matrix of Assessment and Graduate Earnings.
* notes significance at the p > .05 level. In relation to research question two, there is no documented relationship between enactment of department assessment strategies, faculty interest in assessment, and graduate earnings. Differences in earnings – within performance level. One of the objectives of this preliminary study is to begin to document the assumptions that are often made in the academic world, namely, that students that perform at a higher level will have greater earning potential. While the methodology in this study is elementary, there is clear evidence that statistically significant within-group differences exist (Table 3). Table 3. Paired t - test Analysis of Highest, Middle, and Lowest Performing Graduates.
* notes significance at the p = .01 level, ** at the p = .001. Thus, the conclusion that can be drawn from this sample suggests that there is significant variance in earnings at the same performance level. Differences in earnings – between performance levels. More interesting than earnings variance at the same performance level is the significance of differences between performance levels. Table 4 documents the statistically significant relationships between performance levels. Table 4. Paired t – test Analysis Comparing the Highest, Moderate, and Lowest Performing Graduates.
* notes significance at the p = .01 level, ** at the p = .001. The analysis clearly found that students at higher performance levels had significantly higher estimated salaries and answers the third research question. In addition to projected earnings of students, the study also looked at the most common employment sector of graduates by student learning outcome performance. Tables 5, 6 and 7 document differences by employment sector. In addition, Figure 1 compares total percentages across all student learning performance levels. Table 5. Employment Sectors for Highest 10% of Graduates by Department Type.
Table 6. Employment Sectors for Middle 50% of Graduates by Department Type.
Table 7. Employment Sectors for Lowest 10% of Graduates by Department Type.
Figure 1 details a comparative analysis of employment sectors for graduates by highest 10%, middle 50%, and lowest 10%. Figure 1. Employment Sector Analysis by Performance Level.
While the comparison was not tested for statistical significance, the following trends emerge from the data: 1. Graduates with lower levels of achievement are more commonly employed in the service and business related sector. 2. Graduates with higher to moderate achievement are more likely to be employed in the education, medical, public/government, and business sectors. Across all levels of performance, more graduates are projected to be employment in the business sector than any other sector. Discussion and Conclusions Related to the first research question, it appears that the majority of departments sampled have and assessment plan in place and are collecting some sort of outcomes data. It is not surprising that about the same number of departments report that their faculty have at least a minimal interest in assessment. An obvious conclusion emerges from these correlative findings – assessment plans become enacted when faculty have an interest in assessment. While there is no doubt that many departments have assessment planning mandated, the enactment of those plans does not occur without some level of faculty commitment to the process and improving outcomes. Regarding the second and third research question, the link between student academic performance and improved earnings, the findings present an interesting case. This study assessed earning differences between performance categories (highest 10% compared to middle 50% or lowest 10%) as well as within performance categories (highest earnings of highest 10% compared to lowest earnings of highest 10%). In the within comparisons, in every case there was significant variance difference between the highest and lowest earnings as Table 3 documents. This finding simply explains the phenomenon of students with similar abilities taking a wide variety of earning producing jobs – some top grads get great offers, some do not. The more interesting analysis relates to differences between performance levels. When the highest earnings of the highest 10% performing graduates are compared to the highest earnings of the middle 50% graduates there is also a significant difference. In every analysis, the higher performing student was estimated to have greater earnings when compared to the 50% or lowest 10% graduates. These results suggest a qualified link between performance and earnings, with the realization that there is a natural difference asserted because not all graduates of similar abilities will receive the same earnings upon graduation. Future Study Empirical research linking student performance outcomes and “real world” outcomes has been slow to develop. This study has taken a small step in this direction, but the general nature of the questions is both a benefit as well as a deficit. From our limited vantage point it seems that future study should focus on the following directions to more fully develop the body of research in the field. 1. In this study the sample was limited to a statewide distribution. In order to generalize the results of future research, larger and more geographical distributed samples should be queried. 2. The survey instrument employed in this study was basic and predicated on estimates of student learning and earnings. Future studies should gather actual student learning outcome data (i.e. major field exam score) and attempt to relate that to tangible “real world” outcomes (i.e. annual salary two years post-graduation) to the extent possible. 3. The current study was designed to access one estimated outcomes data point – “upon graduation”. Understanding the impact of higher learning on graduates (and non-graduates for that matter) over their lifetime will provide a much better answer to the question “What good is a college degree?” 4. The unit of analysis in this study was the major (department more specifically). It is a testable assumption that department chairs have insight into the relationship between student performance and actual outcomes. This assumption might well be borne out, but more work at the conceptual level testing this assumption would be advisable. © Copyright 2008 by Academic Leadership |
