Current Issue:
Volume: 7
Issue: 3
ISSN: 1533-7812
|
|
|
Last Updated: Aug 11, 2009 - 4:32:37 PM |
y cant they rite?: Integrating Writing Assessment Across the Undergraduate Political Science Major
By Shala Mills, Bryan Bennett
Volume 7 - Issue 2
May 5, 2009 - 9:52:16 AM
|
Email this article
Printer friendly page
|
Introduction
Historically, student assessment in the Political Science Department at Fort Hays State University was left to the individual faculty member to embed into his or her courses via exams and writing assignments. Our curriculum and learning objectives were based largely on faculty interest in particular courses and on broad perspectives of what substantive knowledge a political science major should demonstrate. Over the years, writing courses such as advanced research methods and upper division theory courses served as unofficial capstone experiences. As such, approaches and expectations varied depending upon who was delivering the course.
A few years ago we began to rethink our curriculum. To begin, we made changes to the major and made new hires in order to emphasize areas of specialization in the discipline. In addition, we attempted to bring structure to our capstone experience by creating a senior thesis requirement. Early iterations of both the new curriculum and the senior thesis were flawed. Fortunately, we began to identify these flaws simultaneous with an institutional emphasis called Year of the Department (YOTD) in which departments across our campus were asked to engage in an in-depth program review. Those efforts were reinforced by our participation at the 2007 American Political Science Association (APSA) Teaching and Learning (TLC) conference and the 2007 American Association of Colleges and Universities (AAC&U) conference. These activities informed our thinking about our curriculum both in the substantive knowledge majors should have and in the skills and values majors need in order to be successful in undergraduate, graduate, and professional endeavors. With better articulated learning objectives in mind, we redesigned our curriculum and turned serious attention toward meaningful assessment of our students’ mastery of key learning objectives.
The Larger Context: A New Curriculum for the Major and a New Assessment Plan
Although this paper will focus on assessment efforts and improving writing, we begin by describing the larger context and how the writing portion fits into the broader effort. In 2007 Fort Hays State University announced a new initiative called YOTD. YOTD was designed as “an ongoing strategic initiative for orchestrating change and aligning people, systems and culture at the basic unit level in the Division of Academic Affairs. . . .” (FHSU YOTD). Relying upon Massey’s “Honoring the Trust” (Massey 2003), this call to engagement included five essential areas:
• Desired learning outcomes
• Design of curricula
• Design of teaching and learning processes (redesign of large courses and standardization of courses with multiple sections)
• Student learning assessment
• Use of results/feedback for curricula improvement and quality assurance.
(FHSU YOTD).
The timing of this initiative was excellent for our department. Recent retirements, resignations, and reorganization had resulted in several new faculty members, new leadership, and a natural inclination toward reconsideration of the program. We began by comparing our program with peer and benchmark institutions around the country. We looked at core curriculum, areas of concentration, methods of delivery, and extracurricular opportunities. Not surprisingly, we found that in many ways our program resembled other Political Science departments in Kansas and across the country, and yet in other ways it enjoyed its own areas of distinctiveness. We discussed those curricular and extracurricular similarities and differences and evaluated their strengths and weaknesses, ultimately agreeing on a revised set of courses that would comprise our program’s “core” as well as a set of eight concentration areas encompassing both traditional as well as emerging subfields.
As these considerations were underway, two of our faculty members traveled to New Orleans for the Annual Meeting of the AAC&U (The Real Test: Liberal Education and Democracy’s Big Questions, AAC&U, January 18-20, 2007). A concurrent session on “Assessing Moral Learning: Two New Approaches,” by King's College faculty Jennifer McClinton-Temple and Joel Shuman, sparked the interest of the FHSU faculty. The FHSU faculty were especially attracted to results mapping the King’s College duo described involving a seven-step ladder for tracking student moral development. The approach began with required activities, tracked whether students then chose volunteer endeavors, and recorded whether students were moved to make life-changing commitments. This ladder approach, measuring student engagement and development through student participation on a progressive scale, struck both FHSU faculty members as a useful model for the assessment component of our YOTD work.
A great deal of effort over the next several months produced a new curriculum as well as new assessment efforts. In the end we agreed on a 40-hour major including a 22-hour core and 18 hours of concentration coursework (nine hours in each of two subfields chosen by the student). In keeping with institutional and departmental goals, the faculty agreed on four areas of emphasis across the major: 1) writing, 2) speaking, 3) civic engagement/service learning, and 4) internationalization.
We decided on a student portfolio system for collecting learning assessment materials. In keeping with Wright’s advice in “Assessment Methods – A Close-Up Look” (Wright 2003), we used a system that would:
• Collect samples of work, not everything from everybody
• Use electronic storage and retrieval
• Give students responsibility for maintaining the portfolio
• Invest in good criteria for education’s sake
(Wright 2003).
The Orientation to Political Science course was the logical place to initiate the portfolio. Students would then add material to the portfolio in several courses spread across the program. Finally, the Senior Capstone course would serve to complete the portfolio. The portfolio is used for departmental assessment, but also serves as an important resource for students as they develop a resume and apply to internships, graduate school, and/or places of employment. We are presently using a paper portfolio, but we are exploring opportunities for moving to an e-portfolio at some point in the future.
Agreeing with Golich’s work in “Thinking About Assessment”, an important component of our assessment effort was “making expectations explicit and public” (Golich 1998). Before our student portfolio can serve as a repository of sample work from students and a mapping of their progress in key areas of development, it must first provide students with notice of what it is we are expecting from them. As a result, the portfolio contains an advising worksheet that lays out the curricular requirements for the department major as well as the university’s general education and degree requirements. It also contains departmental affinity diagrams that illustrate how departmental assessment efforts help us determine whether students have met the expected learning outcomes that our curriculum is designed to produce in our majors.
The portfolio’s eight major components are:
1. Student Information
2. Initial Self-Assessments
3. Department Affinity Diagrams
4. Advising Worksheet
5. Expected Learning Outcome Rubrics and their Attachments
a. Writing
b. Civic Engagement
c. Internationalization
d. Presentation
6. Resume and Cover Letter
7. Senior Thesis
8. Final Self-Assessment
The Expected Learning Outcome (ELO) Rubrics each use a three-step ladder loosely modeled after the seven-step ladder introduced by King’s College at the 2007 AAC&U conference mentioned previously. The three levels of development are:
1. Required Elements (core and concentration courses required for that ELO)
2. Elective Elements (elective courses, additional majors/minors/certificates, extracurricular activities, or other activities that develop the student beyond what is required for that ELO)
3. Change Elements (long-term change and/or plans for on-going development in or commitment to that ELO)
Click on the image for more detailed information.
Click on the image for more detailed information.
Click on the image for more detailed information.
Click on the image for more detailed information.
For the required element of each ELO, we designated courses from which sample work would be included in the portfolio. Although many courses in our curriculum might as a matter of fact require students to do work in one or more, or indeed in all, of the ELO areas, we designated certain courses as writing intensive (WI), civic engagement/service learning intensive (CE/SL), international intensive (I), and presentation (P) intensive. We also designed common grading rubrics for each ELO area, and each assignment included in a portfolio area is accompanied by a common grading rubric for that area.
For the elective element, we identified additional courses, programs, and extracurricular activities that a student might choose to engage in that would further develop student growth in the ELO area. For example, the elective element for the civic engagement/service learning rubric includes volunteer activities, campus and community clubs and organizations, and leadership roles held by the student. Obviously, not all students will avail themselves of these elective opportunities. However, the rubric itself helps students understand that engaging in such opportunities is beneficial for their growth and development, and students who do choose to participate will include at least one reflection paper for each area of the rubric in which elective elements are present.
For the change element, we ask students to indicate whether they have made long term commitments in an ELO area. For example: have they chosen a graduate program in International Relations as a result of their internationalization experience, have they decided to become a political speech writer as a result of their writing intensive experience, or have they made a commitment to a particular non-profit organization’s work as a result of a service learning experience? Again, not all students will have experienced long-term change in every, or perhaps even in any, ELO area, but it is our hope that the rubric will serve to encourage students to consider long-term change as a desirable result of their educational experience. To the degree that they recognize such long-term change, they are asked to include at least one reflection paper for each area of the rubric in which change elements are present.
Writing Assessment in the Major: Identifying the Problem
Understanding the overall assessment model helps to put our writing assessment in context. Concern over the quality of student writing is not new, nor is it unique to our department or campus. In our experience, many undergraduate students write at a less than adequate level. A review of studies done at other universities, writing across the curriculum materials, and other sources, reveals that the problem with writing at the undergraduate level is not limited to smaller universities, political science programs or to any one factor in general.
This conclusion is supported not only by anecdotal evidence from teachers, reports in popular media, independent college assessment programs such as the Collegiate Learning Assessment instrument (CLA) and the National Survey of Student Engagement (NSSE), but is also evidenced by the existence and proliferation of journals, conferences and scholarship devoted to the subject. A study of popular journals on the subject such as the Writing Across the Curriculum Journal or the Writing Center Journal reveals a plethora of material ranging from the application of writing principles to algebra courses to developing and sustaining writing tutors in a campus environment.
This realization provided the genesis for an in-depth assessment of the writing required within the political science major. This of course meant taking a hard look at the department, its goals, and perhaps most importantly its methods.
As anyone who has engaged in departmental assessment knows, meaningful self-reflection and constructive change can be a difficult task. In addition to FHSU YOTD goals, we turned to literature on the subject of assessment itself. Work by Deardorff and Folger proved helpful in our initial efforts. In their paper “Assessment that Matters: Integrating the ‘Chore’ of Departmental-Based Assessment with Real Improvements in Undergraduate Political Science Education,” Deardorff and Folger identify common difficulties in meaningful assessment efforts (Deardorff and Folger 2002). One such difficulty is simply the compilation of performance data that can be used in a meaningful way. Material found in a self-assessment performed at the University of Connecticut by Bourbeau and Rich was helpful in this regard, providing examples of self assessment and data collection along with a sample assessment form used in their study of political science undergraduates (Bourbeau 2007). In addition, the American Association for Higher Education (AAHE) standards for assessment of student learning proved insightful (AAHE 1992). These and many other sources were consulted as part of our effort to identify and resolve the problem of student under-performance in writing (Astin 1993; Clifford 2006; Mullin 2007; Sherman and Waismel-Manor 2003).
This task was complicated to an extent by the realization that many writers enter college without fundamental background skills and are thus at a disadvantage when it comes to even beginning the task of writing development (Dickson 1995). So, not only must internal methods and goals be reviewed, student writing ability at the outset must also be measured in some way. So, clearly, we needed students to engage in self-assessment at the beginning of their educational experience with us so that we could better determine what their entry level skills were rather than assuming a level of skill that might or might not be present.
As part of our investigation we uncovered an article by Joan Mullin and Susan Schorn entitled “Enlivening WAC Programs Old and New” (Mullin and Schorn 2007). This article focused on writing across the curriculum programs and their potential pitfalls. It also served to highlight one aspect of our program that was perhaps at fault. As noted previously, development of writing skills within the major had for the most part been left to the individual instructor without significant oversight. It was not that our faculty was not assigning writing projects, but more so that meaningful assessment was lacking and the addition of more writing projects was of course not resolving the underlying problem. We could not fix the problem of weak student writing simply by requiring more of it; we needed a more sane approach.
Addressing the Writing Problem
Development of a “Writing Rubric”
This self-assessment led to the creation of a writing rubric which attempts to incorporate several key factors of student writing. The writing rubric itself is the result of considerable research, analysis and deliberation undertaken over a period of several months. We started by gathering writing rubrics that individual faculty in our department had used in some of their classes. We next gathered sample grading rubrics from colleagues in other departments. Finally, we searched the Internet both for examples of writing rubrics that might provide additional perspectives on the substantive content of the rubric and for other sorts of grading rubrics that might offer a useful format design.
We identified six characteristics that broadly encompassed the expectations our faculty agreed were common for almost any written assignment:
1. Follows Directions
2. Thesis
3. Use of Evidence
4. Analysis, Logic and Argumentation
5. Organization
6. Mechanics
We attempted to provide guidelines which are clear, consistent, understandable, and achievable. To that end, we included detailed descriptions of A, B, C, D, and U level work in each of the six categories.
Click on the image for more detailed information.
We have also attempted to create a rubric that is flexible enough to have general applicability regardless of course, level, or type of assignment. Moreover, although we wanted the rubric to contain common elements all the faculty could agree were necessary in almost any writing assignment, we also wanted the rubric to be adaptable for faculty such that elements could be added or removed, or points allocated differently, according to a particular course or assignment’s needs. Thus, we included specific language notifying students that “Faculty may adapt, adding or removing characteristics, and/or re-weighting components, as appropriate for individual assignments.”
Writing Focused Curriculum
The designation of writing intensive courses in the curriculum was an important part of our assessment process. We deliberately created the Orientation and Senior Capstone courses as bookends for the major, providing us with an opportunity to initialize and finalize the student portfolios as well as ingrain in students the values embodied in the expected learning outcomes for the major. Including both of these courses as writing intensive courses for the student portfolio provided the perfect opportunity to capture each student’s “before” and “after” work in the major, giving us a vivid snapshot of their progress from beginning to end. The required Research Methods course was also an obvious choice as a writing intensive course, providing students with their most rigorous research and writing task in the major and, for many, a foundation for their senior thesis project. Finally, we decided to require students to take one WI course in his/her primary concentration area in the major, reinforcing the importance of strong writing skills across the discipline’s subfields and across the student’s particular disciplinary interests. Thus, every Political Science major at FHSU will have at least one WI assignment (accompanied by its grading rubric) from the Orientation, Capstone and Research Methods courses. Each student in the major will also have a WI assignment/graded rubric from his/her primary concentration area, but because students may choose from among the available concentration areas, the WI concentration contents will vary from portfolio to portfolio. Note, again, that there is no attempt to include every writing assignment a student completes in each course in the major or even in each of the WI courses in the major. Rather, each student provides only sample work from each WI course.
As noted above, the WI courses are designed to span both the core curriculum and every concentration area so that no matter the path a student chooses, writing is reinforced in an intentional, meaningful way at every level. Each WI course should incorporate the writing rubric explained above, include either a series of short papers or a single paper of some length (ex: 20 pages with multiple peer and instructor reviewed drafts). The end result is that, at a bare minimum, every student will have taken 12 credit hours of designated WI course work consisting of approximately 75-100 pages of writing assignments. This of course does not include any of the other myriad writing assignments found within other courses of the major or the general education program.
As students advance through required WI courses, our hope is to see additional student progression in voluntary ways such as taking non-required writing courses, taking on voluntary projects such as self-directed internships or posts on the student newspaper, or through participation in conference presentations. Ultimately, we hope to witness a commitment to more complex and advanced writing as part of a graduate program or job.
Senior Capstone
The first iteration of our senior thesis produced mixed results. When we originally designed it a few years ago, we contemplated a one-hour independent study model. No faculty member was officially designated as the senior thesis instructor and so students turned to faculty in an ad hoc manner, resulting in disproportionate (and unpaid) overloads for faculty. Some students worked closely with a faculty member, often producing excellent final papers that had gone through several revisions. Other students failed to seek faculty guidance, with sometimes disastrous results for students who lacked initiative, discipline, and/or focus.
We have now incorporated the senior thesis as a project in the Senior Capstone course required of all political science students at FHSU. We have also expanded our Senior Capstone experience so that it includes more than merely a paper of medium length (20 pages). At the end of each academic year our department celebrates scholarship during FHSU’s Research and Creativity Week at a reception featuring faculty research activity as well as student research and writing. All Senior Capstone students share their culminating project, their senior thesis, at this event.
Observations of student performance clearly indicated that some students lacked writing skills in areas that were important for obtaining internships, graduate school admissions, and jobs. As part of the effort to alter this problem we have added resume building exercises, work on personal statements and cover letters, and other writing exercises designed to build student professionalism. These materials are likewise added to the portfolio.
The focus of the resume and personal statement exercise is on the development of formal professional driven writing which is useful for advancement to graduate programs or to the job market. Student core and concentration papers provide an opportunity for personal reflection about the core subject matter of the major and the experiences which shape student development and future goals. Formal presentation of the senior thesis paper provides an opportunity for development of presentation and public speaking skills. As such, the Senior Capstone course as a whole is aimed at finalizing skills developed over the course of the undergraduate education, revisiting core topics from the major itself, developing skills for advancement within the field, and overall producing a more well rounded and skillful graduate.
Preliminary Results
The Rubric
We began implementing the new writing grading rubric in select courses fall 2007 with the idea that the rubric could be tested over the 2007-08 academic year, modified as necessary, and then put into general use with the full implementation of the new major and portfolio in fall 2008. We are a department of four full-time and two half-time faculty and about 60-70 majors. Given the small number of students who began using the departmental writing rubric fall 2007 and continuing on into spring 2008 as well as the 2008-2009 academic year, our feedback has been limited to interviews and focus groups. Although no survey has yet been given students to obtain additional feedback, faculty and student response to the rubric has thus far been almost overwhelmingly positive.
Many students have expressed that the rubric, coupled with written comments, has provided more feedback than they had received on previous assignments in our department as well as in other departments. Furthermore, the dissemination of the rubric and the announcement that it would eventually be used across the political science curriculum was likewise welcomed by the students. The rubric itself provides not only a mechanism for evaluating a finished student paper, but also a means for students to evaluate their own work prior to handing in a final paper.
Many students expressed that this checklist format helped reinforce their understanding of expectations and, thus, assured a stronger paper and many times accurately predicted a student’s likely grade on the assignment. Furthermore, the consistency fostered by the use of the grading rubric was also evident. Students noted that comments were uniform for professors using the rubric even where the professor and course subject matter were quite dissimilar. One anecdote illustrates the consistency promoted in using the rubric. We had multiple professors evaluate a single student’s senior thesis using the common rubric. Although individual faculty scores differed slightly regarding some individual areas of student performance, the overall scores were remarkably consistent. Therefore, it seems that the rubric has served thus far to unify departmental standards and help students prepare a higher quality paper.
In addition, despite its potential flaws, the rubric itself seems to encourage additional meaningful responses from faculty to student submissions. Some concern was raised that use of the rubric might require additional work for faculty. However, the opposite seems to be the case. Faculty members who have used the rubric report that the rubric serves as a form of short-hand for providing consistent, meaningful comment on student work. Indeed, use of the rubric thus far has resulted in less time spent grading written work as it builds commentary into the actual scoring process. Faculty then may use the time normally spent providing feedback on student work to instead highlight additional areas of concern and/or provide more detailed feedback about key weaknesses of the work. Students thus far have noted that faculty comments seem to be uniform and this assists in identifying key areas for improvement in their writing.
Our initial results suggest that the rubric has resulted in somewhat lower grades. Our speculation is that the uniform standards of the rubric impose set guidelines which are readily apparent. These guidelines tend to impose a lower grade distribution as instructor satisfaction solely with the content provided plays a lesser role vis-à-vis considerations such as use of evidence, logic, and argumentation. The result is that what would otherwise be a fairly strong student paper based on content alone may suffer under the rubric if there are some flaws in argumentation or use of evidence for example. While some students expressed this as a concern, we feel that this more accurately reflects the true merit of the paper (including the writing component) and also helps alleviate grade inflation through uniform criteria. Some student feedback indicates that although students feel the major is now somewhat more rigorous, they also feel they have increased rapport with faculty regarding their work and, as a result, are learning and benefiting more from that feedback and are ultimately finding their learning process to be more enjoyable.
Senior Capstone
We offered the expanded Senior Capstone course for the first time spring 2008. Thus far the course has generated positive student involvement, demonstrable improvement in student resumes, improved presentation skills, and, hopefully, will result in an overall higher quality of graduate. Compared to overall performance on senior theses in prior years, the spring 2008 senior theses showed improvement. Moreover, student presentations at the annual scholarship reception were, on the whole, superior to prior years.
Closing Thoughts
So…y cant they rite? There are many reasons why undergraduate students do not write as well as we would like, and this paper does not attempt to address all of them. What this paper does, however, is recognize and attempt to address a few of the causes suggested both by the literature and by our personal experience. What we hope to offer is a thoughtful way for academic departments to set clear goals and expectations, communicate essential connections, and integrate assessment, especially writing assessment, across their undergraduate curriculum.
Our proposal is not about busywork for already overworked faculty and administrators. It is not about student checklists that are all form and no substance. It is about creating a culture for growth and development. Our internal audit was an important process for reconsidering our values and expectations as a department and for articulating those values and expectations internally and externally.
Ultimately, our new curriculum and assessment system is about connections and communication. Each of the four key expected learning outcome areas our department chose to focus on is, in its essence, about communication. Of course we wanted students who mastered the content of the discipline. But we also wanted students who could communicate that mastery–in writing, orally, through engagement, and out into the world.
Writing is one of the most important skills a student obtains during the undergraduate experience. Without solid writing ability both graduate school and quality fulfilling employment are difficult to obtain. Indeed, one might say that quality graduates who can successfully move into graduate school and/or professional settings are the ultimate “assessment” of a program. As educators it is incumbent upon us to do all we can to strengthen this important skill set within our students.
Although full implementation of our new curriculum, student portfolio system, and grading rubrics is still in its infancy in the political science department at FHSU, it is our hope that, through adoption of our new curriculum and assessment system, demonstrable results can be achieved, recorded, and measured.
[This article was modified from a paper presented at the American Political Science Association Teaching and Learning conference in San Jose, CA , February 22-24, 2008.]
http://www.allacademic.com/meta/p_mla_apa_research_citation/2/4/5/5/8/p245580_index.html
References
Astin, A. W. 1993. Assessment for excellence. Phoenix: Ornyx Press.
American Association for Higher Education. 1992. Principles of good practice for assessing student learning. Washington D.C., AAHE.
Barberio, R. P. 2004. The one-armed bandit syndrome: Overuse of the internet in student research projects. PS: Political Science and Politics 37(2): 307-312.
Bourbeau, J. R., and J. Rich. 2007. Undergraduate research writing in political science. Paper presented at the American Political Science Association Teaching and Learning Conference, Charlotte, NC.
Clifford, B. 2006. A question and an argument: Enhancing student writing through guided research assignments. PS: Political Science and Politics 34(3): 653-655.
Deardorff, M. D., and P. F. Folger. 2002. Assessment that matters: Integrating the 'chore' of department-based assessment with real improvements in undergraduate political science education. Paper presented at the annual meeting of the American Political Science Association, Boston, MA.
Dickson, M. 1995. It's not like that here: Teaching academic writing and reading to novice. Boyton/Cook.
Dilevko, J., and L. Gottlieb. 2002. Print sources in an electronic age: A vital part of the research process for undergraduate students. The Journal of Academic Librarianship 28(6): 381-392.
Fort Hays State University, 2006. The year of the department: A call to engagement.
Fox, J. C., and S. Keeter. 1996. Improving teaching and its evaluation: A survey of political science departments. Political Science and Politics 29: 174-180.
Golich, V. 1998. Thinking about assessment. Paper presented at Conference for Chairs Special Panel on Academic Program Assessment. www.apsanet.org/section_563.cfm.
Grauerholz, L. 1999. Creating and teaching writing-intensive courses. Teaching Sociology 27: 310-323.
Massey, W. 2003. Honoring the trust: Quality and cost containment in higher education. Bolton, Mass: Anker Publishing.
Mullin, J., and S. Schorn, 2007. Enlivening WAC programs old and new. The WAC Journal 18: 5-18.
National Survey of Student Engagement Survey: http://nsse.iub.edu/pdf/NSSE2007_US_English_paper.pdf.
Raimes, A. 1980. Writing and learning across the curriculum: The experience of a faculty seminar. College English 41(7): 797-801.
Robinson, A. M., and K. Schlegl. 2005. Student use of the internet for research projects: A problem? Our problem? What can we do about it? PS: Political Science and Politics 38(2): 311-316.
Shaughnessy, M. P. 1976. Diving in: An introduction to basic writing. College Composition and Communication 27, 3, 234-239.
Sherman, D. J., and I. Waismel-Manor. 2003. Get it in writing: Using politics to teach writing and writing to teach politics. PS Political Science and Politics 36(4): 755-763.
Thies, C. G., and R. E. Hogan. 2005. The state of undergraduate training in political science. PS: Political Science and Politics 38(2): 293-297.
Wright, B. D. 2003. Assessment methods – A close-up look. www.apsanet.org/section_563.cfm.
© Copyright 2009 by Academic Leadership
Top of Page
|
|
|
|
