Introduction
The Faculty Survey of Student Engagement (FSSE) is a relatively new tool for colleges and universities to use to better understand how students engage in their undergraduate experiences. A companion to the National Survey of Student Engagement (NSSE), many of the items on the FSSE ask about issues believed to be necessary for a strong liberal arts education. It is intended to be used in conjunction with the NSSE, which is completed by students. The FSSE, which collects faculty perceptions of their students’ behavior, provides a counterpoint to the NSSE, suggesting areas of agreement between students and faculty, as well as topics that are seen quite differently by the two populations. It has drawbacks for use in assessing liberal arts education, however, in that it is necessarily broad in its approach. Another drawback, more important for a particular group of institutions, is that the instrument assumes face-to-face delivery of instruction. The Internet as an option for student-student and student-faculty interaction is not addressed. All in all, though, the FSSE instrument provides a solid, broad set of data about higher order thinking and student involvement that should give an institution a good starting place for discussions on improving the liberal arts experience for students. FSSE should be used as a springboard for these discussions; as a starting point, not an end point.
History and Background
What is the development of FSSE?
The FSSE project is coordinated at Indiana University by the National Survey of Student Engagement (NSSE). According to the FSSE invitation, 2004, "it is designed to measure faculty expectations for student engagement in educational practices that are known to be empirically linked with high levels of learning and development." Pilot tested in the spring of 2003, by more than 16,000 faculty members (at 147 institutions), FSSE intends to provide a springboard for discussions about teaching, learning, and the educational experiences of students.
NSSE surveys undergraduate students each year on their level of engagement in different educational practices, based on the belief that these practices represent "collegiate quality" (5). FSSE intends to build on the knowledge an institution gains from NSSE by surveying faculty on their perceptions of NSSE topics to better understand similarities and differences between students and faculty. FSSE is intended to be used in conjunction with NSSE, not as a stand-alone instrument.
Overview of Topics on FSSE
The 25 questions, some of which have up to 10 sub-questions, ask faculty about "one particular undergraduate course section" she or he is teaching (3). The first section asks for the course level (upper or lower division), number of students, number of times previously taught, and general discipline. Other topics include: frequency of student behaviors; amount of assigned work; speculation on how much time students spend on homework; the importance (to the faculty member) of student behaviors; frequency of class activities; faculty’s evaluation practices; faculty’s goals for the course; use of faculty’s work time; beliefs about student activities and their relationships with institutional groups; institutional emphases; and personal characteristics of the faculty member. Sample questions include the following:
About what percentage of students in your selected course section ... talk about career plans with you?
How often do students in your selected course section ... have a serious conversation in your course with students of a different race or ethnicity than their own?
In your selected course section, how important to you is it that your students ... tutor or teach each other (paid or unpaid)?
In your selected course section, on average, what percent of class time is spent on ... lecture?
In your selected course section, how much emphasis do you place on engaging students in each of these cognitive activities: memorizing, analyzing, synthesizing, making judgments, or applying theories [definitions are provided]?
How important is it to you that undergraduates at your institution do ... community service or volunteer work?
To what extent does your institution emphasize ... providing students the support they need to thrive socially?
Audience
Who might find the FSSE worthwhile?
Any institution that wants to gauge its academic culture in terms of broad liberal arts goals like higher order thinking and student involvement could benefit from this instrument. Its usefulness, as noted in the FSSE information, is primarily as an encouragement for internal conversations about where an institution currently stands on broad liberal arts goals, where it wants to go, and how it might get there.
How widely is it distributed?
Institutions that used the NSSE during 2001–2004 were "invited to participate" (FSSE Invitation, 2004). Presumably this is intended to ensure a linkage to NSSE, and to avoid FSSE operating as a stand-alone survey in the pilot phase. The instrument was pilot tested in 2003 and there is no discussion of plans to expand its distribution (2).
Elements of Liberal Arts Education
To develop a framework for the evaluation of FSSE, this section discusses some goals of liberal arts education. These goals are then used in the following sections to critique the usefulness of FSSE for institutions who wish to strengthen their students’ liberal arts experiences.
Center of Inquiry for the Liberal Arts-Definitions
"An elite group of education researchers" convened in August, 2002, and hypothesized that three factors or conditions "must co-exist to support liberal arts education": (a) an emphasis on developing a set of intellectual arts over professional or vocational skills; (b) structures in the curriculum that create "congruence and integrity" in students’ experiences; and (c) a strong institutional value placed on student interactions (i.e., student-student and student-faculty, both in and out of the classroom) (8).
Other Voices
Liberal arts education should involve students with faculty in communities of learning, and the curriculum should be "diverse, integrative, experiential, critical, and pluralistic" (Gamson and Assoc., 1984, as cited in Stark & Lattuca, 1997, p. 99). The Association of American Colleges (1991) issued a challenge to a number of fields, i.e., economics and mathematics, to re-structure the major in their field to meet four objectives: "coherence, connectedness, critical perspective, and inclusiveness" (as cited in Stark & Lattuca, 1997, p. 159).
Key Characteristics
What theoretical grounding does the instrument have?
No information was found through ERIC and other online journal databases about the development of the FSSE instrument.
How does FSSE relate to Astin’s I-E-O model?
Students are expected to learn more if they are more involved in the academic and social aspects of college, taken here as the college environment. The FSSE instrument does not cover topics on what the student brings to higher education, such as high school grade point average or SAT scores. This is appropriate, since college faculty cannot know with certainty about their students’ previous experiences. For the most part FSSE investigates the current classroom environment, largely concentrating on student behaviors thought to lead to strong liberal arts educational experience. A few topics on educational outcomes for students are explored, such as faculty’s evaluation practices, and faculty’s goals for their courses.
FSSE generally covers topics central to liberal arts education as defined here. Questions are asked regarding how frequently and by what means students have contact with faculty about their courses; the nature of the involvement of student assignments, including community-based class projects; and out-of-class work assignments supporting liberal arts goals. Deeper topics, though, are not covered in the survey; rather, it appears that FSSE is intended for a broader scope.
In what ways does FSSE collect data on students’ higher order thinking?
One section evaluates the emphasis faculty place on analyzing, making judgments, applying theory to practical problems, and synthesizing ideas. Another section explores the extent to which faculty structure their courses to achieve such goals as thinking critically, understanding themselves as well as faculty and students of other ethnic backgrounds, and serving their community through volunteer work. In both sections, decidedly non-liberal arts questions are asked, such as how much memorization and lecturing take place in the class. These serve as counterpoints to the liberal arts questions—an effective survey design. All of these explorations are fairly broad in scope; no one thread is followed deeply in the FSSE instrument. The FSSE provides a solid base for a broad institutional picture of the value the academic culture places on higher order thinking.
What flaws or gaps does the instrument contain?
A significant gap implicit in the instrument is the assumption that organized learning takes place only in the classroom. Although it probes the use of e-mail and other methods for out-of-class communication with the faculty, FSSE has limitations with an all-Internet-delivered course. An item such as "frequently comes to class without completing readings or assignments" loses meaning in an Internet course context. Other items could be expanded or explained to fit either in-class or Internet formats, i.e., "Receive prompt feedback (written or oral, or by e-mail) from you on their academic performance". It will be more difficult for institutions that rely more heavily on the Internet for instruction, e.g., the University of Phoenix, to make effective, institution-wide use of this instrument.
Does it claim to measure something that it does not measure?
The claims made for the instrument on the FSSE web site appear congruent with the items included in the survey.
What group piloted the survey?
Pilot test analyses are not provided on the FSSE web site. However, a list of institutions that participated in the pilot test project is provided (1) with no additional information, e.g., Carnegie classification, given about the individual schools. While larger schools such as the University of Georgia, Colorado State University, and the University of Michigan are listed, it is not clear how extensively their faculty were included. Along with these larger schools are such diverse institutions as Wheaton College, Teikyo Post University, the New College of Florida, and Circleville Bible College.
Using the survey over time
As data are collected once each spring, the same course-section could be included year after year to yield information on how faculty change the course over time. In this way FSSE could be used to compare results from over time. However, because the faculty has previously completed the survey, this may have an unwanted influence on subsequent surveys, due to their familiarity with the instrument. It is likely that FSSE is intended as a point-in-time survey, using a different set of faculty each year to assemble, over time, a valid picture of the institution.
Sampling is at the discretion of the institution and cannot be assumed to be random, if for no other reason than the faculty pool at all but the largest institutions limits sampling options. A purposeful sampling approach may offer benefits; each institution should consider this carefully. In this particular course, sections are selected for specific reasons, such as the course is one that is targeted for improvement. The test design is not a pre-test, post-test; faculty complete it once in the spring.
Applications & Criticisms
Why should institutions use FSSE?
FSSE is useful for helping professors understand and enrich their students’ educational experiences (2). Institutions that use the NSSE instrument should find this an additional way to look at student engagement to better understand student experiences. FSSE focuses on higher order thinking behaviors more so than on other goals of liberal arts education, such as student-student interaction. Overall it does not probe the complexities of the goals of a liberal arts education.
What changes might improve FSSE?
One drawback of the FSSE is that the selection of items are more suited for in-class instructional methods. The growth of distance-based learning (such as video-conferencing and internet-based courses) is not addressed in this instrument. Institutions that deliver significant amounts of instruction over the Internet may have to work closely with faculty to make sure that the questions are commonly understood, or may decide that the survey is not suitable for them.
Faculty report their perceptions of a group of students "as a whole" on FSSE. This leads to a fundamental caveat in using FSSE: Not only is the data the faculty’s perceptions (not grounded in specific data or examples), but by asking for students’ behavior, faculty are likely to unconsciously focus on a particular type of behavior, for instance, the most flamboyant student. Even if the faculty think of the common students in answering the survey, FSSE still will not collect data on uncommon students. The richness of diversity of students’ behavior is lost through this. One set of questions handles this better than the typical "very often" to "never" type of questions. This set asks for percentages of students who engage in a behavior. It is more difficult to analyze this type of ipsative data (forced choice) beyond reporting frequencies, however, because the assumption of independence is violated.
In the current arrangement of the FSSE, if 25% of students are reported exhibiting a behavior, the remaining 75% is unclear. For example, if 25% of students are said to "frequently ask questions in class" it is not clear if the other 75% never ask questions or sometimes ask questions, a potentially big distinction. The resulting ipsative data would violate the assumption of independence required for many commonly used statistical analyses. See Cameron & Ettington (6) for an example using ipsative data and Cameron & Quinn (7) for a more in-depth discussion of problems using ipsative data in analysis. While it would make the survey longer, converting more questions to this format would improve it.
Finally, while this is less complicated from a survey administration standpoint, surveys completed on the Internet can be easier for respondents to ignore. This can present low percentages of response, and resulting analysis problems. Vigorous support by the faculty’s administration to encourage participation should be included in project planning.
How can institutions use FSSE?
FSSE materials suggest ways to use this instrument (2), such as internal improvement, faculty development, assessment and improvement, and curricular reform at the individual or program levels. Broader uses include accreditation self-studies and system-wide comparisons.
Examples of FSSE use
Western Michigan University reports planning to use the FSSE in conjunction with their use of NSSE in the future to "identify specific gaps between faculty and student data" (9) that will facilitate future discussions. The University of Wisconsin—Whitewater participated in the pilot test administering both the NSSE and FSSE. Their report (4) discussed the results in terms of diversity, expectations, and mental activities. Diversity considered students’ increased understanding of diversity and involvement with others from different backgrounds. An expectation gap exists between what students expected at UWW and what faculty expected. Knowing this allowed UWW to plan to take action. Finally, use of higher order thinking was reported as less than UWW’s peer institutions, which was another call to action. Several institutions report the output of their FSSE data on their web sites in data table formats, with little explanation of how they plan to use the information.
The "Nuts & Bolts"
What is the cost of using the instrument?
Cost is based on four levels of students enrollment. For institutions with fewer than 250 students, the cost is $1,000. Stepping up three levels, the cost is $1,750 for institutions with more than 750 students.
How is data collected?
The FSSE is relatively easy to administer. Faculty are selected by their institutions and contacted by administrators to set the expectation of participation as they receive their invitations. From these lists, NSSE sends electronic information to the faculty, who complete the survey on the Internet.
When is data collected?
Once a year, around March, with final reports on the results of the survey sent to the institution in August.
What products/output do institutions receive?
FSSE institutions receive customized reports of faculty responses, comparisons of faculty responses to student responses, comparisons to national norms, and a data file of responses (see options within "FSSE Sample Reports" in FSSE Invitation, 2004). These reports list number and percentage of responses by survey item. The data file could be used by the institution to conduct deeper analyses.
Are individual students identified so their data can be linked to other surveys?
Individual students cannot be identified because the survey asks about groups of students as a whole.
Can the instrument be tailored with additional questions from the institution?
There is no indication that this option is available.
What should smaller institutions be aware of in considering the FSSE?
Sample selection should be carefully considered. The FSSE web site offers little on this topic; perhaps FSSE provides more guidance with institutions that use the instrument. Absent that advice, suggesting a purposeful sample to achieve targeted improvement or strategic objectives would not be out of order. Selecting courses that intend to encourage student involvement, promote higher order thinking, and other liberal arts goals would provide a reality check and a good starting point for conversation. Also, a cross-discipline, multi-level (freshman through senior years) sample would provide a more thorough institutional picture. This sampling approach might be most appropriate for institutions embarking on multi-year strategic planning as one way to assess "Where we are now and where do we want to go?"
References
