Institutional Self-studies
Definition: Internal assessment of university programs, policies, and activities generally conducted in conjunction with an external review or evaluation. Campuses engage in self evaluation in preparation for scheduled visits from institutional, departmental, or discipline-based national accreditation or governing agencies (e.g., North Central Regional Accrediting Association, National Collegiate Athletic Association, American Chemical Society, Association of Schools of Music, etc.). National accrediting agencies, external evaluators, function much like auditors to businesses and they evaluate whether the institution is accomplishing the goals it has identified for itself and its students.
How To: Generally, the institutional self-study is a lengthy, involved process lasting two years or more. The process is initiated by the institutional president, chief academic officer or head of institutional research efforts. An institutional charge and direction is issued based on guidelines established or suggested by the accrediting agency. Self-studies involve the appointment of a coordinator or co-coordinators, the establishment of a planning team with multiple areas of research focus, and the creation of various sub-committees and levels of organization. Data used for the self-study can include reports of various surveys and assessments of student outcomes, program effectiveness, and student satisfaction; summaries and evaluations of various academic and co-curricular program activities; quantitative/numeric data related to programs and service, such as number of program participants, and number of client-hours, number of service recipients or users, etc.
Application to Liberal Arts: Self study involves introspection and public dialogue about the core mission and objectives of the institution, and an honest self-appraisal of the extent to which institutional activities are reflective of that mission. Thus the process can assist institutions to orient themselves more intently toward a professed liberal arts focus, if applicable. When institutional mission statements profess goals of liberal arts outcomes such as the ability to think critically or make judgments, then assessing whether the institution is meeting those goals should be in the self-study report. Therefore, data collected in preparation for external evaluations can also be used for internal improvements or adjustments to campus practices.
What it does not measure: Student outcomes measured are often rudimentary. Self-studies are not assessments of higher order student learning or program effectiveness per se, but rather represent something akin to meta-analyses or attempts to summarize inputs, processes, and outputs related to core university functions. They are more an audit of the systems and processes, leading to recommendations for improvement.
Benefits: Self-studies are a comprehensive audit of university functions and the degree to which various academic, administrative, and support units contribute to the core functions of the institution. Self-studies can involve participation of a wide variety of individuals from the institution, resulting in diverse perspectives. Secondarily, institutions benefit from self-studies through the creation and strengthening of relationships across campus.
General challenges: Self-studies are generally done infrequently and usually only in conjunction with accreditation visits. Self-studies are extremely time consuming and labor intensive. The scope of the task can require additional staff or release time for existing staff. Because the bulk of the data collection is done by employees of the institution, the process can lack an outsider’s perspective, with resulting bias limiting the scope and honesty of the evaluation. Because self-study reports are prepared for the external evaluator rather than internally motivated, too often they do not result in substantial changes on the campus. Documenting student outcomes for the self-study when data collection has not been ongoing often limits the meaningfulness or depth of the results.
Integration with other research/assessment: Because this is generally an audit of systems, it can highlight the need for additional information in some areas thereby generating smaller research. It forces reflection and introspection about how and why something is done, causing changes in other evaluative processes such as classroom grading practices.
Strategic Plans
Definition: A strategic plan is a document that outlines, in general or specific terms, the goals, objectives, and intended direction of the institution, and which can serve as a blueprint for human and fiscal resource allocation and program planning. When institutions grow, strategic planning can develop a plan of action if resources exist to add positions, add programs, or in other ways make the changes identified. Strategic planning can establish priorities by which new programs or personnel will be added based on the goals and objectives of the organization. When institutional changes or reductions are needed because of financial limitations or ideological shifts, strategic planning can assist in identifying and protecting the necessary personnel and programs.
How To: Similar to an institutional self-study, the development of a strategic plan generally begins with a charge or direction issued by the university president, governing board, or departmental director. This is followed by the appointment of a strategic planning team that consults with various community members (both internal and external) about the perceived and desired mission, function, and activities of the institution or department. After identifying strengths and weaknesses of the institution and its goals, the committee produces a strategic plan that identifies and prioritizes the future activities of the institution. The plan is then given to the campus leadership for approval and implementation.
Application to Liberal Arts: As with institutional self-studies, the development of a strategic plan ideally involves introspection and public dialogue about the core mission and objectives of the institution or department. The process also involves an assessment of the relative desirability of continuity and change in institutional activities, possibly to reflect evolving priorities related to shifts in enrollment demographics, market position, fiscal variables, or other outside influences. The strategic planning process represents an opportunity for institutions or departments to publicly recommit themselves and bring a new energy and zeal to their liberal arts mission.
What it does not measure: Strategic planning is not a method for measuring student achievement or other student outcomes.
Benefits: The development of a strategic plan ideally involves an examination of existing institutional or departmental data on student learning and program effectiveness, and a comprehensive assessment of strengths and weaknesses in effecting desired liberal arts outcomes. This information can be used as a foundation for outlining the future activities of the institution or department and the appropriation of resources in a manner consistent with stated objectives.
General challenges: Strategic planning can be time consuming and controversial. It may result in difficult conversations and debates between individuals and groups around issues of change and innovation, culture and tradition, resource allocation, stability, and uncertainty. In some instances it can also compete with the ongoing administrative planning of the institution in identifying priorities.
Integration with other research/assessment: If done properly and conveyed throughout the community, a strategic plan can guide the development of programming and curriculum so the goals of the institution or department are achieved. A strategic plan should identify outcomes the institution will want to measure in its assessment process.
Course Evaluations
Definition: Course evaluations are assessments of student experience and satisfaction with an individual course and are generally done at or near the end of the academic term. They provide the professor, department, and institution with student perceptions of the classroom aspect of their educational experience.
How To: Course evaluations can include any number of questions designed to assess student learning, general student satisfaction, perceived effectiveness of the instructor, fairness of performance measures such as written assignments and examinations, classroom climate, and relationships with peers. Although the most common method of course evaluation is collection of a paper and pencil survey at the end of the term, evaluations can also be conducted mid-term or more frequently to provide the instructor with feedback that can be used to make changes in teaching. Evaluations may include multiple choice questions, statements with quantitative responses, and qualitative items such as open-ended questions. Multiple-choice or quantitative items are often written with a Likert-type scale (students rate their agreement with the question or statement on a five point scale) which enables a full range of statistical analyses within and across courses. Qualitative items generally take the form of open-ended questions or unfinished sentences.
Application to Liberal Arts: Course evaluations can be designed to include questions that measure qualities associated with the liberal arts, such as higher order thinking skills, as well as the presence of variables associated with student learning. Evaluations explore such issues as classroom climate, richness and depth of classroom discussion, peer collaboration and dialogue of course matters outside of class, and applicability of course material and discussions to real-life circumstances. If properly developed and administered, evaluations not only serve the instructor in addressing liberal arts education goals, but also may serve the department in evaluating faculty effectiveness in reaching liberal arts goals.
What it does not measure: Student course evaluations have been criticized by some as referenda more on instructor popularity than course content and learning opportunities. Personality and other instructor variables can influence student perception of classroom climate, fairness of evaluation, and instructor effectiveness. Course evaluations are student self-assessments, and thus have limited reliability as measures of student learning compared to other instruments and methodological approaches. Therefore, assessing liberal arts outcomes may not reliable.
Benefits: Course evaluations provide individual instructors and institutional leadership, with valuable data on the degree to which students are pleased with their course experience, student perceptions of the effectiveness of instructors, and the extent to which the course promoted learning and growth. Depending on how instruments are designed, course evaluations can enable comparisons across courses for the same or for different instructors, from year to year, from course to course within an academic major or department, and across majors and disciplines. Because course evaluations can be made public, they can be a visible affirmation of the achievement of stated instructional goals by an institution or department.
General challenges: Obstacles include resistance of some faculty based on legitimate concerns related to the quality and effectiveness of the survey, the reliability of student self-reports, and the extent to which course evaluations are used in performance reviews for promotion and tenure. Institutions must also develop clear and public statements of intent for the use of course evaluations, including conditions of confidentiality and anonymity of instructors.
Integration with other research/assessment: If done prior to the end of the semester, the professor can use evaluation results to adjust graded classroom assignments and classroom practices.
Post-graduate Surveys and Assessments
Definition: An assessment of a student’s overall satisfaction with his or her collegiate experience.
How To: Although generally conducted shortly after completion of the undergraduate degree, post-graduate assessment can occur at any time in the future, including at regularly scheduled intervals to evaluate the perceived outcomes and value of the college experience at various points in life. Assessment can be done via a written survey involving a combination of multiple-choice and open-ended questions, or through interpersonal data collection methods such as telephone, face-to-face, or group interviews. Databases can enable analysis of data over the years for the same students, comparisons across graduating classes, academic programs, or other variables.
Application to Liberal Arts: Post-graduate surveys represent an excellent opportunity to gain insight into how graduates anticipate using a liberal arts education in their lives after college. In interviews, graduates can be asked to share examples or tell stories that illustrate the application of higher order skills in everyday circumstances or to respond to life’s challenges. In addition, longitudinal forms of post-graduate surveys can shed light on the long-term value of a liberal arts education. Student success after graduation, such as completion of a graduate degree, civic involvement, cross-cultural experiences, or other desired behaviors can be collected as part of the interviews or surveys.
What it does not measure: Since post-graduate surveys represent self-reports of perceived outcomes, their validity as true measures of learning are limited. Ability to use the liberal arts curriculum as something akin to an independent variable is limited in the absence of a control or comparison group.
Benefits: Surveys or interviews provide real data on graduate satisfaction with the college experience and the extent to which alumni feel the institution prepared them for professional and personal concerns. Surveys can be structured in such a way as to intentionally measure specific liberal arts outcomes with observed behaviors. Relationships can be established between outcomes and predictors such as academic major, course-taking, co-curricular experiences, and pre-college factors. Surveys can combine objective data on student employment, advanced education, salary, and community involvement to create a more complete picture of graduate outcomes.
General challenges: Data collection requires a programmatic and financial commitment on the part of the institution, particularly if data is collected over multiple years. After graduation, addresses may become outdated and students’ commitment to the institution may fade so the number of students with complete data dwindles. Multiple attempts to encourage graduate response may be needed. Surveys require a system for data storage, as well as staff resources if an interview-based approach is used.
Integration with other research/assessment: Surveys can be used in the context of or in conjunction with an entrance and exit interview of students, allowing comparison of student expectations and abilities prior to college with the same items immediately following and several years after college. Surveys can also be used with other data collected over a number of years so that long-term effects of the educational environment and liberal arts goals can be assessed.
Analysis of Course Syllabi
Definition: Analysis of an instructor’s formal academic course plan, or syllabus. The syllabus generally includes an outline of course topics, materials, and other resources to be used. It may also include a timeline, faculty expectations, and evaluation methods. Course syllabi are usually distributed the first day or week of class and copies are usually kept on file in a departmental or school/college office.
How To: Syllabi may be analyzed using an a priori framework for categorizing content, including items such as number of texts used, total pages of required reading, number and type of examinations, written assignments and other evaluation activities, class presentations, group work, etc. A database can be created to organize syllabi by course, discipline, faculty member, year, or other variables.
Application to Liberal Arts: The idea of academic rigor in critical reading and writing, analytical thinking, and critical analysis is inherent to the Liberal Arts. Some classroom assignments are more likely to hone these skills than others and therefore syllabi should include coursework that focuses on the growth of these abilities.
What it does not measure: Analyzing course syllabi does not measure the extent to which course content and activities conform to the written plan, the extent to which students complete assignments, student satisfaction with outcomes from the course, and growth in student learning. It measures the processes of education, not the outcomes of those processes.
Benefits: Analysis provides insight into the type, amount, and frequency of academic activities assigned such as reading, writing, examinations, and presentations. Resulting data can be analyzed across courses, disciplines, and cohorts. Analysis of data in comparison to various student outcomes can provide information as to the effectiveness of different pedagogical approaches and academic activities.
General challenges: This type of analysis is time consuming. Faculty need to be convinced of the value of the resulting data and informed if the analyses will be used for their performance evaluations. Making syllabi public may lead to intellectual property issues if the information is freely exchanged without the permission of the instructor.
Integration with other research/assessment: Analysis of syllabi is a measure of the activities or processes of education, so it needs to be used with other tools that measure student outcomes in order to fully assess the effectiveness of classroom practices.
Next Section:
Institutional Records