Transcript Analysis
Definition: Analysis of course patterns reported on student academic transcripts. Rather than being new data collection or an assessment instrument, transcript analysis involves examining existing school/student records for patterns. Transcript data can also be matched with other quantitative student data and courses used as predictor variables to see a relationship with student outcomes.
How To: Transcript analysis can be accomplished either through examination of individual academic records or manipulation of computer information systems of all students to produce summary reports of course-taking patterns for different student groups. Student records can be analyzed quantitatively to show course-taking patterns and provide administrative data such as differences in faculty grading patterns or student exposure to full professors (verses teaching assistants or junior faculty).
Application to Liberal Arts: Transcript analysis enables assessment of student enrollment in courses deemed central to or supportive of the liberal arts mission of the institution. Quantitative analysis of student coursework across the departments or distribution requirements within each department can provide assessment of whether students experience the necessary breadth and integration of course content to produce a "well-rounded education." Usually these are measures of the processes of education rather than outcomes of education.
What it does not measure: Transcripts can not measure actual student learning or other student outcomes related to enrollment in a given course beyond the grade earned. Transcript analysis does not measure the extent of student participation or involvement in course activities.
Benefits: Transcript analysis allows institutions to examine patterns and trends in course enrollment over a set period of time, across cohorts, or across academic majors. Transcript analysis can be combined with other data to determine relationships between course-taking and student learning or other post-graduate outcomes. Patterns can also be measured for subgroups of the student body that may be of interest to the institutions, such as students whose parents did not go to college, students who were admitted conditionally because of weak academic preparation, under-represented minority groups, and students who studied abroad.
General challenges: Analysis of individual transcripts is potentially tedious and time consuming. The analyses would likely require the presence of flexible computer information systems that enable the disaggregation of academic data and various combinations of variables. In addition, access to individual transcripts must be handled appropriately because of FERPA restrictions. Course-taking patterns do not show causation for course selection. Certainly, some students take courses because of their reputation as easy or popular courses.
Integration with other research/assessment: Because of the quantitative nature of this data, if information collected through other instruments also contains student identification, the records can be matched. Other data may be student outcome measures gained through assessments or postgraduate surveys. The matched data can be used to measure the relationship between student outcomes and their academic experiences.
Tracking of Website Visits
Definition: Numerical counts of visits to an individual or multiple websites related to academic programs and services, along with possible analysis of the time of day of the visit and the location of the visitor.
How To: Various software programs are available that monitor visits to a website, providing data on the number of visits or "hits," the time of day the visits occur, and the address of the computers used to access the site. Depending on the program, this data is then able to be displayed in various formats. Tracking may be combined with a program that automatically sends an invitation to complete an evaluation of the website to random individuals at set frequencies, such as every 100th or every 500th visitor.
Application to Liberal Arts: Tracking websites may be used to count visits to various websites related to the liberal arts mission, including course-specific and course-related websites, and websites for different academic resources and support services. Combined with an evaluation component, tracking may suggest improvements for web-based services for students.
What it does not measure: As an input or process variable, website tracking would not measure student learning outcomes per se.
Benefits: Data related to frequency and origin of visits to web-based resources can assist those responsible for web development to make improvements to maximize the effectiveness of such sites. Websites may be upgraded, merged, or eliminated based on usage patterns. Data can be used to inform the development of surveys or focus groups designed to assess the relative utility of existing web resources as well as user needs and preferences.
General challenges: Tracking requires staff time and attention to integrate tracking mechanisms and manage resulting data. Given the nature of the tracking systems, visitors may not be specifically identified as students or faculty, or whether multiple visits came from the same user.
Integration with other research/assessment: This assessment tool focuses on the processes of education. Alone it is inadequate to measure student outcomes, but it would be useful to improve services to students and can be used with other data to form a complete picture of the educational experience.
Library and Other Facility Use Patterns
Definition: Tracking usage patterns of library resources, including physical facilities, staff, and circulation materials.
How To: Electronic monitors can log the entrance of library patrons. Library staff may conduct physical counts of library patrons at different times of the day and log the nature of their activities, such as individual or group studying, computer terminal use, or bound material browsing. Library staff can also maintain journals of the number and nature of requests for assistance from in-person visits, telephone calls, and electronic mail. Computer programs can be used to develop lists of the most frequently borrowed books or monitor the use of particular texts or groups of text by topic or discipline. Generally these examples are the quantification of services to students or institutional practices. More qualitative observational data of library usage or group study would provide greater depth of information on the interactions between students and the way materials are used.
Application to Liberal Arts: With investigation being a key element of a liberal arts education, the use of library resources merits analysis as a factor related to this skill set. A close survey of what students are doing in the time they are in the facility can help determine whether the facility is contributing to the aims of a liberal arts education. For example, students using the facility as a place to work collaboratively with each other may illustrate the exchange and synthesis of ideas, thereby contributing to liberal arts objectives.
What it does not measure: Tracking systems do not necessarily provide data on the actual activities of patrons once in the library. Borrowing of bound materials does not provide a measurement of their actual use or study.
Benefits: Tracking usage patterns enables institutions to make more informed decisions regarding opening and closing hours, as well as staffing patterns. Observation of student use and work patterns (including individual/private and group activity) may contribute to a fuller understanding of student culture and learning styles, which might inform pedagogical choices. Tracking of circulation patterns may provide insight into the reading and research habits and patterns of students and faculty, while also helping to inform future materials purchasing and decisions related to stacking, display, and movement to off-site facilities. In addition, the library culture may be altered if the environment is not conducive to the accomplishment of the institution’s goals.
General challenges: The most significant challenge is the set-up and maintenance cost of electronic monitor systems and software. Concerns related to privacy, particularly with analysis of circulation patterns, need to be addressed. Circulation data does not provide information on use of non-circulatory materials, including journals, other periodicals, newspapers, and microfilm/microfiche.
Integration with other research/assessment: Usage patterns measure processes or educational environment and are inadequate to draw conclusions about overall effects. Used in combination with other measures of outcomes and other processes, the data can offer useful information.
Advising Activity Records
Definition: Data on the frequency and nature of student meetings with faculty, academic advisors, or other communication related to academic advising.
How To: Data on advising activity can be logged by faculty, possibly through use of a specially designed computer program that collects information related to the occurrence, duration, type, and nature of advising meetings and correspondence. Some of this data might be quantified, but structured observations of the interactions can provide richer data regarding the influence of these interactions on students. Faculty can use the interaction to obtain data or information from students about their college experience, aspirations, and perceptions of skills and character development. This data can be useful for institutional research but also enriches the advising experience.
Application to Liberal Arts: The academic advising process can be constructed in a way that promotes the development of higher order skill, through the formal recommendation of specific courses or clusters of courses and the engagement of students in conversation related to specific themes and ideas.
What it does not measure: Analysis of advising activity does not necessarily measure student learning or other outcomes, depending on how the advising contact is structured. Data collected about student perceptions of their college experience is anecdotal and not a behavioral outcome.
Benefits: Research into this area can provide quantitative and qualitative data that could be shared with various constituencies interested in the activities of faculty and the nature and scope of faculty-student advising interactions. Data can be analyzed for trends and patterns across cohorts, academic majors and disciplines, as well as over time. Data can also be used to improve the process of advising. Faculty data can be compared to student reports of satisfaction with academic advising to gain insight into the types of faculty activities that are most highly valued by students. Advising data can also be compared to student grades, course completions, and graduation rates to determine the possible relationship between academic advising and various student outcomes. Advising discussions can also provide early warnings of student dissatisfaction or potential departure and can be used to enhance the educational environment.
General challenges: Faculty may resist the time and effort required for analsysis. Faculty may also question the utility of the data and the possible implications or outcomes. Some of the information in advising sessions may be considered private and, therefore, protected by FERPA. Data collected by so many individuals lacks continuity unless a structured protocol is provided to faculty. However, a more structured protocol may hinder the natural conversation of advising. A more qualitative approach may result in an overwhelming amount of data.
Integration with other research/assessment: Faculty or advisors can provide useful data about the educational experience with little additional effort. If designed for qualitative data collection, advising data can provide rich information about the process of education along with some student perceptions of outcomes. Used alone, it provides an incomplete picture of the educational experience and minimal behavioral outcome measures.