Each of the following will describe the method, its strengths and weaknesses, and its application to liberal arts education.
Grounded Theory
Case Studies
Longitudinal Case Studies
Focus Groups
Participant Observation
Qualitative Inquiry and Methods - Introduction
by Glenda Droogsma-Musoba, Indiana University
Amelia Noël-Elkins, Indiana University
Chris Rasmussen, University of Michigan
August 10, 2004
Although this section is designed to review the utility and efficacy of qualitative methods of student assessment, "qualitative inquiry" represents more of a philosophical movement and form of understanding about the nature of meaning than a research methodology. In spite of the fact that much of this section of text is dedicated to identifying ways to use methods associated with qualitative inquiry, the discussion must begin with a brief review of concepts and philosophy. For the sake of brevity, this discussion will not address many of the nuances and controversies surrounding qualitative inquiry within the research community (or for that matter the presence of diverse perspectives of traditional scientific methods). However, an understanding of the nature of knowledge and meaning, how this understanding is pursued within lines of qualitative inquiry, and the operation of methodological principles such as "validity" and "rigor" are important for both the appreciation and application of qualitative work.
The distinction between quantitative and qualitative inquiry is not simply that one involves the use of numeric data while the other relies upon text, or that the former is centered more in scientific principles and the latter based on intuitive judgment. Qualitative inquiry as a movement was a reaction against scientism and its dominance in research. Quantitative methods are based in the positivist tradition, the origins of which are found in enlightenment, empiricism, and the natural sciences. By contrast, qualitative inquiry draws from more postmodern philosophies such as hermeneutics, interpretivism, and constructivism, with broader use in the social sciences and humanities.
Scholars within each of the research traditions approach their work from a range of perspectives. While approaches to qualitative inquiry might be seen as existing on a continuum from empirical interpretivism to radical deconstructionist theory, some within the tradition would reject that dichotomization. A comparison of quantitative and qualitative traditions on beliefs about the nature of knowledge and its acquisition can provide a good entrée into the understanding of their differences and similarities.
How is knowledge acquired?
In the quantitative/empiricist tradition, knowledge is acquired through objective testing using the scientific method. The method was elevated over time to be the superior, if not the only, source of knowledge. Knowledge was thus defined by methodology. Knowledge became that which was empirically verifiable by an objective scientist. If something was not observable or testable, it could not be known in the scientific sense. Complex phenomena were reduced to their individual components, and concepts were operationalized in order to be tested. The knowledge components identified in this manner were then re-pieced together to gain a more detailed big picture than was available at the start, much like building a puzzle from smaller, high-resolution fragments.
In the qualitative tradition, the distinction between researcher and subject is blurred because pure objectivity is an illusion. The researcher and participant are inseparable and interactive. Interpreters are bound to their own traditions, and the question of whether the ‘outsider’ or the ‘insider’ is better at discerning understanding is a matter of debate. Qualitative inquiry, while it is evidence-based, requires more than empirical observation—it also involves the attempted discovery or revelation of shared meanings and collective understanding. Interviews and other forms of text and dialogue represent legitimate evidence because the researcher assumes that the individual’s perceptions and forms of "meaning-making" are important data for understanding human behavior. Acquiring knowledge begins with a broad, open-ended question rather than a testable hypothesis.
What constitutes evidence and valid research?
The nature of evidence and data represents another area of difference between the paradigms, with qualitative data being generally defined as empirical material, objects, or observations that are non-quantitative. Lincoln distinguishes between data and evidence by stating that data "is not evidence until two things happen: first, someone recognizes it as data, and second, an inquirer subjects it to some form of systematic analysis, which turns it into evidence directed toward some question or argument." [1, p. 6]
Scholars have proposed a number of criteria for what constitutes qualitative evidence. For one, evidence should be systematically gathered in a manner consistent with the research tradition, and should be observable. Unlike with quantitative traditions, in qualitative inquiry research participants are partners in the research effort and are given the opportunity to review, clarify, and elaborate on the evidence provided to the researcher. This "member checking" process is one of the ways that individuals who generate research data are "participants" and not "subjects." Further, qualitative data should be made available to the reader of resulting deliverables so that he or she can authenticate the researcher’s interpretations.
If one accepts the legitimacy of the qualitative inquiry paradigm, then the question of what constitutes valid evidence is less troubling and becomes a matter related to the methods employed for data collection and analysis. In qualitative research, data from multiple sources are compared for consistency in a process called triangulation. Data gathering, analysis procedures, and actual data (such as quotes from interviews) are reported in detail so the reader can verify that established practices were followed and follow the path of analysis from participant to interpretation and presentation.
How can qualitative inquiry be useful in a liberal arts context?
Switching research paradigms can make an unanswerable question answerable and reasonable to ask. Because the nature of the research method used should follow from the questions asked, qualitative approaches are a reasonable choice to answer queries about the outcomes of a liberal arts education. Many of these outcomes are difficult to measure in quantifiable ways, particularly the broad descriptions often found in college mission statements. Student qualities and skills such as the seven that are the focus of the Center of Inquiry in the Liberal Arts’ longitudinal study (integration of learning, inclination to inquire and pursue lifelong learning, effective reasoning and problem solving, moral character, intercultural effectiveness, leadership, and well-being) are not always easily quantifiable. For example, while basic writing skills can be quantified in the number of grammatical, spelling, and punctuation errors per words of text, the measurement of a student’s ability to produce original research or original thoughts, rather than simply summarizing the thoughts of others in a new way, is not as readily measured using existing assessment tools that yield numbers or statistics.
Wiggins suggests that liberal arts faculty have a "moral obligation to disturb students intellectually." [2, p. 547] In place of or in addition to exit examinations, the qualitative graduate interview is an example of one way to measure whether students were "intellectually disturbed" while at the institution. On selected campuses, structured interviews are used to assess the level of intellectual development for a sample of students at matriculation and again in a student’s final year. This developmental measure is combined with interview data about salient college experiences to determine relationships between campus programming and student development. Qualitative interview data can then be combined with quantitative analyses of student program participation and other institutional data to examine outcomes such as student persistence and post-graduate activity.
Qualitative inquiry can also be tailored to specific campus contexts and questions. A limitation of externally-provided instruments is the frequent lack of institutional control over the questions asked for the unique context of that particular campus. Qualitative inquiry, by definition, understands knowledge as connected to its context. While transferability of knowledge occurs from one context to the next, qualitative researchers assume each context is unique.
In sum, qualitative inquiry and associated methodologies represent a means by which colleges and universities can assess student learning and other desired outcomes in a rich, campus-specific manner different from that provided by traditional surveys and other quantitative approaches.
References
Qualitative Assessment Methods
Grounded Theory
Definition: Grounded theory uses interviews or other data to construct a new theory to explain a particular experience or event. The product of grounded theory is a premise built from recurring themes in the data. In contrast to the testing of theory in many research methods, grounded theory begins without a theory; rather, the theory emerges from the data.
How to: Interviews with multiple participants are analyzed for recurring themes. Coding categories or themes are discovered or emerge from the data in a post hoc manner rather than the data being analyzed according to a priori hypotheses.
Application to liberal arts: Grounded theory is a broad-based approach to developing a fuller understanding of any number of social phenomena or other human experiences. As such, a grounded theory research design can incorporate queries aimed at providing insight into student perceptions and experiences as they relate to various liberal arts outcomes.
What it does not measure: Grounded theory by definition is not useful to test the validity of a pre-existing theory or the application of a particular theory-based intervention for effectiveness.
Benefits: For those who prefer a quantitative approach, grounded theory is useful for working toward the development of a quantitative study. If the grounded theory is designed properly, an instrument can be developed to test that theory. In the long run, grounded theory is the qualitative approach that may best suit those who prefer quantitative data.
General challenges: Grounded theory can be time consuming. To appropriately construct a theory, extensive interviews should be conducted until "saturation" is reached. At this point, additional interviews contribute little new information or insight that can be used to illuminate or more fully understand the phenomena in question.
Integration with other research/assessment: Grounded theory is most useful in an understudied area where new conceptualizations are needed. After developing a grounded theory, a quantitative and/or qualitative test of the theory often follows.
Literature sources:
Glaser, B.G. & Strauss, A. L. (1967). The discovery of grounded theory: Strategies for Qualitative Research. Chicago: Aldine Publication Company. (This is the original text that articulated grounded theory. It presents grounded theory in its original conception, prior to the ideological split between Glaser and Strauss.)
Heath, H. & Cowley, S. (February 2004). Developing a grounded theory approach: A comparison of Glaser and Strauss. International Journal of Nursing Studies, 41 (2), 141-151. (Nursing research uses grounded theory extensively and produces some clear, concise articles about the use of this method. This article is an overview of the differences in the Glaser and Strauss approaches since their original inception of grounded theory.)
McCann, T.V. & Clark, E. (2003). Grounded theory in nursing research: Part 1–Methodology. Nurse Researcher, 11 (2), p. 7-19.
McCann, T.V. & Clark, E. (2003). Grounded theory in nursing research: Part 2–Critique. Nurse Researcher, 11 (2), p. 19-29.
McCann, T.V. & Clark, E. (2003). Grounded theory in nursing research: Part 3–Application. Nurse Researcher, 11 (2), p. 29-40.
(The three articles listed above provide good background on grounded theory in three concise parts: methodology, critique, and application. For someone who wants to use grounded theory for the first time, these articles provide a good brief introduction and understanding.)
Case Studies
Definition: A case study is essentially the telling of someone’s story in an evidence-based way. It can be the story of an individual, an organization, a department, an institution, or any other definable entity or unit. A case is a unit of analysis. The story identifies how that unit is typical and how it is unique, when compared to similar units. Often associated with sociology, case study involves in-depth analysis and emphasizes understanding of the case or unit in its real-life context.
How to: Data can be collected in multiple ways, including individual and group interviews, document analysis, review of artifacts, and participant observation. Multiple data sources can be used to remove the bias of relying upon only one perspective. The purpose of case study is not only to describe the given unit to the outside world, but also to provide new insights and illumination to unit members, which can be uniquely obtained through external observation and study. The researcher tries to identify both insiders’ and outsiders’ perspectives (both the autobiography and biography) of the unit. Data are analyzed for common themes; the written report provides both the author’s interpretation and the evidence upon which the author based his or her analysis so the reader can authenticate the interpretation. A cluster of case studies of individuals can be useful for generalizations if common themes emerge.
Application to liberal arts: Useful when exploring an unresearched or under-researched area. Because data collection may occur primarily through interviews, the interview protocol or agenda can be developed so the goals of a liberal arts education (moral character, inclination to inquire, effective reasoning, leadership, etc.) can be specifically explored.
What it does not measure: Case study is somewhat limited in its application between units because the analysis is unit specific.
Benefits: Case studies provide rich, detailed data. While they can be limited in breadth, this is generally balanced by their fullness of depth.
General challenges: Case studies are more labor intensive than evaluating existing data.
Integration with other research/assessment: Case studies can be used as a starting point for further research. They can be used to discover emergent themes that result from the case, which can serve as a foundation for subsequent research.
Literature sources:
Creswell, J. W. (1998). Qualitative inquiry and research design: Choosing among five traditions. Thousand Oaks, CA: Sage Publications.
Stake, R. (1995). The art of case study research. Thousand Oaks, CA: Sage.
Yin, R. (1994). Case study design and method. (2nd ed.). Beverly Hills, CA: Sage.
Longitudinal Case Studies
Definition: A longitudinal case study tells the ongoing story of an individual or unit (i.e., a group of people). It can use the same methodology and process of a case study, but examines a unit or cohort of students to observe changes over a longer period of time. While a standard case study represents a snapshot or a given unit’s story at a particular point in time, the goal of a longitudinal case study is to observe and document change.
How to: Typically the research follows a cohort of students over a period of time with interviews or observations conducted at regular intervals to document students’ experiences, perceptions, attitudes, and behaviors. Usually employing interviews, student work or other documents, or observation, the researcher follows the cohort of students over several years at the institution and even beyond.
Application to liberal arts: Documenting changes in student beliefs, attitudes, and behaviors can lead to a broader understanding of the process of developing higher order skills such as reflective judgment and intellectual inquiry. For example, a study could involve interviews with students near the beginning, middle, and end of their college careers; combine this data with a measure of moral development; and find whether those students who were involved in a cross-cultural experience showed higher levels of moral reasoning. In a similar manner, a campus could use structured interviews tailored to assess levels of intellectual or moral development to classify students on one of various developmental scales in their first and final years as one means of measuring growth during college. Additional interview data may reveal insight into the types of college experiences that contributed to this growth (versus maturation that would occur regardless). Such interviews or written surveys may also be given at designated periods after graduation to track desired liberal arts outcomes, such as a desire for lifelong learning, that are best assessed once the student has been away from the college for a period of years.
What it does not measure: If the research does not start out measuring a particular outcome or goal, it cannot subsequently attempt to measure that outcome in later administrations of the survey.
Benefits: Longitudinal case studies enable the researcher to observe progression toward goals or change over time. This is particularly helpful if used as a measure of whether a group or entity is effectively implementing the goals or changes they have set out for the organization.
General challenges: Any time one collects longitudinal data, there exists the challenge of tracking and retaining all students as willing participants in the study. Additionally, the instrument used initially cannot be changed significantly in subsequent interviews or else the data may not be comparable. Therefore, there is little ability to correct for mistakes if the initial instrument is not satisfactory.
Integration with other research/assessment: Because this is a labor intensive and time consuming approach, it can be most effective when used in concert with other approaches that provide more of a "snapshot" view of the unit (such as with a singular case study or focus group).
Focus Groups
Definition: Focus groups are semi-structured interviews with up to 10 participants. The most common application of focus groups is in marketing research.
How to: Focus group interviews can vary in their degree of structure. Most involve some number of predetermined open-ended questions. The group environment allows participants to respond to the comments of others by affirming, contradicting, or building on what has been said. Interviews are usually recorded, with transcripts later analyzed for common themes. The same skills are needed as with individual interviewing, with the additional challenge of managing the group dynamics. For example, dominant individuals can overwhelm other participants or stifle divergent perspectives. Questions should be open-ended, giving participants the opportunity to provide concrete examples and stories in a natural, organic manner.
Application to liberal arts: As with case studies, the focus group protocol or agenda can be developed so the goals of a liberal arts education can be specifically explored.
What it does not measure: Focus groups are generally not effective as a means of gathering individual-level or factual data (e.g., courses taken vs. perceptions, reflections and opinions, or asking one student questions while the other participants patiently wait). With the group structure, some respondents may be hesitant to share personal or sensitive information or be as self-reflective as in individual interviews (i.e., not as useful for autopsy exit interviews). Some subjects may be hesitant to share their true opinion if it is different from the rest of the group.
Benefits: Focus groups offer substantial data with minimal expense. Multiple respondents are interviewed in less time than would be required for individual interviews. The group dynamics, while challenging to manage, are also a strength as participants’ reflections are stimulated by other group members.
General challenges: The method is labor intensive in comparison to the analysis of existing data, particularly if conversation is transcribed and formally coded.
Integration with other research/assessment: Focus groups are a very useful method to pretest a survey or questionnaire or to generate initial data. They are also useful as a strategy to confirm or triangulate data gathered through individual interviews or other sources.
Definition: Participant observation is a method of data collection that involves watching and studying a particular culture or social group in order to describe, explain, and interpret the meaning of its actions. It is the method of choice for researchers following the ethnographic tradition and is most often identified with anthropology. Participant observation can be conducted by either an outside observer or as a participant who watches and investigates while functioning as a quasi-member of the group.
How to: Participant observation can vary from structured to unstructured. It involves the recording of observations or field notes describing participants in a particular situation, environment, or culture. Data analysis includes description, analysis, and interpretation of relationships and events, often focused on language usage, values, and traditions. The product is most often a descriptive narrative report. Whether the researcher/observer is an outsider looking in or an insider reflecting on personal experiences depends to a certain degree on the objectives of the research, the preferences of the researcher, and various factors related to the group itself and its environment.
Application to liberal arts: Participant observation and related ethnographic methods are particularly useful to measure campus culture, environmental factors, and student response to environmental factors. It can also be used to assess students’ cultural interactions and their ability to understand the perspectives of others, through observation of student or classroom discussions. One possible application uses trained outside observers to record students’ and teachers’ classroom behavior on a preset checklist every two minutes. This observation provides a measure of classroom-based learning activity. In a related application, a researcher could observe students in a library group work area and analyze the use of argumentation, critical thinking, and consideration of diverse perspectives in a collaborative learning context. The observation method can also be applied in a number of other out-of-classroom settings such as fraternity or sorority activities, student government, athletic events, public student presentations, etc.
What it does not measure: Participation observation usually does not emphasize quantification.
Benefits: Observation of naturally occurring behavior is ideal for developing an understanding of how individuals respond to, interact with, and make use of various institutional resources. For the individuals being observed, their opinions and perceptions are their reality. Understanding those perceptions can not only give participants cause for self-reflection, but can also provide the researcher with insight and perspective he or she might not otherwise gain. This information can then be used to improve teaching, program content and delivery, and student services.
General challenges: Participant observation is a time-consuming data collection method and depends on the researcher being in the right place at the right time, without the researcher’s presence altering the surrounding events. Researchers must also be aware of bias and how this influences decisions related to data recording, interpretation, and reporting.
Integration with other research/assessment: In general, this approach is made stronger if used in conjunction with other methods to enable comparison and contrast to participants’ views, and to provide both confirming and refuting evidence to support and challenge researcher interpretations and assertions.
Literature sources:
Schwartzman, H. B. (1993). Ethnography in organizations. Qualitative Research Methods Series, 27. Newbury Park, CA: Sage Publications.