by Pat Helland, Vanderbilt University
and Jill Cellars Rogers, Center of Inquiry in the Liberal Arts at Wabash College
Summer 2005
OVERVIEW Highlights of the CAAP:
Uses of the CAAP:
Introduction
The Collegiate Assessment of Academic Proficiency (CAAP), developed by ACT in 1990, is a series of paper-and-pencil test modules that assess general education outcomes and measure achievement levels in the areas of reading, writing skills, essay writing, mathematics, science, and critical thinking. Each module has up to 72 questions and takes approximately 40 minutes to complete.
ACT’s CAAP website provides information about using the CAAP, the individual test modules, the reports that are available, ordering information, reference materials, and contact information.
The following report provides a summary of how the CAAP assesses outcomes of a liberal arts education, how the CAAP is administered, what the CAAP measures, how the CAAP is used, and limitations of the CAAP.
Liberal Arts Outcomes
Wolniak, Seifert, and Blaich enumerate curricular and co-curricular practices that, when implemented effectively and coherently, lead to positive gains in CAAP outcomes. [6] These practices include instructional clarity, challenging coursework, academic effort, essay writing in coursework, and integration of ideas, among others. Resulting learning gains are seen in areas of reading comprehension, science reasoning, and writing skills. Moreover, students who consistently experience high levels of these "best practices" not only show gains in the areas listed above, they also demonstrate growth in broader liberal arts outcomes like openness to diversity, learning for self-understanding, preference for deep and difficult intellectual work, positive attitude toward literacy, and a sense of responsibility for one’s own academic success. [6]
The individual CAAP modules aim to test the ability to connect and communicate major ideas, and rhetorical, reasoning, and organizational skills. The CAAP does not measure formula recall/memorization or mastery of a particular skill set. Additionally, the modules attempt to draw from many arts and sciences disciplines. For example, the reading test includes passages from fiction, humanities, social sciences, and natural sciences. The science test draws content from biology, chemistry, physics, and the physical sciences. The critical thinking test includes case study, debate, dialogue, overlapping positions, statistical argument, experimental results, or editorials. [1] Theoretically, then, the CAAP addresses (1) curricular content, (2) reasoning and thinking skills (not memorization), and (3) integration of course material and concepts.
Participation
Those interested in using the CAAP must complete a Participation Agreement form and pay an annual participation fee of $330 each academic year. This fee includes a Standard Reporting Package, which is intended to provide documentation of students’ achievement levels on an individual and group basis. ACT provides a CAAP Planning and Forms Manual to help institutions determine if the test is appropriate for them, and to guide them through the administration of the instrument.
Costs for scoring the reading, writing skills, math, science, and critical thinking modules vary and are based on the number of students participating, as well as on the number of objective modules taken per student. As an example, if an institution gave 250 students one objective module, the cost to the institution would be $2887.50. If they opted for 2–5 modules, the cost would be $4462.50. The only subjective module, the Writing Essay, has its own fee schedule ranging from $3.45 to $11.55 per respondent based on whether ACT or the institution does the scoring, and if other objective modules are purchased.
Reporting
ACT offers several reporting packages. As mentioned above, the standard package is always provided. It includes (1) the Institutional Summary Report, (2) student roster reports, (3) student score reports, (4) certificates of achievement, (5) up to three previously specified subgroup reports (supplemental reports are available for an additional fee). The Institutional Summary Report gives average scores for each demographic area and a summary of student self-reported motivation.
CAAP also offers reports that enable institutions to document student improvement and provide analysis of the relative strengths and weaknesses of student groups. Faculty can use this information to determine specific areas of their general education programs that are working or need enhancement.
About the CAAP Modules
Each CAAP module has been analyzed and found to demonstrate acceptable internal consistency and reliability. [2] Below is a brief description of each module.
Reading
The reading module consists of 36 questions designed to conceptualize referring and reasoning skills. Under the reading test, four prose passages from the content areas of prose fiction, humanities, social studies, and natural sciences are used to represent the levels of writing encountered in typical college curricula. Each passage then has nine multiple-choice questions. An example question under the prose fiction passage asks, "Which of the following statements represents a justifiable interpretation of the meaning of the story?"
Writing Skills
The writing skills module does not measure rote knowledge of spelling and vocabulary. Rather, it contains 72 items that measure students’ understanding of the "conventions of standard written English in punctuation, grammar, sentence structure, strategy, organization, and style." This test has six prose passages and each is followed by a set of 12 multiple-choice test items. Sentences or phrases are broken out from the passages and students are asked to correct the sentence if necessary. An example item with responses is, "In the end, everyone gives up jogging. Some find that their strenuous efforts to earn a living drains away their energy. A. NO CHANGE, B. drain, C. has drained, D. is draining."
Writing Essay
For the writing essay, two 20-minute writing tasks are defined by a short prompt that identifies a specific hypothetical situation and audience. The student is then instructed to take a position on the issue and to explain to the audience why the position taken is the better alternative. This approach is designed to test the student’s skills in formulating an assertion, testing that assertion, organizing and connecting ideas, and clearly expressing those ideas.
Mathematics
The mathematics module emphasizes quantitative reasoning rather than formula memorization. It is a 35-item test with content areas of pre algebra, elementary algebra, intermediate algebra, coordinate geometry, college algebra, and trigonometry. An example of a pre-algebra question is, "How much greater is the product of -3, -7, and 5 than their sum? A. -110, B. -100, C. 90, D. 100, E. 110."
Science
The science module emphasizes reasoning skills instead of rote scientific knowledge. The 45 questions are drawn from the biological sciences (e.g., biology, botany, and zoology), chemistry, physics, and the physical sciences (e.g., geology, astronomy, and meteorology). There are eight passage sets along with a set of multi-choice questions. An example taken from a passage that summarizes two scientific tables is as follows:
The data suggest that subjecting plants to which of the following conditions would result in the greatest seed masses?
A. 8 hours of light, adequate water supply, and 23°C
B. 8 hours of light, decreased water supply, and 23°C
C. 14 hours of light, adequate water supply, and 23°C
D. 14 hours of light, decreased water supply, and 29°C
Critical Thinking
The critical thinking test asks students to clarify, analyze, evaluate, and extend arguments. This module has four passages with a total of 32 items. Each passage uses several formats to present the arguments and is followed by a set of multiple-choice test items. An example question under a passage with two persons expressing differing opinions is, "A’s argument in favor of social welfare programs relies on which of the following assumptions?"
Administration
Institutions can customize their testing regimes to suit their purposes. The CAAP can be administered at different times of the year and may be given to students only once or several times over a student’s college career. Institutions should carefully choose the testing scenario and the independent modules that will provide them with information addressing the particular outcomes in question. Examples of possible administration scenarios are given below.
Using the CAAP
The CAAP is used extensively by institutions for many purposes. For states emphasizing public accountability, the CAAP has been used to track entire systems over a period of time, using test results as a benchmark for progress. Often, these results are published in a "public report card." [3]
Universities, state schools, two- and four-year colleges, and business schools use the CAAP to establish student learning gains, evaluate the development of student skills over time, and study how this growth compares to students at other colleges of similar type. This information may be used for many purposes, such as documenting student performance, evaluating institutional effectiveness, analyzing program success, or identifying when intervention is warranted. Institutions can choose which module(s) to use based on what question is being asked.
Within institutions, researchers in higher education often study certain student characteristics (such as GPA, minority status, or age when entering college) or environmental factors (such as commuting or courses taken), and how these qualities impact learning. CAAP results can be used to assess the learning component in these types of studies. The test itself asks for data about student characteristics and demographics (such as gender, ethnicity, and major), allowing for "built-in" subgroup analysis. For example, comparing the CAAP scores of biology majors and English majors might indicate a relationship between a particular curricular track and certain learning outcomes. CAAP scores might also be combined with data from other surveys (the CIRP Freshman Survey or the National Survey on Student Engagement/NSSE, for instance), retention and graduation rates, course grades, etc. Investigation as to why subgroups differ is suggested in order to work towards intentional improvement.
The CAAP survey also can be a useful tool for assessing the effectiveness of an educational practice or curriculum. One might administer one or more CAAP module(s) both before and after a specific educational experience (e.g., a colloquium or general education curriculum) to evaluate how that experience affects the learning outcomes in question. Again, exploring why certain practices or curricula are effective (or are not) is an important component of this type of assessment.
Limitations
In general, the CAAP is a dependable instrument offering institutions flexibility and convenience. However, the test has several shortcomings. Because the CAAP is a paper-and-pencil administration, the level of a student's motivation can effect test results. Hoyt demonstrates that average test scores increase when students increase their efforts on the exam. [4] CAAP addresses this issue by including questions that ask about motivation.
Similarly, the ability to customize the CAAP from school to school introduces a mix of sampling scenarios. Whereas one institution may require students to take the test or offer incentives for them to participate, other colleges may sample a much smaller, more select group of their student bodies. This is a concern when comparing data among schools. [4]
This point suggests another drawback of the CAAP. Because the test is so ready-made and easy to administer, it may be used without thoughtful regard to whether the test actually assesses institutional goals. The University of Tennessee, Knoxville, provides an illustration of this notion. This university examined how CAAP content represented its general education goals. Faculty indicated less than one-third of the CAAP content mirrored the goals established by the university and the test was deemed insensitive to the general education coursework at the university. [5] Another college dropped the Essay Writing test after they found it did not provide sufficient sensitivity to a change in student essay skills. [4] These examples underscore the importance of choosing an assessment tool that can answer the study questions and of combining qualitative tools with standardized tests.
Finally, ACT acknowledges that the cross-sectional design—that is, giving the test to entering first-year students and then later in the year to upperclass students—poses a challenge in matching student characteristics across groups. It also cautions that the CAAP should not be used as the sole instrument for high-stakes student evaluation or by individual faculty for course evaluation. [1]
Conclusion
The CAAP is a standardized test measuring general education outcomes. The test is widely used for a variety of purposes from evaluating statewide higher education performance to looking at individual student development. The CAAP offers institutions flexibility in timing and study design. The six independent modules of the CAAP include reading, writing skills, writing essay, mathematics, science, and critical thinking. Any or all of these modules can be administered in a number of study designs, including a one-time-only test or a linkage of the CAAP with previous pre-college tests. Furthermore, ACT provides a wealth of information and support regarding testing administration, score interpretation, study design, etc. The modules of the CAAP aim to address a student’s ability to reason effectively and integrate concepts, and therefore could be useful in assessing some aspects of a liberal arts education. Certainly, when assessing liberal arts outcomes, other instruments including qualitative tools should be employed to better evaluate the overall institutional culture.
References