This project used a selection of the public 12^th grade civics questions from the National Assessment of Educational Progress to ascertain how CSLA students did compared with national norms. POLS 150 classes during Winter, 2001, participated in the project, with half the classes taking both a pretest and a posttest, and half the classes taking the posttest only.

The use of the 12^th grade civics questions provided a reasonable test of student learning in POLS 150. Faculty teaching the course felt the questions were appropriate for POLS 150 students. The students achieved results that were favorable compared with the NAEP results, with the pretest falling two percentage points short of the national norm (but two percentage points better than a composite score weighted to the CSLA racial/ethnic demographics). The posttest showed a 5% increase on the average.

Compared with the NAEP composite results, CSLA students did relatively well on the questions relating to judicial process and civil rights, but less strong regarding Congress and redistricting. However, more questions need to be asked in order to focus on the areas of strength and weakness, which points toward a study using the existing final examinations.

The goal of this project was to undertake several efforts to evaluate the effectiveness of POLS 150, Government and American Society, in teaching students the basics of American and California government. The original project called for:

• Two principal investigators (Professors Anagnoson and Wagner) who would use the National Assessment of Educational Progress 12^th grade civics questions plus locally generated California government questions to assess students on general American and California government topics (in particular, the constitution, federalism, elections and voting, Congress, the presidency, the judiciary, and political behavior), and
• One principal investigator (Prof. Lim) to investigate how students thought about politics and whether such thinking changed as a result of taking POLS 150.

The project approved included only one-third the amount of money, however, and consequently Professors Wagner and Lim withdrew from the project, as there were no funds to compensate their time, and Professor Anagnoson unexpectedly assumed the chairpersonship of the Department on September 1, 2000. Consequently, Professor Nadine Koch assumed responsibility for the project and used the four units of release time, consulting with Professor Anagnoson as necessary. A student assistant (Mr. Adam Fraser) inputted the data into a spreadsheet. Professor Anagnoson converted the data to a statistical package, did the analysis, and wrote the final report in consultation with Prof. Koch.

The project was narrowed simply to the 12^th grade civics questions from the National Assessment of Educational Progress; both the California government sections and the "how students think about politics" portions were postponed to a future project.

Background.

POLS 150 is one of four courses listed in the American Institutions area of the General Education program. History 202A or 202B, the two halves of American history, meet the U.S. History part of the requirement. POLS 150 meets both the U.S. Constitution and the State/Local Government part of the requirement. POLS 200, California State and Local Government, meets the State/Local Government part of the requirement only and is generally taken by those who have had an introductory course in American politics out of state.

The Department of Political Science offers 8-9 sections of POLS 150 each quarter during the academic year; it offers one section of POLS 200 each quarter (usually on a Credit by Exam basis). About 500 students take POLS 150 each quarter. It is a course characterized by standard textbooks on which there is broad, general agreement as to the subject matter covered.

The purpose of the assessment is to determine (1) the extent to which the course is successful in teaching students the basics of American and California politics; (2) the areas of the course that are strongest and weakest.

The methodology is to use the public questions on the 1998 Civics Report Card of the National Assessment of Educational Progress (the questions used were for 12^th graders). The 38 public questions cover the range of topics usually taught in POLS 150, with a strong emphasis on constitutional knowledge and interpretation. The questions are appealing because they have been given to a substantial national sample in the 12^th grade, just one to two years before most students take POLS 150 at Cal State LA. And for each question, we not only have the proportion of 12^th grade seniors who got the question right, we have a breakdown of the number selecting each option by gender, race/ethnicity, parents’ highest level of education, type of high school, region of the country, type of location (urban, suburban, rural), Title I participation, eligibility for the National School Lunch program, and by achievement level on the exam of advanced, proficient, basic, and below basic).

POLS 150 sections in Winter, 2001 were used for the study. About half the sections received a pretest of 11 questions. Two of the questions were qualitatively scored by Profs. Anagnoson and Koch. The posttest was given to all sections and included seven questions also asked on the pretest.

The constraints on the questions and the project included:

• A maximum of one page, two sides was to be used for the questions, as previous exam administrations through the Ad Hoc GE Assessment Faculty Working Group, of which Anagnoson was a member, had shown that students, in spite of exhortations to the contrary, "blew off" longer exams.
• Instructors had the option of counting the posttest questions as part of the final exam.
• Instructors were to administer the pretest on the second day of class, before any meaningful material had been covered.
• Instructors were to administer the posttest as part of the final exam or on the last day of class.
• Students were told that the pretest would not count toward any grade, were urged not to "blow off" the test, and that the endeavor was important for improving the way we teach POLS 150 and students learn the material.
• Individual identification of students was necessary on the pretest and posttest in order to match the same students on the pretest and posttest, but the identifications were not used for any other purpose.

The data table shows the general results. The columns are labeled as follows:

The NAEP National scores show the performance of a random sample of 12^th graders, tested in their senior year. The range of correct answers on the 11 questions chosen is from 31% to 90%. The average is 53%, and the median is 55%.

The CSLA pretest results, shown in the third column, labeled "Pretest Percent," are similar, with a range from 20% (on the same question, about the result for most bills introduced in Congress, that the national sample got 31% correct) to 90%. The mean is 55%, and the median is 58%, just a shade higher than the national sample of 12^th graders.

Two of the questions were open-ended, where Profs. Anagnoson and Koch had to grade the students’ answers, and the standards used by the NAEP graders are not public. As a result, the CSLA sample had 62% of the students achieving a successful answer on question 1 ("Describe one policy area where state governments affect the lives of citizens" and "Describe one way in which citizens can affect state government’s policy in this area") and 75% on question 8 ("Describe two factors besides party identification that influence voter preference"), while the national sample got only 37% and 59% fully correct respectively. Consequently, those two items were removed from the comparison, although their results are reported in the data table.

Compared with the national sample on the remaining 9 items from the pretest, CSLA students did better than the national average on three questions, tied it on two, and did worse on four, for an average of –2%.

However, when the national results are weighted by the CSLA ethnic percentages to replicate the campus averages of 53% Latino, 9% African-American, 16% white non-Hispanic, and 22% Asian-American, the results are 2% above the national averages, rather than 2% below, as noted in the NAEP Composite column and the "Pretest v. Composite" column.

In general, then, CSLA students, on the pretest, achieved results roughly comparable to the national sample of 12^th graders.

Seven of the 11 questions on the posttest were the same as those on the pretest. For the posttest as a whole, the average national result is 50% correct. For the national sample weighted to be the same as the campus figures, the average national result is 46% correct. The students in POLS 150 achieved an average result of 53%. [One would expect them to do a bit better given that they took POLS 150 and were being tested at the end of the course. Of course, the test questions were optional, and most instructors administered the two page exam on the last day of class, thus removing the incentive to try on the questions as hard as one would try to do well on the final exam.]

Compared with the national sample (unweighted), they did an average of 3% better, with 9 of the 11 questions having a better result. Compared with the weighted national sample, they did 7% better.

At the bottom of the data table are the results for the seven questions that were the same on the pretest and the posttest. Two of these had 11% and 17% improvements from the pretest to the posttest; four others showed a 1% to 8% improvement. One went down – the question on Social Security where 90% of those who took the pretest got the question correct.

In general, then, there is an improvement between the pretest and the posttest on the POLS 150 questions, with the students in the course scoring very favorably compared with the 12^th grader (some of whom would have had 12^th grade American government courses) sample used in the National Assessment of Educational Progress. A question on judicial reasoning and one on the outcome for bills introduced in Congress showed particular strength.

In general, the use of the 12^th grade civics questions provided a reasonable test of student learning in POLS 150. Faculty teaching the course felt the questions were appropriate for POLS 150 students. The students achieved results that are quite favorable compared with the NAEP results. Future assessments in POLS 150 should focus on particular areas that might be taught well or poorly. One possibility is having outside examiners read final exam essays in POLS 150, or collecting the final exams, classifying the multiple choice questions that are very common on such exams by subject matter, and then seeing if there are particularly easy or difficult topics for CSLA students.

Compared with the NAEP composite results, CSLA students did relatively well on the questions relating to judicial process and civil rights, but less strong regarding Congress and redistricting. However, more questions need to be asked in order to focus on the areas of strength and weakness, which points toward a study using the existing final examinations.

The budget for the project was cut by two-thirds by the selection committee and proved to be inadequate even for this reduced project. The four units of release time did not compensate the investigators adequately. They had to ask for an extra $750 in assessment money for a graduate assistant to help compile the data (some 543 observations and 40 variables) during Spring quarter. The budget allocated was used for four units of release time and supplies/duplicating for the project. Anagnoson received no compensation for writing the final report and analyzing the data; Koch received inadequate release time for her work coordinating the project and compiling the data.

J. T. Anagnoson

N. S. Koch

October, 2001