Examining the Reliability and Validity of the ASSC/WASSC School Climate Assessment Instrument (SCAI)

What is the Function of the SCAI?
The primary function of the ASSC SCAI is to provide a mirror with which those within an individual school may explore the quality of their school's climate. It provides a scoring procedure that allows for a quasi-quantitative analysis of the quality of climate across 8 dimensions. While it does produce numeric rating data, it is not intended as a school-to-school comparison instrument. In fact at ASSC, we do not feel there is a great deal of meaningful value to school-to-school comparison. However, the SCAI can be a valid and reliable tool for pre and post assessment measurements, or a longitudinal examination of climate at one school over time.

The Validity of the SCAI
The ASSC SCAI instruments assume a comprehensive definition of school climate. The ASSC definition of climate is - "the perceptions and practical realities of those within a school as a result of everything that happens within that school." As a result, we define climate across 8 separate but inter-related dimensions. Given this definition the SCAI instruments provide a valid measure of overall school climate. Much of the validity comes from the fact that there are up to 80 items on the instruments that are designed to obtain a broad range of the essential features of the school. The validity of the SCAI instruments is demonstrated in the following areas:

• Face Validity - when participants examine the items within each of the SCAI, they will find that what is being described is familiar to them and reflects an accurate analysis of what takes place in a school. These items are further validated by current research findings and recognized characteristics of effective schools.
• Construct Validity - each of the eight scales is based in a theoretical set of constructs. Items within each scale relate to one another on both the practical and theoretical levels. In other words, at the basis of the items are a set of principles that predict school efficacy, and therefore when one finds certain circumstances within a school one also tends to find others.
• External Validity - if you make your school better by your own definition, you will find that it is reflected in your score on the SCAI. In addition you will find that your results are noticeable to others and reflect what others would perceive as change for the better, this includes both behavioral as well as academic gains.

The Reliability of the SCAI
The ASSC SCAI instruments will tend to achieve greater levels of reliability than instruments that use a yes or no structure due to the descriptive nature of the items themselves. Analytic type measures (i.e., rubrics) such as that used in the SCAI have been shown to obtain higher degrees of reliability when compared to ratings obtained from undefined Likert scales or yes - no items. Subjectivity is greatly decreased in analytic type items when compared to purely Likert-type items. However, it should be pointed out again, that the ASSC SCAI is not intended as a tool for school to school comparison, therefore the primary psychometric concerns in this area were related to instrument soundness. As a result, the reliability aspect of soundness for the ASSC SCAI is continuously examined in the following areas:

• Dimension-level Sub-scale Reliability - each SCAI subscale will achieve reliability measures that are generally acceptable, typically around .7 - .8 - but this will vary depending on the characteristics of the school. Effective sampling is encouraged. In addition, ASSC encourages dis-aggregation of the data to better interpret results.
• Intra-rater Reliability - given an adequate amount of time and a common path through a school the SCAI has show a high degree of inter-rater agreement. This is primarily a result of the clear and concrete language used to define the levels at each item. A school can expect to obtain around .9 levels of inter-rater reliability among independent observers.
• Inter-dimension Reliability - one of the most notable features of the SCAI will be the fact that ratings across dimension show a high level of correlation. This is primarily a function of the nature of how school climate manifests itself, but it also reflects upon the reliability of the instruments. The implications of all 8 dimensions being related are profound. It points to one of our main assertions about school climate - that is that everything is related. On a psychometric level this shows that principles that characterize the health of a school are integrated within each dimension separately as well as across dimensions. On a practical school assessment level, it implies that improvement efforts that take into consideration a broad spectrum of changes and a holistic focus will be more successful than those that are defined by piece-meal efforts or isolated interventions.