Examining the Reliability and Validity of the ASSC School Climate Assessment Instrument (SCAI)

What is the Function of the SCAI?
The primary function of the ASSC SCAI is to provide a mirror with which those within an individual school may explore the quality of their school's climate. It provides a scoring procedure that allows for a highly valid and reliable indicator of the quality of school climate across 8 dimensions.

The Validity of the SCAI
The ASSC SCAI instruments assume a comprehensive definition of school climate. The ASSC definition of climate is - "the perceptions and practical realities of those within a school as a result of everything that happens within that school." As a result, we define climate across 8 separate but inter-related dimensions. Given this definition the SCAI instruments provide a valid measure of overall school climate. Much of the validity comes from the fact that there are enough items on the instruments that are designed to obtain a broad range of the essential features of the school. The validity of the SCAI instruments is demonstrated in the following areas:

• Face Validity - when participants examine the items within each of the SCAI, they will find that what is being described is familiar to them and reflects an accurate analysis of what takes place in a school. These items are further validated by current research findings and recognized characteristics of effective schools.
• Construct Validity - each of the eight scales is based in a theoretical set of constructs (defined in part by the 3 psychological dimensions of "psychology of success" (POS). Items within each scale relate to one another on both the practical and theoretical levels. In other words, at the basis of the items are a set of principles that predict school efficacy, and therefore when one finds certain circumstances within a school one also tends to find others.
• Predictive Validity - As depicted in the table below, The ASSC SCAI has been shown to be correlated at approximately the 0.7 level with student achievement scores by school or district (i.e., CA API). Therefore the SCAI is predictive of student achievement. And as a reliable measure of internal locus of control producing behaviors it is predictive of outcomes related to the level of internal LOC (i.e., health, reliance, use of violence, obesity, drug and alcohol use, and gang affiliation).

The Reliability of the SCAI
The ASSC SCAI instruments will tend to achieve greater levels of reliability than instruments that use a yes or no structure due to the descriptive nature of the items themselves. Analytic type measures (i.e., rubrics) such as that used in the SCAI have been shown to obtain higher degrees of reliability when compared to ratings obtained from undefined Likert scales or yes - no items. Subjectivity is greatly decreased in analytic type items when compared to purely Likert-type items.

In practice, the ASSC SCAI demonstrates exceptionally high levels of reliability as measured by the Chronbach's Alpha reliability test (0.97). The accepted standard for a reliable instrument is 0.7. Each of the sub-scales of the SCAI full version reflect alpha scores much better than that standard as well as other known school climate instruments.

• Dimension-level Sub-scale Reliability - As shown in the table above, each of the SCAI sub-scales generates a Chronbach's Alpha reliability measure of .73 or above. The overall Chronbach's Alpha for each instrument and data set ranged from .97 to .98.
• Intra-rater Reliability - given an adequate amount of time and a common path through a school the SCAI has show a high degree of inter-rater agreement. This is primarily a result of the clear and concrete language used to define the levels at each item. A school can expect to obtain around .9 levels of inter-rater reliability among independent observers. In addition, the mean standard deviation for scores for student surveys is 1.1, and for teacher surveys it is 0.8 on a 5 point scale. Given that all students do not have the same experience of school, this modest range is explainable.
• Inter-dimension Reliability - one of the most notable features of the SCAI will be the fact that ratings across dimension show a high level of correlation to one another (0.7 - 0.9). This is primarily a function of the nature of how school climate manifests itself, but it also reflects upon the reliability of the instruments. The implications of all 8 dimensions being related are profound. It points to one of our main assertions about school climate - that is that everything is related. On a psychometric level this shows that principles that characterize the health of a school are integrated within each dimension separately as well as across dimensions. On a practical school assessment level, it implies that improvement efforts that take into consideration a broad spectrum of changes and a holistic focus will be more successful than those that are defined by piece-meal efforts or isolated interventions.