Multilevel exploratory factor analysis: illustrating its surplus value

This paper illustrates the surplus value of multilevel exploratory factor analysis in
educational effectiveness research. Educational researchers often use measures for
process variables at the class or school level to explain differences in student
outcomes. Recently, van de Vijver and Poortinga (2002) have developed a
procedure for multilevel exploratory factor analysis which can be extremely useful
in this kind of application. Their procedure, which is based on the ‘‘multilevel
confirmatory factor analysis’’ framework of Muthe´n (1994), is demonstrated by
means of constructing measures for school process variables based on teacher
data. Every step of the procedure is nicely illustrated and commented on using
these data. Furthermore, the meaning of the findings and challenges when using
multilevel exploratory factor analysis are emphasized.

Keywords: multilevel exploratory factor analysis; school process variables;
educational effectiveness research

Introduction
Research into the relevance of certain class- or school-level process variables for the
learning of students has a very important informative role for both policy-makers
and school heads. In addition, researchers developing comprehensive educational
effectiveness models also have a considerable interest in knowing which process
variables matter for students (e.g., Creemers & Kyriakides, 2006; Scheerens, 1990).
Process variables concern elements of the learning environment that relate to the
internal functioning of the class or the school (Scheerens & Bosker, 1997). Schoollevel
process variables, for instance, include the extent to which teachers cooperate
within the school or the decision-making policy in the school. At the class level,
process variables often involve aspects of the teacher’s behaviour, such as the way
the teacher structures the lessons or the way he or she interacts with the pupils.
Numerous studies have shown a positive association of several process variables at
the class or school level with children’s academic achievement, other cognitive
performances, and psychosocial functioning (e.g., Greene, Miller, Crowson, Duke, &Akey, 2004; Johnson & Stevens, 2006; Muijs & Reynolds, 2003; Opdenakker & Van
Damme, 2000; Webster & Fisher, 2003).
Different means are available to collect information about class or school process
variables, such as questionnaires, interviews, and observations (Freiberg, 1999).
Being both the most efficient and the least expensive way to reach a large group of
information suppliers, questionnaires have been used by the majority of the
researchers investigating process variables in the class or school environment. In the
course of completing such a questionnaire, several participants provide their
perceptions of the class or school environment, such as teachers describing their
school environment (Fraser & Walberg, 1991; Halpin & Croft, 1962; Hoy, Tarter, &
Kottkamp, 1991). Additional benefits of this kind of inquiry entail the availability of
different views of the same environment and the fact that judgments are typically
based on a large amount of experiences within the particular class or school, which
hopefully contributes to a more differentiated as well as a more accurate picture (den
Brok, Bergen, Stahl, & Brekelmans, 2004). Reviews have shown that, despite the
long history of debate about the validity of student ratings of their learning
environment, reliable and valid data can result under certain conditions, like the aid
of professionals when constructing the questionnaires (e.g., Aleamoni, 1999). A
similar conclusion may be drawn for the validity of teacher ratings of their school
environment. Yet, there are also drawbacks related to the use of questionnaires. In
comparison with interviews, for example, it is not possible to request additional
information. In relation to observations, some may argue that subjective instead of
objective data are obtained. On the other hand, it might be plausible that subjective
perceptions are more closely related to student outcomes or teaching behaviour than
objective observations (e.g., den Brok et al., 2004). Perhaps the most fundamental
disadvantage of using questionnaires rather than observations concerns the need to
develop reliable and meaningful measures for class and school process variables,
based on the individual perceptions.
Regarding the traditional approach, a first step in developing these measures for
process variables involves performing a standard factor analysis on the individuallevel
data. A factor analysis investigates the underlying structure of the items in a
questionnaire by defining a set of common latent dimensions or factors. To this end,
associations between the items (i.e., a correlation or covariance matrix) are
examined, yielding a number of latent variables that account for a considerable
proportion of variance in the associations (Hair, Anderson, Tatham, & Black, 1998).
These latent variables or factors represent for instance the way in which individual
teachers perceive their school environment. In case of an a priori theory,
confirmatory factor analysis (CFA) is recommended, whereas exploratory factor
analysis (EFA) should be used when no prior structural hypotheses are available. As
a next step, the factor scores or factor scales are aggregated into the group level,
which represent measures for process variables.
However, several shortcomings are related to the aggregation of individual-level
factors that are constructed by means of standard EFA or CFA. At first, this
conventional method does not account for the dependency that is usually present in
this kind of data. Consider several teachers who evaluate their school environment.
Evidently, teachers from the same school are expected to give more similar answers
concerning their school environment than do teachers from different schools. But,
when applying a standard factor analysis, one treats all teacher scores as perceptions
of a separate school environment. From a technical-statistical point of view, not taking this dependency into account leads to overestimated inter-item correlations or
covariances and, as a consequence, to misleading standard errors for parameter
estimates and model fit statistics. This has been repeatedly shown in the
methodological literature (e.g., Julian, 2001; Muthe´n & Satorra, 1995). Even in
1976, Cronbach has already mentioned the need of considering the hierarchical data
structure when analyzing students’ perceptions of their learning environment. A
second important drawback relates to a more conceptual issue. When aggregating
the individual-level factor scores, one assumes absolute correspondence between the
individual student- or teacher-level underlying dimensions and the relevant class or
school process variables. But, among the few studies addressing factor analysis at
both the individual and the group level, several have demonstrated distinct latent
factors at both levels when making use of perceptual data (den Brok, Brekelmans, &
Wubbels, 2006; Holfve-Sabel & Gustafsson, 2005; Westling Allodi, 2002).
An appropriate way to overcome the limitations of standard factor analysis plus
aggregation involves the use of multilevel factor analysis (Hox, 2002; Muthe´n, 1991,
1994). Among others, Muthe´n (1990) has developed an approach to perform
multilevel confirmatory factor analysis (MCFA), which has already been used to a
limited extent in educational effectiveness research (e.g., den Brok et al., 2006;
Holfve-Sabel & Gustafsson, 2005; Toland & De Ayala, 2005). It concerns a rather
complex methodology that requires an a priori theoretical framework. This
alternative procedure separates the collective part of the perceptions from the
individual, idiosyncratic part (den Brok et al., 2004). This way, two separate factor
analysis models are obtained: one that accounts for the structure of the items
between individuals within groups and one that accounts for the structure of the
items between groups (Muthe´n, 1994). In order to achieve two separate factor
solutions, the total variance in the observed items is decomposed into four main
categories: variance due to latent factors at the group level, residual variance at the
group level, variance due to latent factors at the individual level, and residual
variance at the individual level. Consequently, the factors at the group level
correspond to the measures for process characteristics. For more technical details,
see Muthe´n (1989, 1990).
Unfortunately, within the domain of educational effectiveness research, a
theoretical background has seldom been available (Scheerens & Creemers, 1989;
Thrupp, 2002). Furthermore, due to the very limited amount of research conducted
so far, it is unfeasible to define an a priori factor structure at the group level. This
obviously limits the applicability of MCFA in educational effectiveness research at
present. Only when we acquire more knowledge about the relevant dimensions at the
group level, this approach becomes very useful. The authors van de Vijver and
Poortinga (2002) have acknowledged this drawback and have proposed a similar
multilevel approach based on EFA, which has clear advantages in the early
exploratory research stages. Their procedure has been derived from Muthe´n’s
proposed method (Muthe´n, 1994) and has been adapted to specific objectives in
cross-cultural research. We present their procedure as modified for use in
educational research. As a first step, they recommend to conduct EFA on the total
dataset without taking into account the hierarchical structure. Next, intraclass
correlation coefficients (ICCs), which indicate the proportion of variance observed
between groups (e.g., classes or schools), need to be estimated to determine the
appropriateness of multilevel exploratory factor analysis (MEFA). Thirdly, the
between-group and pooled-within-group correlation or covariance matrices are computed. While the pooled-within-group matrix is based on the individual
deviations from the corresponding group mean, the between-group matrix is based
on the group deviations from the grand mean (see Muthe´n, 1994). Subsequently, van
de Vijver and Poortinga suggest carrying out an EFA of the pooled-within-group
matrix. Finally, EFA is conducted on the between-group matrix to evaluate the
correspondence with the pooled-within-group structure.
Thus far, we are unaware of any educational effectiveness studies applying
MEFA. Therefore, we propose to illustrate the surplus value of this multilevel
approach for educational and school effectiveness researchers. We will present a
didactic step-by-step guide analogue to Reise, Ventura, Nuechterlein, and Kim
(2005), who provided such a guide specifically for personality researchers. MEFA
will be used to construct measures for school process variables based on teachers’
perceptions.

Methodology
The data used in this study were collected as part of a longitudinal research project in
secondary education, LOSO (i.e., the Dutch acronym for Longitudinal Research in
Secondary Education). Starting from the beginning of the 1990s, the secondary
educational career of 6,411 students, entering one of 57 schools, was tracked in
Flanders (i.e., the Dutch-speaking part of Belgium). In Flanders, secondary
education consists of six grades that are classified into three cycles. By reason of
feasibility, almost all schools from three regions in Flanders were included,
representing to a certain extent the Flemish secondary school in general.
Characteristics such as the curriculum offered, the school size, the school type,
and the school system (i.e., the participation of Catholic and public schools)
corresponded to the educational situation in Flanders. Within the LOSO project,
data were gathered from students, teachers, principals, and parents. Since students
continued to be followed even when changing schools, the entire student sample is
nested in 90 schools, among which approximately 65 per cycle (Van Damme, De
Fraine, Van Landeghem, Opdenakker, & Onghena, 2002).
Sample
In the current study, data provided by a teacher sample were used. During the school
year 1990–1991, teachers were administered a questionnaire concerning school
characteristics. The inquiry was done separately for the teachers from the first cycle
(i.e., junior high school in Flanders) and for the teachers from the highest two cycles
(i.e., senior high school in Flanders). To this end, representative samples (i.e., with
regard to the courses given) of approximately 15 Cycle 1 teachers and 17 Cycle 2 and
Cycle 3 teachers were drawn from each school, where possible. In four schools with a
small number of teachers, representative samples of approximately 17 teachers were
taken across all three cycles (Van Damme, Van Landeghem, De Fraine,
Opdenakker, & Onghena, 2004).
Due to a different organization of study programmes between Cycle 1, on the one
hand, and Cycles 2 and 3, on the other hand, we assumed that differences may exist
with regard to at least some process variables. Therefore, we used MEFA once with
the teacher data grouped within schools including all teachers from a school (i.e.,
without differentiating between the cycles) and once differentiating between Cycle 1 teachers and Cycle 2 and Cycle 3 teachers. Thus, in each of both analyses two
separate levels were distinguished. In order to compare the results of both ways of
grouping univocally, the four schools with a small number of teachers were not
included in our analyses. As a consequence, our final sample included 1,550 teachers
nested in 82 schools. Of the 82 schools, 22 only provided Cycle 1 education, 28 only
provided Cycles 2 and 3 education, and 32 provided education in all three cycles.
These 82 schools could be divided into 113 cycle groups (i.e., Cycle 1 versus Cycles 2
and 3). The teacher sample ranged from 1 to 32 within each school and from 1 to 18
within each cycle group. Since we had a sufficient number of groups to perform
multilevel factor analysis (i.e., minimum between 50 and 100; see Muthe´n, 1994), the
variation in group sizes did not affect the stability of the factor structure.
This final sample comprised approximately 44% teachers from the first cycle and
56% teachers from the highest two cycles. Both sexes were (unintentionally) nearly
equally represented, and the teachers’ age (with 69 missing) ranged from 23 until 63,
with a mean age of approximately 42 (SD ¼ 7.80).
Measures
Data from the ‘‘School Characteristics Questionnaire for Teachers’’ were selected to
develop measures for school process variables. The construction of the questionnaire
was based on (a) the consultation of the most important school (characteristics)
studies from The Netherlands, Flanders, and some non-Dutch countries until the
first half of 1989 (see Opdenakker, 2003); (b) the special issue ‘‘Developments in
school effectiveness research’’ of the International Journal of Educational Research,
edited by Creemers and Scheerens (1989); and (c) theoretical reflections, as presented
by Opdenakker.
In particular, teachers were asked to answer more than 300 items concerning
their background, teaching practice, instructional and pedagogical framework, and
daily school life (Van Damme et al., 2004). Due to reasons of parsimony, 66 items
which had to be judged on a 4- or 5-point scale were selected (listed in Appendix 1).
During the selection process, considerable attention was given to theoretical
rationales. First, items involving the daily school life of teachers were chosen, since
the other issues addressed in the questionnaire rather concerned teachers’ selfperceptions.
Second, empirical support concerning the association with several
student outcomes was taken into account, although we acknowledge that different
operationalizations of certain process variables may distort the comparability
between studies. As a result, four different components of the daily school life of
teachers were included in the present study:
(1) Cooperation between teachers: Both international and Flemish literature
have revealed a direct positive association between teacher cooperation and
student achievement (e.g., Anderson, 1982; Mortimore, 1997; Scheerens,
1990; Scheerens & Bosker, 1997; Verhoeven et al., 1992), as well as between
teacher cooperation and school well-being (Vandenberghe, Bohets, Claus,
Vernelen, & Viaene, 1994).
(2) Decision-making, in particular regarding instructional aspects: While the
positive relationship of participatory decision-making of teachers concerning
various decision domains (e.g., curriculum development, personnel, student
life, and financial issues) with student achievement has often been found to be indirect (Verhoeven et al., 1992), Sweetland and Hoy (2000) have
demonstrated a direct positive association when only including involvement
in decision-making with regard to class and instructional features.
(3) Social system: Sweetland and Hoy (2000) have shown that relationships
among teachers as well as between teachers and the principal are positively
associated with the participatory decision-making of teachers concerning
class and instruction. As such, relationships among school staff are indirectly
related to student achievement.
(4) Rules: The establishment of rules concerns one aspect of an orderly school
atmosphere (Scheerens, 1990; Scheerens & Bosker, 1997; Stockard &
Mayberry, 1992). Since the rules described in the selected items involve
both learning (e.g., tests and homework) and the everyday class and school
environment (e.g., interruption of the lesson) of the students, a direct
association with student achievement may be expected.
Data analysis
Prior to conducting the analyses, our data were screened, and necessary assumptions
were assessed. Furthermore, some of the item scores were reversed to assure that a
high score equals a positive feature. This is particularly important with regard to the
estimation of the pooled-within-group and between-group matrices to minimize
convergence problems (Gustafsson & Stahl, 2005). Thus, after reversal, all of the
items related to ‘‘the social system’’ in the school contributed positively to good
social relationships at the school. A high score on the items involving ‘‘autonomy in
decision-making’’ indicated high teacher autonomy in decision-making at the school.
Analogously, a high score on the items about ‘‘participatory decision-making’’
indicated high participation of teachers in the decision-making process at the school.
High cooperation between teachers was associated with a high score on the items
about ‘‘meetings among teachers’’. Finally, all of the items concerning ‘‘rules’’
contributed to a strong establishment of rules at the school.
The step-by-step procedure suggested by van de Vijver and Poortinga (2002) was
slightly adjusted to our research objectives. Since we were only interested in
including items that sufficiently differentiated between schools as well as between
Cycle 1 versus Cycles 2 and 3, the ICC(1) was at first calculated for each of the items.
This intraclass correlation coefficient indicates the proportion of total variance that
can be attributed to between-school or between-cycle group differences (see Lu¨ dtke,
Trautwein, Kunter, & Baumert, 2006):
ICCð1Þ ¼
tb
tb þ tp
where tb is the variance between schools or cycle groups and tp is the variance within
schools or cycle groups.
Next, the necessary correlation matrices were computed. To this end, conventional
software for covariance structure analysis can be used, like Mplus (Muthe´n &
Muthe´n, 1998–2006) and LISREL (Jo¨ reskog & So¨ rbom, 1996). We used STREAMS
3.0.4 (Gustafsson & Stahl, 2005) in combination with Mplus 4.0. STREAMS (i.e.,
STRuctural Equation Modeling made Simple) provides pre- and postprocessors to
214 E. D’Haenens et al.
Downloaded
software programs for covariance structure analysis, which makes it very easy to set
up and estimate the necessary correlation matrices. Correlation matrices instead of
covariance matrices were computed to take account of the scaling differences between
the items (Hair et al., 1998). Full Information Maximum Likelihood estimation was
used to deal with missing data (i.e., FIML; Muthe´n & Muthe´n, 1998–2006). The
pooled-within-group and between-group correlation matrices were estimated once
regarding schools and once regarding cycle groups. This way, we obtained: (a) a total
correlation matrix of the teacher data without accounting for the nesting within
schools or cycle groups; (b) a pooled-within correlation matrix at the teacher level,
reflecting differences between teachers within schools; (c) a between-group correlation
matrix at the school level, reflecting differences between schools; (d) a pooled-within
correlation matrix at the teacher level, reflecting differences between teachers within
cycle groups; and (e) a between-group correlation matrix at the cycle group level,
reflecting differences between cycle groups.
Subsequently, EFAs were conducted on each of the five calculated matrices using
SAS (SAS Institute Inc., 2003). Principal factor analysis was chosen to extract the
factors, with squared multiple correlations used as prior communality estimates.
This is advised when one desires to identify the underlying latent dimensions, since
principal factor analysis only takes into account the common variance present in the
data (Hair et al., 1998). Several criteria were used to decide on the number of factors:
(a) the Kaiser criterion (i.e., eigenvalues greater than 1; Kaiser, 1958), (b) the scree
plot of the eigenvalues (Cattell & Vogelmann, 1977), and (c) parallel analysis
(O’Connor, 2000). Based on these results, different factor solutions were compared,
regarding (a) the interpretation of the factors; (b) the number of items included in
each factor; (c) the size of the pattern and structure coefficients; and (d) the
percentage of variance explained between the items, both in general and with regard
to each factor. As suggested by Fabrigar, Wegener, MacCallum, and Strahan (1999),
an oblique rotation using promax was at first performed to determine the size of the
correlations between the extracted factors. When correlations existed between the
factors, the oblique solution was retained. In case the factors were uncorrelated, an
orthogonal varimax rotation was carried out.
Finally, the internal consistency of the factors was calculated. With regard to
EFA of the total correlation matrix, standardized Cronbach’s alpha was computed.
However, this alpha value ignores the nested data structure. Therefore, regarding
EFA of the pooled-within correlation matrix and of the between-group correlation
matrix, a different strategy was needed (see Kamata, Bauer, & Miyazaki, 2008). The
teacher-level reliability was obtained as follows:
^ap ¼
^tp
^tp þ ^s2
n
;
where ^tp is the estimated variance at the teacher level; ^s2 is the estimated variance at
the item level; and n is the number of items within the factor. Likewise, the grouplevel
internal consistency was obtained by taking the average of the reliability
coefficients for each school or cycle group k:
^ab ¼
1
KX
K
k¼1
^abk;