Factor analysis and multilevel factor analysis

Factor analysis describes a family of statistical techniques that seek to identify continuous unobserved variables that most plausibly account for covariances among observed variables (Brown, 2006). The fundamental assumption in factor analysis models is that unobserved variables exist and are quantitatively distributed in people and are fundamental causes to multiples of the observed variables (e.g., responses to questions on EMA prompting following a lapse event). The goal of a factor analysis is to identify measurement models and quantify how well the observed variables measure the presumed underlying variables. When measurement models are strong, such as when there are many items (more than 4) that measure the underlying construct, each providing some unique information (relative to the other items) that is relevant to understanding a persons’ position on the underlying latent variable, the factor model itself can be viewed as a successful test of the validity of the construct (Borsboom, 2005). Ideally factor loadings will be high. Factor loadings describe the strength with which items measure the underlying construct, and are sometimes also referred to as measurement slopes. Factor loading is generally reserved for the standardized form of a measurement slope, and is on the scale of a correlation coefficient and describes the correlation between a standard normal underlying latent trait and standard normal observed indicator.

Multilevel factor analysis involves estimating within (repeated assessments within people) and between (person level) covariance matrices, and separately factor analyzing the covariance structure at these two levels. The within and between level covariance structures are orthogonal to one another, which implies that the factor structure revealed at the within level need not be the same as the factor structure revealed at the between level. When data are clustered, it is best practice is to model both the within and between factor structures (Huang, 2018).

Figure 1. Multilevel factor analysis The figure illustrates a multi-level factor analysis with distinct within and between level factor structure. A single between level latent trait (\(\theta\)) accounts for covariation at the person level, and two within level latent variables (\(y, z\)) account for covariation at the within level.

Evaluation of factor models. Strong factor models have items with high loadings in the underlying trait, but not so high as to imply the items are exchangeable with each other. One way to summarize the strength of the factor loadings is with the omega coefficient (Peters, 2014), which can be interpreted along the same lines as coefficient alpha for internal consistency reliability (Cronbach, 1951). Good models will also satisfy fit criteria based on how well the model-implied mean and covariance structure approximates the observed mean and covariance structure. We will interpret good fit of the model to the data when the effective sample size is at least 200 and the standardized root mean square residual (SRMR) ≤ 0.08 and there are no large (>.2) standardized residuals. The SRMR is the mean standardized residual among mean and covariance estimates for model-implied and observed values (Asparouhov, 2018).

We will also report the root mean squared error of approximation (RMSEA) and confirmatory fit index (CFI). The RMSEA is the model discrepancy (square root of the model chi-square, computed based on the difference of observed and model-implied mean and covariance matrices) per degree of freedom. The CFI is a relative fit measure and compares the fit (chi-square) of a target model to a null model or baseline model. CFI values are computed to range between 0 and 1, with higher values indicating better explanatory power of the target model relative to the null model. It is desirable to have a CFI that is at least .95. RMSEA values are bounded at 0 and well-fitting models are generally taken to have an RMSEA of < 0.5 (Hu, 1999). We will cautiously, and with judgments based on substantive theory, make model modifications on the basis lack of fit as suggested by the individual residuals and/or the SRMR.

In the multilevel context, we allow for the possibility that covariation among an observed set of items can be caused at two (or more) levels of organization, commonly referred to as “between” and “within”. In our study, where the multilevel structure derives from individuals repeatedly observed over time, the “within” level refers to specific lapse events and the “between” level refers to individual person level effects that carry across repeated assessment occasions. Our primary concern is to achieve well-fitting models with good measurement properties (i.e., high omega coefficient) at the within-person level.

Sample size and statistical power considerations. Please see Meuleman and Billiet (2009), who suggest that with sample size of 100 at the between level, the probability that within-level population values for factor loadings are contained within the sample-estimated 95% confidence intervals at the within level is 94.6% for factor loadings.

Dynamic Structural Equation Models and Multilevel Location-Scale models

When longitudinal data are avaialble, we can build more complicated and potentially informative models. Expanding upon the multilevel approach and bringing in the longitudinal, or time series, data we can use a modeling approach called multilevel location-scale models and estimated within a dynamic structural equation models (SEM) approach (McNeish, 2020).

Dynamic SEM methods were developed to accommodate intensive longitudinal data, such as those collected with EMA. The location-scale model is notable for bringing means and variances of personal characteristics into the model. In this way we will model a participant’s level of reactivity to lapse events across multiple dimensions that may predict weight loss outcomes, but the breadth or severity of these reactions, and the degree to which these multiple dimensions covary with one another. We illustrate the modeling approach in Figure 2.

Figure 2. Dynamic SEM

Figure 2 describes a within-person model and a between-persons model. The within-person part illustrates two processes observed during an EMA bout (\(y\) and \(z\)). An auto-regressive time series model is estimated for these constructs as observed over EMA assessments, which are not expected to show systematic growth but are instead stationary. Mean levels on these phenotype variables are modeled at the between level (\(y_i^b, z_i^b\)). Variances about these means (the up-and-down chatter in the line plot in Figure 2; \(\sigma^2_{1i}, \sigma^2_{2i}\), for phenotype indicators \(y\) and \(z\), respectively) are also parameterized at the between level (\(ln(\sigma^2_{1i}), ln(\sigma^2_{2i})\) and can themselves be predictors of the clinically relevant outcomes (in the Figure, energy intake and weight change). Finally, the stability (\(\phi_{1i}, \phi_{2i}\)) mutual dependence of the phenotypes and co-dependence of the phenotypes (\(\phi_{3i}, \phi_{4i}\)) at the within-level can also be evaluated in terms of their importance for understanding energy intake and weight change.

References

Asparouhov, T., & Muthén, B. (2018, 2 May). SRMR in Mplus. Muthén & Muthén Retrieved 26 May from http://www.statmodel.com/download/SRMR2.pdf

Borsboom, D. (2005). Measuring the mind: Conceptual issues in contemporary psychometrics. Cambridge Univ Pr.

Brown, T. A. (2006). Confirmatory Factor Analysis for Applied Research. Guilford Publications.

Cronbach, L. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16, 297-334.

Hu, L. t., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6(1), 1-55. https://doi.org/10.1080/10705519909540118

Huang, F. L. (2018). Multilevel modeling myths. School Psychology Quarterly, 33(3), 492.

McNeish, D., & Hamaker, E. L. (2020). A primer on two-level dynamic structural equation models for intensive longitudinal data in Mplus. Psychological methods, 25(5), 610.

Meuleman, B., & Billiet, J. (2009). A Monte Carlo sample size study: How many countries are needed for accurate multilevel SEM? Survey Research Methods, 3(1), 45-58.

Nunnally, J. C., & Bernstein, I. H. (1994). Psychometric theory (3rd ed.). McGraw-Hill College Division.

Peters, G.-J. Y. (2014). The alpha and the omega of scale reliability and validity: why and how to abandon Cronbach’s alpha and the route towards more comprehensive assessment of scale quality. European Health Psychologist, 16(2), 56-69. https://doi.org/10.31234/osf.io/h47fv

(fin)