User:Michael Hardy/Envelope model
Envelope models, or envelopes, include a class of statistical models that lie at the intersection of multivariate analysis and sufficient dimension reduction. The first envelope model was introduced by Cook, Li, and Chiaromonte in 2010 under the framework of multivariate linear regression.[1] It applies dimension reduction techniques to remove the immaterial information in the data and achieve efficient estimation of the regression coefficients. In cases where there is a lot of immaterial information, the envelope model can obtain substantial efficiency gains in the estimation of the coefficients compared to the ordinary least squares method (the latter of which does not account for the covariance structure of the responses). The basic idea of envelopes is to build a link that connects the covariance matrix and the regression coefficients, which results in a parsimonious parameterization of the multivariate linear regression model. Envelopes have been extended to many fields in multivariate analysis such as discriminant analysis, partial least squares, Bayesian analysis, variable selection, reduced rank regression, and generalized linear models.
Introduction
Multivariate linear regression model
The envelope model was originally introduced under the framework of multivariate linear regression:
where is the random response vector and is the non-stochastic predictor vector centered to have mean zero. Under this model, and are the unknown intercept and coefficient parameter respectively. The error vector is assumed to have mean and covariance , where is a positive-definite matrix.
Assumptions
Let be a subspace that divides into two disjoint parts: the material part and the immaterial part . is the projection matrix onto and , where is the identity matrix, and is the projection onto the orthogonal complement of (denoted by ). We assume that the following two conditions hold:
where "A ~ B" means A and B are identically distributed, "A | B" denotes the conditional distribution of A given B and denotes that A and B are statistically independent. Assumption 1 states that the conditional distribution of the immaterial part given is the same as the distribution of the immaterial part. Assumption 2 states that given , the material part and immaterial part are independent of each other.
Under the linear model, the first assumption indicates that the immaterial part does not give any information about , while the second assumption means that the covariance matrix of (i.e. the covariance matrix of the error) can be decomposed into two parts: the variability according to the material part and the variability according to immaterial part. Since can be decomposed as , we see that carries all the material information and maybe some immaterial information about , while carries just immaterial information about . To ensure that we exclude immaterial information from our estimate of , we aim to the find the smallest that carries only material information about .
Definition of envelope
In 2010 Cook et al.[1] showed that assumptions 1 and 2 are equivalent to
- where
In (i), the span denotes the linear subpace spanned by the columns of . When (ii) holds, is called a reducing subspace of .
The -envelope of , denoted by , is defined as the intersection of all reducing subspaces of that contain .
We let denote the dimension of envelope subspace, where . From (ii), it is clear that decomposes into the variation of the material and immaterial parts of . Namely, we have and .
Properties of envelope subspace
Let denote the class of all symmetric matrices. The following propositions establish some important properties of the envelope subspace , namely its relationship with the eigenstructure of and its behavior under linear transformations of .[1]
- Proposition: Let and let be the projection onto the eigenspaces of , and let be a subspace of . Then is the intersection of all reducing subspaces of that contain .
- Proposition: Let be the matrix that commute with and let be a subspace of span. Then and the following equivariance holds: . If, in addition, and reduces , then the following invariance holds: .
- Proposition: Under the standard model, let be the covariance matrix of . Then and:
Coordinate form of the envelope model
Under the envelope parameterization, the multivariate linear regression model (which we call the standard model) can be written as
where , is an orthonormal basis for the envelope subspace , and is an orthonormal basis for the orthogonal complement of the envelope subspace and the completion of (that is, is an orthogonal matrix).
carries the coordinates of with respect to and and are positive definite matrices which carry the coordinates of with respect to and respectively.
Under the standard model, the total number of parameters is
where is the number of parameters in , is the number of parameters in , and is the number of parameters in which is the covariance of . Under the envelope model, the total number of parameters is
where is the number of parameters in , is the number of parameters in , is the number of parameters in , is the number of parameters in and is the number of parameters in . When we add them together, we obtain .
Therefore, . So if , the envelope model reduces the number of parameters that need to be estimated. If , the envelope model reduces to the standard model. If , it means that .
Estimation
The envelope model parameterization does not depend on normality. However, if we assume that the errors are normally distributed, i.e. , then we can use maximum likelihood estimation (MLE) to estimate the unknown parameters: , , , and . In the envelope model, is not identifiable. However, we can uniquely estimate . In order to estimate , we need to solve the following optimization problem:
- ,
where is the sample covariance matrix of the residuals from ordinary least squares regression and is the sample covariance of , denotes an Grassmann manifold in (that is, the collection of all -dimensional subspaces of ). If we do not have normality we can still use the above objective function to estimate the envelope model. If the error has finite fourth moments then we can still get a root-n consistent estimator of the parameters. This objective function can be solved by some standard manifold optimization algorithm such as Manopt . Recently, a very fast algorithm for estimating , which does not require optimization over a Grassmannian, has also been developed.[2]
Let denote the minimizer of this objective function and be the orthonormal basis of . Then , where denotes the ordinary least squares (OLS) estimator , is the centered data matrix of the response (that is, every element in the row of is the data point in the sample of minus the sample mean ) and is the centered data matrix of the predictor. The envelope estimator of is then . We see that the envelope estimator is the projection of the OLS estimator (i.e. the estimator under the standard model) onto the estimated envelope subspace which removes the immaterial information. Since the immaterial information is identified and removed in subsequent analysis, the envelope estimator is potentially more efficient than . If , then the envelope estimator reduces to the OLS estimator, i.e. . For the full derivation, refer to Cook et al. (2010).[1]
Efficiency gains from envelope estimation
In Cook et al.[1] it is shown that under normality assumptions, the asymptotic variance of satisfies
where is the vectorization of matrix and is the asymptotic variance; that is, if , then . This means, the envelope estimator performs at least as good as the OLS estimator in terms of efficiency.
In particular, if , where represents the matrix spectral norm, then the envelope estimator is expected to provide substantial efficiency gains. That is, if the immaterial part is more variant than material part, by identifying and removing the immaterial information, we expect to realize substantial efficieny gain (i.e. much smaller standard errors).
Choosing envelope dimension size
The dimension of can be chosen by different methods such as likelihood ratio testing (LRT), Akaike information criterion (AIC), Bayesian information criterion (BIC), or cross-validation.
Examples
Comparing two population means
To illustrate the efficiency gains by envelope models, consider the problem of estimating two means of two bivariate normal populations, and . This can be studied in the envelope framework by considering as a bivariate response vector and as an indicator variable taking value in the first population and in the second population. We can parametrize in such a way that is the mean of the first population and is the mean difference. Therefore, the multivariate linear model is
The standard estimator of is the difference in the sample means for the two populations and the corresponding estimator of is the intra-sample covariance matrix. In the standard model, the estimator does not make use of the dependence between the responses and is equivalent to performing two univariate two univariate regressions of on .
In contrast, by considering envelope model, we obtain a different estimator for that accounts for the dependence structure of and and that removes all the immaterial information to the group differences. The envelope estimator, has smaller standard errors than , indicating the estimation for is greatly improved. The graphical illustration of those analysis is shown in the figure. For further analysis, refer to [3]
In the figure, without loss of generality, it is assumed . Therefore, . The left panel illustrats the standard analysis. It directly projects the data onto the axis, following the dashed line marked A, and then proceeds with inference based on the resulting univariate sample. The curves along the horizontal axis in the left panel stand for the projected distributions from the two populations. A standard analysis might involve constructing a two-sample t-test on samples drawn from these populations. There is considerable overlap between the two projected distributions, so it may take a large sample size to infer that (or ) in a standard analysis, even though it is clear that the bivariate populations have different means. This illustration is based on to facilitate visualization; the same conclusions could be reached using a different linear combination of the elements of .
The maximum likelihood envelope estimator of , , however, can be formed by first projecting the data onto to remove the immaterial information, and then projecting onto horizontal axis, as shown by the paths marked "B" in the right panel. The two curves at the horizontal axis are the projection distribution from envelope analysis. The two populations have been arranged so that they have equal distribution when projected onto ; that is, , but if the distribution is projected onto , they have different distributions. The dashed line direction is the immaterial part that is in orthogonal complement of envelope subpace, and the solid line direction is material part of population that is the envelope subspace. From this figure, it is obvious that the two population means and differ significantly.
Wheat protein data example
Another example is given in [3] and deals with wheat protein data. This dataset contains measurements and the logarithms of near infrared reflectance at six wavelengths across the range 1680-2310 nm, measured on each of samples ground wheat. We take wavelengths as responses and convert the continuous measure of protein into a categorical predictor indicating low and high protein (with sample sizes 24 and 26 respectively). The mean difference corresponds to the parameter vector in the standard model with representing a binary indicator: for high protein, and for low protein wheat.
For this dataset, , with standard errors and respectively. There is no indication from these marginal results that depends on , since the likelihood ratio test statistic has the value 27.5 on 2 degrees of freedom. Under a standard analysis, the simultaneous occurrence of relatively small -scores and a relatively large likelihood ratio statistic indicates that an envelope analysis offers advantages, although these conditions are certainly not necessary.
In particular, the envelope estimator is , with standard errors of and . We see that the standard errors for the envelope estimates of the coefficients are significantly lower than the standard errors for the OLS estimates, indicating substantial efficiency gains in estimation. To achieve the magnitude of this drop in standard errors under the standard model, we would need a sample size of to reduce the standard error from 9.4 to .46.[3]
Extensions
The envelope model has since been extended well beyond multivariate linear regression to many other areas in statistics. In 2011, Su and Cook introduced the partial envelope model[4] which allows us to apply the envelope method to a subset of predictors of interest. In 2013, Cook et al. developed the predictor envelope[5] which applies the envelope method to the predictor space, , where is stochastic with mean and covariance matrix . In 2015, Cook et al. proposed the reduced-rank envelope model,[6] which combines the strength of reduced-rank regression and envelope model. Cook and Zhang[7] built the envelope model to generalized linear models such as weighted least squares, Cox regression, discriminant analysis, logistic regression, and Poisson regression.
The envelope model can also be applied to other contexts, such as variable selection, Bayesian inference, and high-dimensional data analysis. In 2016, Su et al. introduced the sparse envelope model[8] which performs variable selection of the response variables, allowing us to identify the response variables for which the regression coefficients are zero. In 2017, Khare et al. developed the Bayesian envelope model[9] which allows us to incorporate prior information of the parameters into the analysis. The Bayesian envelope model is also applicable where the sample size is smaller than the number of responses.
Implementation of envelopes
The Matlab toolbox envlp and R package Renvlp implement a variety of envelope estimators under the framework of multivariate linear regression, including the envelope model, the partial envelope model, and the predictor envelope. The capabilities of this toolbox include estimation of the model parameters, as well as performing standard multivariate inference in the context of envelope models; for example, hypothesis tests, and bootstrap. Examples and datasets are contained in the toolbox to illustrate the use of each model.[10]
See also
- Bivariate analysis
- Dimension reduction
- Functional data analysis
- Generalized linear model
- Linear discriminant analysis
- Maximum likelihood estimation
- Multivariate analysis
- Multivariate linear regression
- Ordinary least squares
- Partial least squares regression
- Regression analysis
- Statistical interference
- Sufficient dimension reduction
References
- ^ a b c d e Cook, R.D.; Li, B.; Chiaromonte,F. (2010). "Envelope Models for parsimonious and Efficient Multivariate Linear Regression". Statistica Sinica. 20 (3): 927–960. CiteSeerX 10.1.1.362.5659. JSTOR 2240725.
- ^ Cook, R.D.; Forzani, L.; Su, Z. (2016). "A Note on Fast Envelope Estimation". Journal of Multivariate Analysis. 150: 42–54.
- ^ a b c d Cook, R.D. An Introduction to envelope Models and Methods
- ^ Su, Z.; Cook, R.D. (2011). "Partial Envelopes for Efficient Estimation in Multivariate Linear Regression". Biometrika. 98 (1): 133–146.
- ^ Cook, R.D.; Helland, I.S.; Su, Z. (2013). "Envelopes and Partial Least Squares Regression". Journal of the Royal Statistical Society: Series B (Statistical Methodology). 75 (5): 851–877.
- ^ Cook, R.D.; Forzani, L.; Zhang, X. (2015). "Envelopes and reduced-rank regression". Biometrika. 102 (2): 439–456.
- ^ Cook, R.D.; Zhang, X. (2015). "Foundations for envelope models and methods". Journal of the American Statistical Association. 110 (510): 599–611.
- ^ Su, Z.; Zhu, G.; Chen, X.; Yang, X. (2016). "Sparse Envelope Model: Efficient Estimation and Response Variable Selection in Multivariate Linear Regression". Biometrika. 103 (3): 579–593.
- ^ Khare, K.; Pal, S.; Su,Z. (2017). "A Bayesian Approach for Envelope Models". The Annals of Statistics. 45 (1): 196–222.
- ^ Cook, R.D.; Su, Z.; Yang, Y. (2015). "envlp: A MATLAB Toolbox for Computing Envelope Estimators in Multivariate Linear Regression". Journal of Statistical Software. 62 (1): 1–20.
This article incorporates text by R. Dennis Cook, Zhihua Su, Yi Yang available under the CC BY 3.0 license.