High dimensional covariance estimation provides accessible and comprehensive coverage of the classical and modern approaches for estimating covariance matrices as well as their applications to the rapidly developing areas lying at the intersection of statistics and machine learning. Highdimensional covariance matrix estimation in approximate. High dimensional covariance estimation provides accessible and comprehensive coverage of. Estimating a covariance matrix or a dispersion matrix is a fundamental problem in statistical signal processing. Robust estimation of highdimensional covariance and. High dimensional covariance estimation by minimizing l1. Methods for estimating sparse and large covariance matrices covariance and correlation matrices play fundamental roles in every aspect of the analysis of multivariate data collected from a variety of fields including business and economics, health care, engineering, and environmental and physical sciences. Estimating and testing highdimensional mediation effects in epigenetic studies. Popular regularization methods of directly exploiting sparsity are not directly applicable to many financial problems. Highdimensional covariance estimation mohsen pourahmadi. High dimensional covariance matrix estimation ming yuan department of statistics university of wisconsinmadison and morgridge institute for research. This implies that the choice to take the sample covariance as the pilot estimator. Highdimensional covariance estimation provides accessible and comprehensive coverage of the classical and modern approaches for estimating covariance. Classical methods of estimating the covariance matrices are based on the strict factor models, assuming independent idiosyncratic.
Covariance estimation for high dimensional data vectors. High dimensional data are often most plausibly generated from distributions with complex structure and leptokurtosis in some or all components. Systems in the university of michigan 2011 doctoral committee. May 21, 2011 the variance covariance matrix plays a central role in the inferential theories of high dimensional factor models in finance and economics. High dimensional sparse inverse covariance estimation using greedy methods recent resurgence of greedy methods. Highdimensional covariance estimation can be classified into two main categories, one. Tony cai department of statistics, the wharton school, university of pennsylvania, philadelphia, pa 19104, usa email.
Highdimensional covariance estimation by minimizing. Estimating high dimensional covariance matrices is intrinsically challenging. Sisnota good estimator of in fact, i for n estimating high dimensional covariance matrices 201 2. In particular, when p 200 there are more than 20,000 parameters in the covariance matrix.
In the following, nis referred to as the number of variables, or the number. High dimensional inverse covariance matrix estimation via. High dimensional low rank and sparse covariance matrix estimation via convex minimization. Robust shrinkage estimation of highdimensional covariance.
Estimating structured high dimensional covariance and precision matrices. Testing and estimating changepoints in the covariance matrix. Why we need a sparse estimation of a covariance matrix. Estimating covariance matrices is an important part of portfolio selection, risk management, and asset pricing. Even if p software that scale up linearly with the number of observations per function. High dimensional low rank and sparse covariance matrix. High dimensional covariance estimation by minimizing l1penalized logdeterminant divergence pradeep ravikumar, martin wainwright, bin yu, garvesh raskutti abstract. Rp, we study the problem of estimating both its covariance matrix, and its inverse covariance or concentration matrix. Matlab software for disciplined convex programming, version 2. When the dimension of the covariance matrix is large, the estimation problem. Minimax rates of convergence for estimating several classes of structured covariance and precision matrices, including bandable, toeplitz, and sparse covariance matrices as well as sparse precision matrices, are given under the spectral norm loss.
Many techniques for detection and estimation rely on accurate estimation of the true covariance. The limitations of the sample covariance matrix are discussed. This paper presents a new method for estimating high dimensional covariance matrices. Covariance estimation for high dimensional data vectors using. Jul 26, 20 high dimensional covariance estimation provides accessible and comprehensive coverage of the classical and modern approaches for estimating covariance matrices as well as their applications to the rapidly developing areas lying at the intersection of statistics and machine learning. Wainwright, garvesh raskutti, and bin yu more by pradeep ravikumar. Fast covariance estimation for highdimensional functional.
Rejoinder of estimating structured highdimensional covariance and precision matrices. Another relation can be made to the method by rutimann. Highdimensional data are often most plausibly generated from distributions with. Jan 01, 2011 however, the sample covariance matrix is an inappropriate estimator in high dimensional settings. Covariance estimation for high dimensional data vectors using the sparse matrix transform guangzhi cao and charles a. Asymptotic distributions of highdimensional nonparametric inference with distance correlation. Highdimensional sparse inverse covariance estimation using. With highdimensional data wiley series in probability and statistics kindle edition by pourahmadi, mohsen. Estimating high dimensional covariance matrices and its. The method, permuted rankpenalized leastsquares prls, is based on a. In this paper, we propose a maximum likelihood ml approach to covariance estimation, which employs a novel sparsity constraint. Rp, estimate both its covariance matrix, and its inverse covariance or concentration matrix. The abundance of high dimensional data is one reason for the interest in the problem. Minimax rates of convergence for estimating several classes of.
Highdimensional covariance estimation provides accessible and comprehensive coverage of the classical and modern approaches for estimating covariance matrices as well as their applications to the rapidly developing areas lying at the intersection of statistics and machine learning. Most available methods and software cannot smooth covariance matrices of dimension j \times j with j500. Sparse estimation of highdimensional covariance matrices. High dimensional covariance matrix estimation by penalizing. Perhaps the most natural candidate for estimating is the empirical sample covariance matrix, but this is known to behave poorly in highdimensional settings. The assumed framework allows for a large class of multivariate linear processes including vector autoregressive moving average varma models of growing dimension and spiked covariance models.
Highdimensional covariance estimation researchgate. Regularized estimation of high dimensional covariance matrices by yilun chen a dissertation submitted in partial ful llment of the requirements for the degree of doctor of philosophy electrical engineering. Fast covariance estimation for highdimensional functional data. When n high dimensional covariance estimation provides accessible and comprehensive coverage of the classical and modern approaches for estimating covariance matrices as well as their applications to the rapidly developing areas lying at the intersection of statistics and machine learning. An overview on the estimation of large covariance and. Robust estimation of highdimensional covariance and precision. In recent years, estimating a high dimensional p pcovariance matrix under small sample size nhas attracted considerable attention. Bouman school of electrical and computer engineering purdue university west lafayette, in 47907 april 29, 2008 1 introduction many problems in statistical pattern recognition and analysis require the classi cation and. For example, in portfolio allocation and risk management, the number of stocks p, which is typically of the same order as the sample size n, can well be in the order of hundreds. Minimax rates of convergence for estimating several classes of structured covariance and precision matrices, including bandable, toeplitz, sparse, and sparse spiked covariance matrices as well as.
Covariance and precision matrices provide a useful summary of such structure, yet the performance of popular matrix estimators typically hinges upon a subgaussianity assumption. Estimation of large dimensional sparse covariance matrices. Covariance estimation for high dimensional vectors is a classically dif. This paper studies methods for testing and estimating changepoints in the covariance structure of a high dimensional linear time series. Estimation of large covariance matrices, particularly in situations where the data dimension p is comparable to or larger than the sample size n, has attracted a lot of attention recently. Focusing on methodology and computation more than on theorems and proofs, this book provides computationally feasible and statistically efficient methods for estimating sparse and large covariance matrices of high dimensional data. Robust estimation of high dimensional covariance and precision matrices by marcoavellamedina sloan school of management, massachusetts institute oftechnology, 30 memorial drive, cambridge, massachusetts 02142, u. Highdimensional covariance estimation based on gaussian. Download it once and read it on your kindle device, pc, phones or tablets. These perform simple forward steps adding parameters greedily, and possibly also backward steps removing parameters greed. In our later simulations and applications, the tolerance value is set to be 0. In this paper, we propose a maximum likelihood ml approach to covariance estimation, which employs a. Covariance estimation in high dimensions via kronecker.
Battey department of mathematics, imperial college london, 545 huxley building. Pdf highdimensional covariance matrix estimation in. Instead, we introduce robust pilot estimators in 4 that satisfy the conditions 1 and 2. High dimensional covariance matrix estimation using a factor. We study high dimensional covariance precision matrix estimation under the assumption that the covariance precision matrix can be decomposed into a lowrank component l and a diagonal component d. Estimating structured highdimensional covariance and. Highdimensional data are often most plausibly generated from distributions with complex structure and leptokurtosis in some or all components.
For example, x itcan be the return for asset iin period t, t 1. Wolf, a wellconditioned estimator for largedimensional covariance matrices, journal of multivariate analysis, volume 88, issue 2, february 2004, pages 365. Robust covariance estimation for highdimensional compositional data with application to microbial communities analysis nasaads microbial communities analysis is drawing growing attention due to the rapid development of highthroughput sequencing techniques nowadays. High dimensional inverse covariance matrix estimation via linear programming. Xi luo brown university november 7, 2011 abstract this paper introduces a general framework of covariance structures that can be veri. A related line of recent work on learning sparse models has focused on \stagewise greedy algorithms. The key requirement on for optimal covariance estimation is that. Regularized estimation of highdimensional covariance matrices. Download citation highdimensional covariance estimation penalized regression methods for inducing sparsity in the precision matrix have grown rapidly in. Most available methods and software cannot smooth covariance matrices of dimension j 500.
295 340 855 1053 1048 1365 490 1401 640 706 1099 666 948 97 908 309 1314 302 34 782 976 799 924 1513 1056 1228 1320 1431 905 508 972 843 1306 587 992 247 531 486 132 536 451 2 1345 813