Graph-based estimation and inference for high-dimensional data

Abstract

High-dimensional data are becoming increasingly influential and are of considerable interest within the statistical community. Existing tools may be inadequate when confronted with high-dimensional data or lack theoretical support when the dimension increases. The graph-based approach is a framework that uses the similarity graph as input to address various statistical problems. We develop graph-based statistics to target fundamental statistics problem. Due to its flexibility and power, these graph-theoretic statistics are well suited for high-dimensional settings across various problems.

This dissertation includes three studies. The first study focuses on the two-sample test problem. A robust graph-based test is introduced that could overcome the effect of hubness, which is defined as a node in the similarity graph with a large node degree, presented in high-dimensional data. The second study explores the estimation of the number of clusters. Choosing the number of clusters could be challenging for high-dimensional data since the existing methods usually utilize the within or between cluster dispersion, which may be inefficient in high-dimension. We proposed a graph-based method to choose the number of clusters based on the true densities. The consistency of the estimated number of clusters is provided. The third study investigates testing the random effects in the mixed effects model. Without estimation of any parameters, a graph-based method is proposed that can test whether the random effect is zero in the mixed model with high-dimensional fixed effects.