One of the most widely applied paradigms for training deep neural networks is the stochastic first-order method (S-FOM). While deterministic first-order methods have been mostly recognized to be dimension-free, many S-FOMs, such as the stochastic gradient descent (SGD), incur complexities that deteriorate in the problem dimensionality. As a result, for increasingly common high-dimensional applications of AI models with millions (or even billions) of fitting parameters, their training process is growingly costly. In addressing the issue of high dimensionality, we present a novel dimension-free stochastic first-order method (DF-FOM) for both convex and non-convex stochastic optimization. We prove that the DF-FOM achieves iteration and sample complexities that are independent of the problem dimensionality. This algorithm and its extensions are directly applicable to training deep neural networks. Our numerical results are consistent with our theoretical findings.
Dr. Hongcheng Liu is an Assistant Professor of Industrial and Systems Engineering at the University of Florida (UF). His research interest lies in stochastic optimization, high-dimensional statistical and machine learning, and their engineering applications. He received his Ph.D. in 2015 from the Pennsylvania State University. Prior to joining the UF, he worked as a postdoctoral researcher in Radiation Oncology at Stanford University between 2015 and 2017.