Miercuri (Wednesday), 24.07.2024, ora 11, sala 804 (Ciprian Foias) Institutul de Matematica „Simion Stoilow” al Academiei Romane
Prof. Arnulf Jentzen, The Chinese University of Hong Kong, Shenzhen (CUHK-Shenzhen) & University of Münster will give the following talk:
Convergence and non-convergence results for accelerated and adaptive stochastic gradient descent optimization methods
Deep learning algorithms – typically consisting of a class of deep neural networks trained by a stochastic gradient descent (SGD) optimization method – are nowadays the key ingredients in many artificial intelligence (AI) systems and have revolutionized our ways of working and living in modern societies. In practical relevant learning problems, usually not the plain vanilla standard SGD optimization method is used for the training of ANNs but instead more sophisticated suitably accelerated and adapted SGD optimization methods are employed. As of today, maybe the most popular variant of such accelerated and adaptive SGD optimization methods is the famous Adam optimizer proposed by Kingma & Ba in 2014. Despite the popularity of the Adam optimizer in implementations, it remained an open problem of research to provide a convergence analysis for the Adam optimizer even in the situation of simple quadratic stochastic optimization problems where the objective function (the function one intends to minimize) is strongly convex. In this talk we present optimal convergence rates for the Adam optimizer for a large class of stochastic optimization problems, in particular, covering simple quadratic stochastic optimization problems. Moreover, in ANN training scenarios we prove that the Adam optimizer and a large class of other SGD optimization methods do with high probability not converge to global minimizers in the optimization landscape. The talk is based on joint works with Steffen Dereich (University of Münster) and Adrian Riekert (University of Münster).
Short bio:
Brief bio:
Arnulf Jentzen (*November 1983) is appointed as a presidential chair professor at the Chinese University of Hong Kong, Shenzhen (since 2021) and as a full professor at the University of Münster (since 2019). In 2004 he started his undergraduate studies in mathematics at Goethe University Frankfurt in Germany, in 2007 he received his diploma degree at this university, and in 2009 he completed his PhD in mathematics at this university. The core research topics of his research group are machine learning approximation algorithms, computational stochastics, numerical analysis for high dimensional partial differential equations (PDEs), stochastic analysis, and computational finance. Currently, he serves in the editorial boards of several scientific journals such as the Annals of Applied Probability, the Journal of Machine Learning, the SIAM Journal on Scientific Computing, the SIAM Journal on Numerical Analysis, and the SIAM/ASA Journal on Uncertainty Quantification. His research activities has been recognized through several major awards such as the Felix Klein Prize of the European Mathematical Society (EMS) (2020), an ERC Consolidator Grant from the European Research Council (ERC) (2022), the Joseph F. Traub Prize for Achievement in Information-Based Complexity (2022), and a Frontier of Science Award in Mathematics (jointly with Jiequn Han and Weinan E) by the International Congress of Basic Science (ICBS) (2024). Further details on the activities of his research group can be found at the webpage http://www.ajentzen.de.
(prezentare în cadrul Seminarului de Probabilitati si statistica)