Title
The bilevel optimization renaissance through machine learning: lessons and challenges
Abstract
Bilevel optimization has been part of machine learning for over 4 decades now, although perhaps not
always in an obvious way. The interconnection between the two topics started appearing more clearly
in publications since about 20 years now, and in the last 10 years, the number of machine learning
applications of bilevel optimization has literally exploded. This rise of bilevel optimization in machine
learning has been highly positive, as it has come with many innovations in the theoretical and numerical
perspectives in understanding and solving the problem, especially with the rebirth of the implicit
function approach, which seemed to have been abandoned at some point.
Overall, machine learning has set the bar very high for the whole field of bilevel optimization with
regards to the development of numerical methods and the associated convergence theory, as well as
the introduction of efficient tools to speed up components such as derivative calculations among other
things. However, it remains unclear how the techniques from the machine learning literature can be
extended to other applications of bilevel programming. For instance, many machine learning loss
functions and the special problem structures enable the fulfilment of some qualification conditions that
will fail for multiple other applications of bilevel optimization.
We will start this talk with the definition of a bilevel optimization and some applications in economics.
We will then provide an overview of machine learning applications of bilevel optimization, while giving
a flavor of corresponding solution algorithms and their limitations. Subsequently, we will discuss
possible paths for algorithms that can tackle more complicated machine learning applications of bilevel
optimization, while also highlighting lessons that can be learned for more general bilevel programs.
Bio
Alain is a full professor of mathematical optimization at the School of Mathematical Sciences within the University of Southampton where he is affiliated to the OR Group and CORMSIS. Prior to joining Southampton, he was a research fellow at the University of Birmingham and had previously worked as a research associate at the Technical University of Freiberg. Alain is currently an Alexander von Humboldt Experienced Fellow 2024-2026 with the Karlsruhe Institute of Technology in Germany, and is also a fellow of both the Institute of Mathematics & Its Applications and the Higher Education Academy. He previously served as a fellow of the Alan Turing Institute for Data Science and Artificial Intelligence for around 5 years.
Alain is broadly interested in nonconvex and nonsmooth continuous optimization problems. More specifically, his primary research interests has so far revolved around optimization problems with a hierarchical structure; in particular, he has worked extensively on two-level optimization problems commonly known as bilevel programming problems. More recently, he has developed interest in machine learning modelling and theory, as well optimization related algorithms.