IPC Lab Seminars | Stefan Vlaski – Learning over Graphs – Beyond Consensus and Convexity

We describe two recent directions in the study learning algorithms over graphs. First, we will deviate from the wide-spread “consensus optimization” setting, where agents are forced to agreement on a common model, resulting in poor performance in heterogeneous settings. We will show how different task-relatedness models give rise to a family of multi-task learning algorithms over graphs, which allow for improved learning performance without the need to force consensus. We will then show how, in the absence of a task-relatedness model, multi-task learning over graphs is still possible via a decentralized variant of model-agnostic meta-learning (MAML).

The second part will study the impact of the loss function chosen to quantify model fit. The dynamics of decentralized learning algorithms with convex loss functions is fairly well understood, yet performance guarantees in non-convex environments have long remained elusive. This is due to the fact that non-convex loss functions can be riddled with local minima and saddle-points, where the gradient vanishes (and hence gradient descent stagnates), while performance can be arbitrarily poor. This is in contrast to the empirical success of deep learning, which gives rise to non-convex loss surfaces, suggesting that stochastic gradient descent, as implemented via backpropagation, avoids saddle-points. We review recent results shedding light on these dynamics and show how decentralized algorithms can continue to match centralized ones, even when it comes to evading saddle-points.