Munich AI Lectures - Ludovic Righetti - Learning complex behaviors with nonlinear MPC

Nonlinear model predictive control (MPC) is a reliable technology to generate a variety of robotic behaviors, from flying robots to humanoids. While MPC is a rigorous framework to generate, in principle, any kind of behavior from a single algorithm, major limitations remain. For example, current approaches do not allow easy inclusion of multi-modal sensing, especially visual and force feedback, and algorithms struggle to optimize in real-time multi-contact behaviors necessary for complex manipulation or locomotion. On the other hand, learning-based methodologies, which heavily rely on offline compute, do not seem to struggle with these issues.

In this talk, I will present our recent work tackling those problems with a particular eye towards unifying learning and numerical optimal control. First, I will argue for the benefits of “textbook” numerical optimization methods to develop reliable solvers. Then I will discuss how to include multi-modal sensing and accelerate the computation of multi-contact behaviors through a mixture of offline compute (learning) and online optimization (MPC). I will then show how this can lead to improved performance for movement generation in the context of locomotion and manipulation and discuss on-going challenges.