70067

Robot Learning

Module aims

Robot Learning is an exciting new field, which studies how physical robots can learn skills using machine learning techniques, and can be seen as an "advanced reinforcement learning" module. The module first motivates the need for robot learning, by describing classical robot control methods and their limitations. Then, the module explains how reinforcement learning can be applied to physical robots acting in the real world. Finally, the module explores how robots can learn new skills by observing and interacting with humans. Lab sessions and courseworks teach students how to implement these methods in Python for a simulated robot learning to solve tasks, which culminates in a fun live competition in the final lecture.

The module assumes knowledge of the Reinforcement Learning module in the previous term, so taking Reinforcement Learning is strongly recommended, unless students have already taken a similar module elsewhere. For example, the module assumes familiarity with Markov decision processes, Q-learning, and deep reinforcement learning, although these will be briefly recapped

Learning outcomes

At the end of the module, students should be able to:

1. Explain the need for robot learning with respect to classical control methods.
2. Describe the design choices for a given robot learning problem.
3. Compare the strengths and weaknesses of different robot learning strategies.
4. Analyse a given robot learning algorithm for a given task, and hypothesise about its likely performance.
5. Explain the roles of various mathematical formulae within a given robot learning algorithm.
6. Design and implement a robot learning strategy in Python for a given task, and evaluate its performance.   

Module syllabus

1. Specification of robot learning problems: states, observations, actions, rewards, and policies.
2. Analytical methods: kinematics and dynamics models, planning and optimisation, and classical control.
3. Model-based reinforcement learning: model learning, exploration, and integration of planning and learning.
4. Imitation learning: behavioural cloning, inverse reinforcement learning, and learning interactively with humans.
5. Advanced topics: state-of-the-art research, open problems, and future directions (non-examinable).

Pre-requisites

Teaching methods

The module consists of 7 weeks of timetabled sessions. Each week consists of a 2-hour lecture, and a 2-hour computer lab session. Each lecture involves a presentation using slides describing the core content, together with discussions in breakout groups, and small quizzes. Each lab session involves a written tutorial with exercises in Python, associated with the lecture of that week. These exercises are unassessed, but help students to understand the lecture material and to complete the courseworks. Labs are supported by a number of TAs, and an online service will be used as a discussion forum for the module.   

.An online service will be used as a discussion forum for the module. 

Assessments

The module is assessed by two courseworks, each contributing 15% of the module's grade, and a written exam, contributing 70% of the module's grade. The first coursework requires students to answer a number of questions, which are based on the Python exercises done during the lab sessions. The second coursework requires students to design and implement their own robot learning strategy for a simulated robot, with freedom to explore a range of methods taught in the module. The second coursework culminates in a live competition. Both courseworks are done and submitted individually.

Each lecture involves quizzes on the content of that lecture, after which answers are provided. As well as the coursework grade, each student will receive written feedback on each coursework submission, following a marking scheme set by the module leader. This feedback explains where students lost marks, and how they could have obtained those marks. General feedback on the courseworks will also be provided during the lectures. The coursework content overlaps with the exam content, so this feedback is provided before the exam to help students prepare for the exam.

Module leaders