Software Engineering for Machine Learning Systems
Module aims
We'll cover the engineering concepts required to build robust and trustworthy systems that make use of machine learning. We'll look at all aspects of systems, from data ingestion to user experience, while considering the influence of regulation and wider society. We'll spend half of our time in lectures, and half in the lab. Over the course of the module, the lab and coursework we will implement, and reliably operate, a simplified machine learning based system in a simulated environment inspired by a real-world problem.
We will not cover the design of machine learning models themselves, we’ll focus on the systems that surround and support them.
Learning outcomes
Upon successful completion of this module you will be able to:
* Train a machine learning model, and integrate it into a user facing application.
* Build a system that is robust in the presence of common data-centric failure modes.
* Safely evolve systems in response to changing requirements.
* Make reasoned design choices for systems handling sensitive data.
* Adapt technical designs to regulation and the need to maintain societal trust.
* Critique and assess elements of user experience design related to ML systems.
Module syllabus
We'll cover the engineering theory and skills specific to machine learning systems, including:
* The differences between software systems that make use of machine learning, and those that don’t.
* Architectures for training models, and the influences on them.
* Architectures for model inference, and the influences on them.
* Monitoring and reliability in machine learning systems.
* Evolving a running machine learning system.
* The social context of machine learning, and its influence on system design - including frameworks for responsible engineering and privacy preserving technologies.
* The legal context of machine learning, and its influence on system design - including GDPR and ISO-13485.
* The impact of user experience design on the underlying system design.
Teaching methods
Over the course of the module, you will use simulated data and machine learning to build an end-to-end system based on a real-world problem. Each week we’ll use lectures to introduce a new aspect of machine learning system design, and implement that aspect within our evolving system. We’ll introduce the implementation in a lab session, and expect it to be completed as coursework. While initially individual, coursework will transition to groups of 3 to 4. We’ll plan 2hrs of classroom time, and 2hrs of labs each week. We expect a typical group to spend half a day a week on coursework.
An online service will be used as a discussion forum for the module.
Assessments
The module will be assessed through coursework, including:
* Implementing the end-to-end machine learning system. Part of the assessment will be automatic, based on the system’s response to a simulated environment, part will be based on a human view of engineering quality, as described during lectures.
* Operating the system continually and reliably throughout the second-half of the module, in the presence of evolving conditions. Part of the assessment will be automatic, based on the system’s ability to meet reliability metrics, part of the assessment will be based on a human view of engineering practice, as described during lectures.
* Communicating technical design decisions to an engineering audience through documentation, as described during lectures.
* Communicating the implication of technical decisions to a non-engineering, domain-expert, audience through documentation, as described during lectures.
The course is assessed fully through coursework - there will be no written exam. The in-class discussions and lab sessions will be largely formative in nature, so you will have the opportunity to ask questions and get feedback on your ideas outside of the formal, summative, assessment.
* Implementations and documentation produced during labs, and for coursework, will be assessed with feedback delivered either orally or in writing.
* Quality metrics will be computed automatically for certain aspects of system implementation, for example, the ratio of events successfully detected in the simulated environment. These metrics will be both shared with you and used for assessment.
* Some implementation choices will be explored as a group through structured design reviews.