CS7DS2 – Optimisation Algorithms for Machine Learning

Module CodeCS7DS2
Module NameOptimisation Algorithms for Machine Learning
ECTS Weighting [1]5 ECTS
Semester TaughtSemester 2
Module Coordinator/s  Professor Douglas Leith

Module Learning Outcomes

On successful completion of this module, students will be able to:

  1. Identify optimisation problems and their applications in data analysis and machine learning;
  2. Evaluate and design practical algorithms for solving machine learning optimisation problems;
  3. Compare different algorithms in terms of speed, complexity and scalability.

Module Content

The “learning” part of supervised “machine learning” almost always involves optimisation e.g. selecting model parameters to minimise a measure of prediction error. Commonly, some variant of gradient descent is used for this learning/training step, often in a “black box” fashion with standard/default settings.

In this module you’ll look inside the “black box” and learn how popular optimisation algorithms really work, their key properties (speed of convergence, scalability to big data etc) and the design choices/trade-offs involved. The module will take a hands-on approach and aims to develop practical understanding of the how to select and use optimisation algorithms for learning/training in supervised machine learning.

Topics that we’ll cover include:

  1. Popular gradient descent algorithms for ML: Adam, Adagrad, RMSprop, Polyak Momentum, Nesterov Acceleration;
  2. Use of approximation/sampling for better scalability: stochastic gradient descent, mini-batch methods;
  3. How to include constraints: projected gradient descent, penalty methods, Frank-Wolfe;
  4. Gradient-free algorithms e.g. for hyperparameter optimisation.

Teaching and Learning Methods

Lectures and coursework.

Assessment Details

Assessment ComponentBrief DescriptionLearning Outcomes Addressed% of TotalWeek SetWeek Due
Individual ProjectAssignment applying optimisation methods.LO1 – LO3 60%Week 8TBC
CourseworkWeekly assignmentsLO1 – LO340%Week 1TBC

Reassessment Details

Assignment (100%).

Contact Hours and Indicative Student Workload

Contact Hours (scheduled hours per student over full module), broken down by:22 hours
Lecture22 hours
Laboratory0 hours
Independent Study (outside scheduled contact hours), broken down by:94 hours
Preparation for classes and review of material (including preparation for examination, if applicable)44 hours
Completion of assessments (including examination, if applicable)50 hours
Total Hours116 hours

Module Pre-requisites

Prerequisite modules:

CS7CS4/CSU44061: Introduction to Machine Learning (or equivalent). Familiarity with programming in python. Basic algebra (but this does not aim to be a maths module, we’ll focus on a hands-on approach).

Module Co-requisites

N/A

Module Website

Blackboard

[1] TEP Glossary

[2] TEP Guidelines on Workload and Assessment