CSU44062 – 4CSLL5 Advanced Computational Linguistics

Module Code	CSU44062
Module Name	Advanced Computational Linguistics
ECTS Weighting [1]	5 ECTS
Semester Taught	Semester 1
Module Coordinator/s	Dr Martin Emms

Module Learning Outcomes

When students have successfully completed this module they should be able to:

Understand in general what a probabilistic model is, the distinction between so-called visible and hidden variables,and the distinctive nature of models where each datum is a sequence of varying length, rather then a fixed-size set of features;
Understand the general idea of unsupervised training as way to set model parameters concerning hidden variables from evidence only on visible variables;
Describe the Expectation Maximisation (EM) as a general unsupervised technique, make an implementation & identify the crucial steps in proofs of its convergence and property of increasing data likelihood;
Describe specific instances of this via the models and training algorithms in Machine Translation and Speech Recognition including being able to explain the details of how seemingly infeasibly costly calculations can in fact be feasibly done;
Understand in general the notion of a probabilistic language model and more specifically several different approaches to these such as n-gram models and Neural Network language models
The aim is to give a grounding in a range of probablistic machine learning techniques, giving some prominence to so-called unsupervised techniques. Many of the techniques though they have been found useful lin anguage-processing technologies, have also uses beyond that, and students will gain some insight into this also.

Module Content

Course is about how to have machines learn from data, with emphasising techniques which can be applied in
language-related areas, such as Machine Translation, Speech Recognition and Topic Modeling,
though the techniques themselves are applicable in areas unrelated to language. There is large emphasis on unsupervised machine learning, possibly an aspect distinguishing this from other modules concerning machine learning. Another trait is the concern with data items of arbitrary length.

Below an enumeration of topic covered:

Probability for variables with discrete outcomes : joint, marginal and conditional probalities; chain rule; conditional independence. Fundamentals of ‘inferring’ a value of an unknown variable using other known variables. Classical n-gram models of word sequences of arbitrary length
Idea of machine learning as parameter estimation to maximise likelihood of training data. Classic supervised parameter estimation with all model variables visible in data; illustrations/proofs that intuitive relative frequency approaches are maximum likelihood estimators. Contrasting case of unsupervised parameter estimation, with some model variables hidden in data. Details of how seemingly impossible problem of finding likelihood maximising estimate in this case is solved by the famous EM algorithm
Statistical Machine Translation: general (source|target) x target formulation and learning from corpus of sentence pairs; idea of ‘hidden’ alignment variables between sentence pairs; the so-called IBM alignment models; brute-force EM for learning alignment models then efficient exact algorithms avoiding the exponential cost of brute-force EM;
Generalisation to so-called ‘Phrase-based SMT’. Dealing with the algorithmic challenges of ‘decoding’ ie. finding a best solution;
Speech Recognition: general Hidden Markov Model (O|S) x S formulation where O is observable speech, and S is hidden state sequence. Brute-force EM for learning HMM parameters from corpus of observed speech; Baum-Welch algorithm‘s surprising feat of avoiding, without approximation, the exponential cost of brute-force EM;
Topic Modelling: related technique seeing documents as involving hidden/latent ‘topic’ variables, how learnt ‘topic’ parameters of such models can possibly assist in the navigation of huge document collections.
Neural Network (NN) Language Models: classic NN as mapping between fixed-length number (or ‘node’) sequences, via intermediate node sequences computed by passing of node-specific weighted sums through differentiable ‘activation’ functions. Learning as tuning towards target outputs by weight adjustments proportional to the gradient of the error wrt the weight; these gradients’ surprisingly efficient computation (via back-propagation), no matter how remote/deep the affected node. Reflections on whether nodes are ‘hidden variables’. Then key innovations that bridge gap between classic NNs and languag (i) word inputs handled by (learnable) mappings from words to /number/ sequences (‘word-embeddings’) (ii) word outputs via differentiable /’soft-max’ parameterisation of word probabilities/ (iii) actual next word as /repeated training target for learnt next word probabilities/ (iv) variable length input via either /recurrent/ NNs, or /transformer/ NNs with ‘attention’ and position embeddings
Outline of such components in ‘end-to-end’ machine translation
In each case, alongside the explanation of the algorithms, there will be practical work, either developing instances of them, or deploying existing implementations and running them on data sets to concretely see their properties.

Teaching and Learning Methods

There is a mixture of lectures, tutorials and lab sessions. Most frequently there will be a 2 lectures per week, but there will be occasions where 1 of the timetabled lecture sessions will actually be a lab-session or a tutorial. This may happen in anticipation of the setting of a course work assignment.

In some cases there will be alternate forms of an assignment, one form more mathematical requiring ‘pen-and-pencil’ calculations, the other more implementational requiring the implementation of some (or part of some) algorithm.

There will be many further exercises in online materials, all of which students will be encouraged to attempt. To all of the exercises suggested answers will be provided some time after the exercise has been first made available

Assessment Details

Assessment Component	Brief Description	Learning Outcomes Addressed	% of Total	Week Set	Week Due
Coursework 1	Probabilistic Models and Inference	LO1, LO6	6%	Week 2	Week 3
Coursework 2	EM in a Coin-tossing Scenario	LO1, LO2, LO3	6%	Week 5	Week 6
Coursework 3	IBM Model Training	LO1, LO2, LO3, LO4	8%	Week 9	Week 10
Examination	In Person Exam	LO1, LO2, LO3, LO4, LO5, LO6	80%	N/A	N/A

The breakdown of Coursework into individual assignments summarises the previous year; it should be treated as an indicator of what will happen this year rather than an exact schedule

Reassessment Details

In Person Exam, 2 hours (100%).

Contact Hours and Indicative Student Workload

Contact Hours (scheduled hours per student over full module), broken down by:	25 hours
Lecture	22 hours
Laboratory	3 hours

Other	0 hours
Independent Study (outside scheduled contact hours), broken down by:	77 hours
Preparation for classes and review of material (including preparation for examination, if applicable)	37 hours
Completion of assessments (including examination, if applicable)	40 hours
Total Hours	102 hours

Module Pre-requisites

Prerequisite modules: N/A

Other/alternative non-module prerequisites: The module is self-contained. To be able to do some variants of assignments, ability to program is required. As noted above, there will be an alternative non-programming more maths-based version of any assignment involving programming.

Module Co-requisites

N/A

Module Website

For lecture materials the location will be www.scss.tcd.ie/Martin.Emms/4062

For handling assignments use will be made of a Blackboard module.