# Machine Learning, Spring 2019

#### Jacobs University Bremen, Spring 2019, Herbert Jaeger

**Class
sessions:** Mondays 8:15-9:45 (Lecture Hall Res. III) and
Wednesdays 8:15-9:45 (Lecture Hall Res. III)

**Tutorial sessions**: Tuesdays 17:15-18:30, West Hall 4.

**TAs**: Steven Abreu (s.abreu at
jacobs-university.de) and Tianlin Liu (t.liu at jacobs-university.de)

**Course
description. **Machine learning (ML) is all about algorithms
which are fed with (large quantities of) real-world data, and which
return a compressed model' of the data. An example is
the world model' of a robot: the input data are sensor data
streams,
from which the robot learns a model of its environment -- needed, for
instance, for navigation. Another example is a spoken language model:
the input data are speech recordings, from which ML methods build a
model of spoken English -- useful, for instance, in automated speech
recognition systems. There is a large number of formalisms in which
such
models can be cast, and an equally large diversity of learning
algorithms. However, there is a relatively small number of fundamental
challenges which are common to all of these formalisms and algorithms:
most notably, the "curse of dimensionality'' and the almost
deadly-dangerous problem of under- vs. overfitting. This lecture
introduces such fundamental concepts and illustrates them with a
choice of elementary model formalisms (linear classifiers and
regressors, radial basis function networks, clustering, mixtures of
Gaussians, Parzen windows). Furthermore, the course also provides a
refresher of the requisite concepts from probability theory,
statistics, and linear algebra.

**Homework. **There will be two kinds of homeworks,
which are treated quite differently. **A. Paper-and-pencil
problems**.
These homeworks give an opportunity to exercise the theoretical
concepts introduced in the lecture. These homeworks will not be
checked or graded, and doing them is not mandatory. Instead, the
problems will be discussed and show-solved in weekly tutorial sessions
held by the TA. Model solutions will be put online a week after
issuing the problem
sheets. **B. Programming miniprojects.** The other type
of
homework comes in the form of small-sized machine learning
programming projects. Students work in teams of two, each team
submitting a single solution, by email to the TA, consisting of the
code and a documentation (typeset pdf document, preferably generated
in Latex, other word processing software allowed). These miniproject
homeworks will be graded. Programming can be done in Matlab or
Python.

**Grading and exams:** Grading and exams:
The final course grade will be composed from programming homeworks
(20%), quizzes (50%) and a final exam (30%). There will be three quizzes
(written in class, 30 minutes), the best two of which will each
account to 25% of the final grade (worst will be dropped). All quizzes and the final exam are open
book.

**Quiz makeup rules:** if a quiz is missed without excuse, it
will be graded with 0 points. One makeup will be offered soon after the quiz for
medically excused quizzes according to the Jacobs rules (especially,
the medical excuse must be announced to me *before* the quiz).
Non-medical excuses can be accepted and makeups be arranged on a
case-by-case basis. If the first makeup is likewise missed for medical reasons, similar rules apply to get admitted to a second makeup (medical excuse must be announced to me before the makeup). The second makeup is then to sit for the quiz in the next year's edition of this course; or the student may opt to get the grade of the final exam counted also as grade for the quiz.

**The 2018 final exam ** for your private study and preparation is here. And the solutions are here.

**Fully self-contained lecture notes** are
here
(version 1.11, last update Mar 5, change: typo in Appendix A (eqn. 74) corrected) .

**Schedule **(this
will be filled in synchrony with reality as we go along)

Feb 6 | Introduction |

Feb 11 | Introducing the TICS example. Continuous <-> discrete data transformations. Reading: Sections 2.1, 2.2 in the lecture notes |

Feb 13 | A quick recap of basic concepts from probability theory. Reading: Appendix A in the lecture notes Exercise sheet 1 | Solutions |

Feb 18 | The curse of dimensionality and the concept of manifolds in high-dimensional vector spaces. Reading: Section 2.3 |

Feb 20 | The field of ML: overview and navigation guide. Reading: Section 3 Exercise sheet 2 | Solutions |

Feb 25 | Basics of pattern classification. A look in passing at decision trees. Optimal decision boundaries. Reading: Section 4 of LNs |

Feb 27 | Dimension reduction through vector quantization: K-means clustering. Reading: LN Section 5.1 Exercise sheet 3 | Solutions | Miniproject 1 The first programming miniproject - this will be graded! |

Mar 4 | Principal Component Analysis - principle. Reading: LN Section 5.2 |

Mar 6 | PCA - mathematical properties, algorithm. Eigendigits etc. Reading: LN Section 5.3, 5.4, 5.5 Exercise sheet 4 (paper and pencil exercise, not to be returned, not graded) | Solutions ... and at noontime: first miniquiz. Time: 12:45-13:15 Location: CNLH |

Mar 11 | Linear regresssion, part 1. Reading: LN Section 6.1, 6.2 up to (including) Equation 19). |

Mar 13 | Linear regresssion, part 2. Reading: LN Section 6, complete. |

Mar 18 | A probability refresher: expectation, variance, covariance. -- Training and testing errors. Reading: LN Appendix D and LN Section 7.1 Exercise sheet 5 | Solutions |

Mar 20 | The problem of overfitting. Reading: LN Section 7.2 |

Mar 25 | Supervised learning: formal theory. Risk minimization through adapting model size. Reading: LN Sections 7.3, 7.4 |

Mar 27 | Cross-validation: the key to everybody's success in ML. Exercise sheet 6 | Solutions | Miniproject 2 The second programming miniproject - this will be graded! --- Reading: LN Section 7.5 ... and at noontime: second miniquiz. Time:
12:45-13:15 Location: CNLH |

Apr 1 | Using regularization to fight overfitting. Ridge regression. Reading: LN Sections 7.6, 7.7 |

Apr 3 | Why it's called the Bias-Variance Dilemma. Reading: LN Sections 7.8 Exercise 7 | Solutions |

Apr 8 | Neural networks: introduction. Historical forefather: the perceptron. No reading. Slides |

Apr 10 | A zoo of neural networks -- "connectionist", associative memory networks, Boltzmann machines, spiking. No reading. |

Apr 24 | The multilayer perceptron: architecture. Universal approximation property. Reading: LN Section 8.2 |

Apr 29 | Why deep is good. |

May 6 | Training MLPs: general scheme. Remarks on model optimization by gradient descent. Reading: LN Sections 8.3, 8.4 |

May 8 | The backpropagation algorithm. A wonderful webpage! *** Exercise (fun) Nr. 8 Reading: LN Section 8.5... and at noontime: third miniquiz. Time:
12:45-13:15 Location: CNLH |

May 13 | Introduction to recurrent neural networks. This material will not be queried in the final exam! |

May 15 | Reservoir computing. This material will not be queried in the final exam! |

May 23 | 9:00-11:00, SSC Hall 3 and 4: Final exam |

**References**

*The online lecture notes are
self-contained, and no further literature is necessary for this
course. However, if you want to study some topics in more depth, the
following are recommended references.*

Bishop, Christopher
M.: Neural Networks for Pattern Recognition (Oxford
Univ. Press, 1995.) IRC: QA76.87 .B574 1995* A recommendable basic
reference (beyond the online lecture notes) *

Bishop,
Christopher M.: Pattern Recognition and Machine Learning. Springer
Verlag, 2006 *Much
more up-to-date and comprehensive than the previously mentioned
Bishop book, but I dare say too thick and advanced for an
undergraduate course (730 pages) -- more like a handbook for
practicians. To find your way into ML, the older, slimmer Bishop book
will work better. *

Michie, D., Spiegelhalter, D.J.,
Taylor, C.C.: *Machine Learning, Neural and Statistical
Classification* (1994) Free and online at http://www.amsta.leeds.ac.uk/~charles/statlog/
and at the course
resource repository. *A transparently written book,
concentrating on classification. Good backup reading. Thanks to Mantas
for pointing this out!*

Duda, R.O., Hart, P.E., Stork,
D.G.: *Pattern Classification*, 2nd edition (John Wiley, 2001)
IRC: Q327 .D83 2001 *Covers more than the Bishop book, more
detailed and more mathematically oriented. Backup reference for the
deep probers*

T. Hastie, R. Tibshirani, J. Friedman,
*The Elements of Statistical Learning: Data Mining, Inference, and
Prediction*. Springer Verlag 2001. IRC: Q325.75 .H37
2001*
I have found this book only recently and haven't studied it in detail
–
looks extremely well written, combining (statistical) maths with
applications and principal methods of machine learning, full of
illuminating color graphics. May become my favourite. *

Farhang-Boroujeny, B.: *Adaptive Filters, Theory and
Applications* (John Wiley, 1999). IRC: TK7872.F5 F37 1998
*Some
initial portions of this book describe online linear filtering with
the
LMS algorithm, which will possibly be covered in the course*

Goodfellow, I., Bengio, Y., Courville, A.: Deep learning. MIT
Press, 2016. Legal online
version available. *The "bible" of deep
learning. *

Mitchell, Tom M.: *Machine
Learning* (McGraw-Hill, 1997) IRC: Q325.5 .M58 1997. *More
general and more comprehensive than the course, covers many branches
of
ML that are not treated in the course. Gives a good overview of the
larger picture of ML*

Nabney, Ian T.: *NETLAB:
Algorithms for Pattern Recognition* (Springer Verlag, 2001). IRC:
TA1637 .N33 2002. *A
companion book to the Bishop book, concentrating on Matlab
implementations of the main techniques described in the Bishop book.
Matlab code is public and can be downloaded from http://www.aston.ac.uk/eas/research/groups/ncrg/resources/netlab/
*

Brownlee, J.: (author's own publication, online at author's ML service portal
). * A decidedly user-friendly, hands-on intro to linear algebra, targetting ML usage, with Python exercises.*