Student projects: preparatory information

Considering doing a project in MINDS?

This page provides decision-making-helping information for students who think about doing a project in the MINDS group. The information given below is equally relevant for Bachelor thesis projects, Master thesis projects, and individual research projects.

What is “Machine Learning”?

ML is a collection of algorithm design methods which is firmly rooted in probability theory, statistics, linear algebra and some calculus. ML is a particularly math-heavy area of Computer Science. The input to an ML algorithm is (large quantities of) data - for example images, video, biosignals, geo-surveillance data, scientific measurements, financial records, web-crawling harvests. An ML algorithm tries to find regularities (structure, redundancies, symmetries...) in the data and use these to create a formal model of the data, in a format which should be much more compact than the original data. After having thus learnt a model from the data, the model can be exploited for a variety of purposes, for instance data interpolation or prediction, pattern classification, control, fault monitoring, robot action selection, or pattern generation.

Should you go for it?

If you are contemplating choosing ML for your project, the first thing for you is to find out whether you really like this kind of stuff and whether it is accessible to you. The standard way to find out is to take my 2nd year undergraduate course Neural Networks (AI) (KIB.NNKI03) or my 2nd year Master course Machine Learning (KIM.ML09) and emerge from that course with enthusiasm. If you haven’t taken one of those courses (or related ones offered in the CS department), you face a rather steep working-in challenge because you will have to self-study the basics of machine learning. My lecture notes for KIB.NNKI03 or my lecture notes for KIM.ML09 may be good fast-self-study readings if you are determined and dedicated. - Students of mathematics usually have little difficulties absorbing the requisite background knowledge and are by default welcome to engage in a machine learning project.

Prerequisites and admission procedure

Required qualification. Since embarking on a ML project really only makes sense when you have made friends with elementary concepts of this field, we will supervise you only if both you and I can be sure that you muster the basics of the field. We will take this for granted if you have passed the written exam of the 2nd year Bachelor course Neural Networks, or of the 2nd year Master course Machine Learning with a grade of 7.00 or better. If you did not take one of these courses, but have acquired reasonably substantial ML background otherwise, we will carry out a qualification exam.

Admission. Since the supervision capacity in MINDS is bounded and ML is a most popular subject, there are more qualified students wanting to do a project than we can host. We generally accept qualified students in the order of incoming, seriously considered applications. A quick email to get a time stamp is not enough - you should spend a little effort of formulating an application text (say, 1 page max with an outline of your motivation, qualification, project theme suggestions that are either relating to the topics you find on this webpage, or a self-designed ML project). However it is a good idea to first ask me whether there is supervision capacity left at all in MINDS.

Inside MINDS: which subject options?

Student projects in MINDS are usually related to the active research areas of our group, though we can also negotiate very individual project themes in cases where the student knows what s/he is up to and is qualified to work without close guidance. Our group’s three default thematic areas are

  • Echo State Networks, a learning paradigm for recurrent neural networks - this is likely the most accessible kind of subject and has been the typical choice of CS, IMS, and DE students in the past,
  • Observable Operator Models, the most “mathy” and arguably most elegant and foundational kind of research we can offer - this has been opted for by most mathematics students in the past,
  • Conceptors, the most complex topic because it builds on echo state networks, maybe a choice for the daring ones with a very solid linear algebra background and an interest in cognitive neural dynamics.
  • Theory of non-digital computing, the most recent, most demanding and currently most active research theme in MINDS.

We don’t have ready-made project topics for you to pick from. Rather, we’ll be having a cup of coffee together and define a project topic tailored to your interests and background.

Below you find startup reading suggestions for each of these four fields.

Startup reading for Echo State Networks

To get a clearer picture of this subject, please consult the following papers in the given order:

  1. A Scholarpedia article for a first overview.
  2. A short highlight paper: H. Jaeger and H. Haas, Harnessing Nonlinearity: Predicting Chaotic Systems and Saving Energy in Wireless Communication. Science 304, 2 April 2004, pp. 78-80 (preprint pdf)
  3. An easy overview paper of the status of echo state network research in the current landscape of ML: M. Lukoševičius, H. Jaeger, B. Schrauwen (2012): Reservoir Computing Trends. KI - Künstliche Intelligenz, 1-7 (Preprint pdf)
  4. An introductory, more detailed technical report which has a large number of examples in it: H. Jaeger (2001): Short term memory in echo state networks. GMD Report 152, German National Research Center for Information Technology, 2001 (60 pp.) (pdf)

You might also wish to check out a (very good) BSc thesis written on an ESN theme (by Valentin Vasiliu, 2016).

Startup reading for Observable Operator Models

There is unfortunately no really easy introduction paper for OOMs. The most accessible exposition is given in the first part in the 2-part paper

  1. H. Jaeger, M. Zhao, K. Kretzschmar, T. Oberstein, D. Popovici, A. Kolling (2006): Learning observable operator models via the ES algorithm. In: S. Haykin, J. Principe, T. Sejnowski, J. McWhirter (eds.), New Directions in Statistical Signal Processing: from Systems to Brain. MIT Press, Cambridge, MA., 417-464 (draft version, pdf)

Note: OOM theses written in the MINDS group seem to be good for carving an academic carreer path: Cristian Danescu-Mizil (Master thesis 2007) is now an assistant professor at Cornell, Anca Dragan (BSc thesis 2009) is assistant professor at Berkeley, and Josip Djolonga (BSc thesis 2011) is a PhD student at ETH Zurich with a Google European Doctoral Fellowship. You can check out Josip's BSc thesis if you want to get an OOM feeling.

Startup reading for Conceptors

Finally, research on conceptors is young and there is not much literature yet. Before you embark on this expedition, please familiarize yourself a little with echo state networks, because conceptor theory is framed within that context. Here is what can be offered:

  1. A short teaser paper is H. Jaeger (2014): Conceptors: an easy introduction. (arXiv)
  2. The "official coming-out" paper: H. Jaeger (2017): Using Conceptors to Manage Neural Long-Term Memories for Temporal Patterns. Journal of Machine Learning Research 18, 1-43 (pdf at JMRL)

There have already been a number of theses, quite different ones for that matter, that were centered on conceptors. Example 1: conceptors examined as a tool in continual learning of image segmentation (Master, deep learning flavor). Example 2: conceptors as a tool to reconstruct semantic networks from text data (Bachelor, AI flavor)

Startup reading for Theory of non-digital computing

Research in this domain is very interdisciplinary and we collaborate with groups from materials science in the Zernike (for unconventional physical devices and hardware) and the mathematics department (for exploring dynamical systems as carriers of computing). We furthermore experiment ourselves with non-digital, unclocked, spiking neuromorphic microprocessors. This research area is the most advanced and difficult of all the areas where we offer student projects. It is suitable (only) for students with very strong interest in abstract theory-building. To get a feeling, take a look at

  1. A methodological (and long and detailed) survey and position paper: H. Jaeger (2021): Toward a Generalized Theory Comprising Digital, Neuromorphic, and Unconventional Computing. Neuromorphic Computing and Engineering 1(1) (open access)
  2. Another long and detailed survey and position paper: H. Jaeger and F. Catthoor (2022): Toward a theory of timescales in neuromorphic computing. Project deliverable report )for the MemScales Horizon 2020 EU project Memory technologies with multi-scale time constants for neuromorphic architectures )