Tanja Samardžić, University of Geneva, Lecture notes, Autumn 2024

Introduction to NLP

(Traitement automatique du langage naturel - TALN)

These notes should be used as a guide for acquiring the most important notions and terminology in contemporary Natural Language Processing. Most of the notions mentioned in the notes are explained in the listed sources. A few visualisations are included in the notes for a better overview and intuitive understanding. This course includes a practical part too, which is managed on Moodle.

 

Online textbooks:

• Dan Jurafsky and James H. Martin: Speech and Language Processing (3rd ed. draft)
• Jacob Eisenstein: Natural Language Processing
• Yoav Goldberg: A Primer on Neural Network Models for Natural Language Processing
• Tom Mitchell: Machine Learning - an old classic

Blogs and other learning resources:

• Benoît Sagot, Apprendre les langues aux machines, cours du Collège de France
• Luigi Rizzi, General linguistics, cours du Collège de France
• YouTube channel Crash Course Linguistics
• Lena Voita's NLP course
• Hugging Face Course
• Graham Neubig, YouTube channel CMU Advanced NLP
• Jay Alammar's blog
• Serrano Academy
• YouTube channel 3blue1brown

Topics

1. NLP tasks, data sets, benchmarks

2. Large language models (LLMs), Artificial Intelligence (AI) and Natural language processing (NLP), history of NLP

3. Evaluation, data splits

4. History of language modelling

5. Representing the meaning of words with word2vec

6. Language modelling with Transformers NNs

7. Attention in language modelling

8. Subword tokenization

9. Transfer learning: performing tasks with encode-type pre-trained models

10. Transfer learning: performing tasks with decoder-type pre-trained models

11. Cross-lingual transfer and multilingual NLP

12. History of NN architectures in NLP: CNNs, LSTMs

13. What is knowledge about language?