Advanced NLP (SCIA / ANLP1 & ANLP2)

Objectives

1. Knowlegde: Understand how the main components of modern NLP pipelines work.
2. Hands-on Practice: Develop the ability to build NLP systems for most major NLP tasks (classification, question-answering, ChatGPT-like assistant...).
3. Critical Thinking: Understand the flaws and challenges of NLP, and come up with creative ideas to improve current systems.

Course organization

Part 1: General NLP

• Session 1 (3/10, Célia): Recap
• Session 2 (17/10, Francis): Tokenization
• Session 3 (24/10, Francis): Language Modeling
• Session 4 (31/10, Francis): Modern NLP with limited resources
• Session 5 (7/11, Francis): Modern Interpretability
• Session 6 (14/11, Francis/Célia): Midterm project session

Part 2: Advanced NLP Applications

• Session 7 (21/11, Célia): Safety, Ethics, and Alignment of LMs
• Session 8 (28/11, Rian): Advanced NLP Tasks
• Session 9 (5/12, Rian): Domain-specific NLP
• Session 10 (12/12, Rian): Multilingual NLP
• Session 11 (19/12, Célia): Multimodal NLP
• Session 12 (16/01, Francis/Célia/Rian): Final Presentations!

Materials

1. Recap on Deep Learning & basic NLP (slides / lab session / lab correction )
2. Tokenization (slides / lab session)
3. Language Modeling (slides / lab session)
4. Efficient NLP (slides / lab session)
5. Modern Interpretability (slides / lab session)
6. Safety, Ethics, and Alignment of LMs (slides / lab session)
7. Advanced NLP tasks (slides / lab session)
8. Domain-specific NLP (slides / lab session)
9. Multilingual NLP (slides / lab session)
10. Multimodal NLP (slides / lab session)

Evaluation

The evaluation consists in a team project (3-5 people). The choice of the subject is free but needs to follow some basic rules:

• Obviously, the project must be highly related with NLP and especially with the notions we will cover in the course
• You can only use open-source LLM that you serve yourself. In other words, no API / ChatGPT-like must be used, except for final comparison with your model.
• You must identify and address a challenging problem (e.g. not only can a LLM do X?, but can a LLM that runs on a CPU do X?, or can I make a LLM better at X?)
• It must be reasonably doable: you will not be able to fine-tune (even to use) a 405B parameters model, or to train a model from scratch. That's fine, there are a lot of smaller models that should be good enough, like the Pythia models, TinyLLama, the 1B parameter OLMo, or the small models from the Llama3.2 suite.

⏰ The project follows 3 deadlines:

• Project announcement (before 2025/10/17): send an email to francis.kulumba@inria.fr with cc's celia.nouri@inria.fr and rian.touchent@inria.fr explaining
  • The team members (also cc'ed)
  • A small description of the project (it can change later on)
• Project proposal (25% of final grade, before 2025/11/21): following this template, produce a project proposal explaining first attempts (e.g. version alpha), how they failed/succeeded and what you want to do before the delivery.
• Project delivery (75% of final grade, 2026/01/09): delivery of a short report detailing each person's contributions, a GitHub repo with an explanatory README + oral presentation on January 16th

Inspiring articles

Tokenization

• A Vocabulary-Free Multilingual Neural Tokenizer for End-to-End Task Learning (https://arxiv.org/abs/2204.10815)
• BPE-Dropout: Simple and Effective Subword Regularization (https://aclanthology.org/2020.acl-main.170/)
• FOCUS: Effective Embedding Initialization for Monolingual Specialization of Multilingual Models (https://aclanthology.org/2023.emnlp-main.829/)

Fast inference

• Efficient Streaming Language Models with Attention Sinks (https://arxiv.org/abs/2309.17453)
• Lookahead decoding (https://lmsys.org/blog/2023-11-21-lookahead-decoding/)
• Efficient Memory Management for Large Language Model Serving with PagedAttention (https://arxiv.org/pdf/2309.06180.pdf)

Inference-time scaling (OpenAI's o1 model)

• Chain-of-Thought Prompting Elicits Reasoning in Large Language Models (https://arxiv.org/abs/2201.11903)
• Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters (https://arxiv.org/abs/2408.03314v1)

LLM detection

• Detecting Pretraining Data from Large Language Models (https://arxiv.org/abs/2310.16789)
• Proving Test Set Contamination in Black Box Language Models (https://arxiv.org/abs/2310.17623)

SSMs (off-program)

• Mamba: Linear-Time Sequence Modeling with Selective State Spaces (https://arxiv.org/abs/2312.00752)

Alignment & Safety

• Null It Out: Guarding Protected Attributes by Iterative Nullspace Projection (https://aclanthology.org/2020.acl-main.647/)
• Direct Preference Optimization: Your Language Model is Secretly a Reward Model (https://arxiv.org/abs/2305.18290)
• Text Embeddings Reveal (Almost) As Much As Text (https://arxiv.org/abs/2310.06816)