Introduction to Transformer Models for NLP

Course details

This course provides a comprehensive overview of large language models (LLMs), transformers, and the mechanisms—attention, embedding, and tokenization—that set the stage for state-of-the-art NLP models like BERT and ChatGPT to flourish. Instructor Sinan Ozdemir helps you develop a practical, comprehensive, and functional understanding of transformer architectures and how they are used to create modern NLP pipelines. Along the way, Sinan brings theory to life with detailed illustrations, mathematical equations, and concrete examples of Python in Jupyter notebooks.

Learning objectives
Recognize which type of transformer-based model is best for a given task.
Understand how transformers process text and make predictions.
Fine-tune transformer-based models with custom data.
Create actionable pipelines using fine-tuned models.
Deploy fine-tuned models and use them in production.
Leverage techniques in prompt engineering to optimize outputs from GPT-3 and ChatGPT.

Concepts

0. Introduction

01 - Introduction

1. Introduction to Attention and Language Models

02 - Lesson 1 topics
03 - A brief history of NLP
04 - Paying attention with attention
05 - Encoder-decoder architectures
06 - How language models look at text

2. How Transformers Use Attention to Process Text

07 - Lesson 2 topics
08 - Introduction to transformers
09 - Scaled dot product attention
10 - Multi-headed attention

3. Transfer Learning

11 - Lesson 3 topics
12 - Introduction to transfer learning
13 - Introduction to PyTorch
14 - Fine-tuning transformers with PyTorch

4. Natural Language Understanding with BERT

15 - Lesson 4 topics
16 - Introduction to BERT
17 - WordPiece tokenization
18 - The many embeddings of BERT

5. Pretraining and Fine-Tuning BERT

19 - Lesson 5 topics
20 - The Masked Language Modeling task
21 - The Next Sentence Prediction task
22 - Fine-tuning BERT to solve NLP tasks

6. Hands-On BERT

23 - Lesson 6 topics
24 - Flavors of BERT
25 - BERT for sequence classification
26 - BERT for token classification
27 - BERT for question answering

7. Natural Language Generation with GPT

28 - Lesson 7 topics
29 - Introduction to the GPT family
30 - Masked multi-headed attention
31 - Pretraining GPT
32 - Few-shot learning

8. Hands-On GPT

33 - Lesson 8 topics
34 - GPT for style completion
35 - GPT for code dictation

9. Further Applications of BERT and GPT

36 - Lesson 9 topics
37 - Siamese BERT networks for semantic searching
38 - Teaching GPT multiple tasks at once with prompt engineering

10. T5 Back to Basics

39 - Lesson 10 topics
40 - Encoders and decoders welcome - T5's architecture
41 - Cross-attention

11. Hands-On T5

42 - Lesson 11 topics
43 - Off-the-shelf results with T5
44 - Using T5 for abstractive summarization

12. The Vision Transformer

45 - Lesson 12 topics
46 - Introduction to the vision transformer (ViT)
47 - Fine-tuning an image captioning system

13. Deploying Transformer Models

48 - Lesson 13 topics
49 - Introduction to MLOps
50 - Sharing our models on HuggingFace
51 - Deploying a fine-tuned BERT model using FastAPI

14. Using Massively Large Language Models

52 - Lesson 14 topics
53 - Modern large language models
54 - GPT-3 and ChatGPT
55 - Other LLMs and semantic search with OpenAI embeddings

Conclusion

56 - Summary