📚 The Ilya 30u30 Deep Learning Compendium

A free comprehensive book based on the 30 papers and resources recommended by Ilya Sutskever for mastering Artificial Intelligence.

📋 Table of Contents

Jump to: Part I • Part II • Part III • Part IV • Part V • Part VI • Part VII

📖 About This Book

This book transforms Ilya Sutskever’s legendary “30u30” reading list into an accessible, structured learning journey. Each chapter distills complex research papers into clear explanations with visual Mermaid diagrams.

Source: Based on Ilya’s 30u30 Reading List and Primers • Ilya Sutskever’s Top 30

📖 Table of Contents

Part I: Foundations of Learning and Complexity

Understanding the theoretical bedrock of machine learning

Ch Title Paper/Source
1 The Minimum Description Length Principle Grünwald, 2004
2 Kolmogorov Complexity and Algorithmic Randomness Shen et al.
3 Keeping Neural Networks Simple Hinton & Van Camp, 1993
4 The Coffee Automaton Aaronson et al., 2014
5 The First Law of Complexodynamics Aaronson Blog

Part II: Convolutional Neural Networks

The revolution in visual understanding

Ch Title Paper/Source
6 AlexNet - The ImageNet Breakthrough Krizhevsky et al., 2012
7 CS231n - CNNs for Visual Recognition Stanford Course
8 Deep Residual Learning (ResNet) He et al., 2015
9 Identity Mappings in Deep Residual Networks He et al., 2016
10 Dilated Convolutions for Multi-Scale Context Yu & Koltun, 2015

Part III: Sequence Models and Recurrent Networks

Learning from sequential data

Ch Title Paper/Source
11 The Unreasonable Effectiveness of RNNs Karpathy Blog, 2015
12 Understanding LSTM Networks Colah’s Blog, 2015
13 Recurrent Neural Network Regularization Zaremba et al., 2014
14 Relational Recurrent Neural Networks Santoro et al., 2018

Part IV: Attention and Transformers

The attention revolution

Ch Title Paper/Source
15 Neural Machine Translation with Attention Bahdanau et al., 2014
16 Attention Is All You Need (Transformers) Vaswani et al., 2017
17 The Annotated Transformer Harvard NLP

Part V: Advanced Architectures

Specialized neural network designs

Ch Title Paper/Source
18 Pointer Networks Vinyals et al., 2015
19 Order Matters: Sequence to Sequence for Sets Vinyals et al., 2015
20 Neural Turing Machines Graves et al., 2014
21 Neural Message Passing for Quantum Chemistry Gilmer et al., 2017
22 A Simple Neural Network Module for Relational Reasoning Santoro et al., 2017
23 Variational Lossy Autoencoder Chen et al., 2016

Part VI: Scaling and Efficiency

Training neural networks at scale

Ch Title Paper/Source
24 Deep Speech 2 Amodei et al., 2015
25 Scaling Laws for Neural Language Models Kaplan et al., 2020
26 GPipe - Pipeline Parallelism Huang et al., 2018

Part VII: The Future of Intelligence

Philosophical and theoretical perspectives

Ch Title Paper/Source
27 Machine Super Intelligence Shane Legg, 2008

🎓 How to Read This Book

Each chapter includes:

  • 📖 Accessible explanations - Complex concepts made simple
  • 📊 Mermaid diagrams - Visual representations of key ideas
  • 🔢 Key equations - Essential formulas with intuitive explanations
  • 🔗 Connections - Links between related papers and concepts
  • 💡 Modern applications - How ideas are used today
  • 📚 References - Original papers and further reading

🗺️ Suggested Reading Paths

🎯 Standard Path (Recommended)

Read chapters 1-27 in order for a complete journey from theory to practice.

⚡ Practitioner’s Path

If you want to build things quickly:

  1. Chapter 6 (AlexNet) → Chapter 8 (ResNet)
  2. Chapter 11-12 (RNNs/LSTMs)
  3. Chapter 15-17 (Attention/Transformers)
  4. Chapter 25 (Scaling Laws)

🧠 Theorist’s Path

If you love theory and foundations:

  1. Chapters 1-5 (Full Part I)
  2. Chapter 27 (Superintelligence)
  3. Then practical chapters as needed

🔬 Researcher’s Path

For cutting-edge architectures:

  1. Chapters 16-17 (Transformers)
  2. Chapters 18-23 (Advanced Architectures)
  3. Chapters 25-26 (Scaling)

⭐ Support This Project

If you find this book useful, please consider ⭐ starring the GitHub repository to help others discover it!

📄 License

Educational content based on public research papers. All original papers are cited with links to their sources.

Table of contents

Back to top

Educational content based on public research papers. All original papers are cited with links to their sources.