The Generative Landscape
================

<!-- WARNING: THIS FILE WAS AUTOGENERATED! DO NOT EDIT! -->

This is still a work in progress, but should be ready some time in
November. For now, if you’re interested in these topics you might want
to check out the previous course I ran on AI ART:
[AIAIART](https://github.com/johnowhitaker/aiaiart).

If you want to be notified when the course goes ‘live’ you can sign up
for an email at <http://thegenerativelandscape.com/>

If you want to support this effort, I now have a patreon:
<https://www.patreon.com/johnowhitaker>

The material will show in github pages at
<https://johnowhitaker.github.io/tglcourse/> (and the
http://thegenerativelandscape.com will redirect there once we launch).
Hooray for the magic of nbdev.

<div>

[![](https://github.com/johnowhitaker/tglcourse/actions/workflows/test.yaml/badge.svg)](https://github.com/johnowhitaker/tglcourse/actions/workflows/test.yaml)

</div>

![](index_files/figure-gfm/cell-2-output-1.png)

### The plan and TODOs

The idea is to have a core curriculum building up to an understanding of
key generative modelling techniques, split into three rough sections.
The first 5 lessons will cover basics of building NNs and crafting loss
functions. Lessons 6-10 will introduce generative modelling, GANs and
working with sequences using transformers. Lessons 11-15 will be a deep
dive into diffusion models, and lesson 16 will wrap up and give
sugestions for new directions to explore.

Alongside this will be a number of ‘bonus’ notebooks that don’t need to
be completed but which augment the core content. Managing datasets,
experiment tracking, sharing demos and so on. Some will augment specific
lessons, some will add functionality to the library and some will just
be standalone topics I think are cool. The latter category is likely to
continue to grow even after the course launches :)

There will be three suggested projects (after lessons 5, 11 and 14) to
mark milestones in the course, and the final lesson will also encourage
you to do a larger project at the end. Still working on details for how
we’ll share and encourage these student projects.

This table has a rough status on the main lessons. It looks more drastic
than it is - for many of these I have content lurking in other places so
even those marked ‘Not Started’ should come together a lot faster than
if this were being done completely from sratch!

| Lesson                              | Description                                                                                                                                      | TODOs                                                                          | Status           |
|-------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------|------------------|
| Lesson 1: PyTorch Basics            | Intro to PT, tensor manipulation, images as tensors, LeastAverageImage exercise                                                                  | Video, tidy                                                                    | Rough Draft Done |
| Lesson 2: Optimization              | Intro to GD, optimization examples exercise                                                                                                      | Video, Optimize with gen+loss without explaining                               | Rough Draft Done |
| Lesson 3: Building NNs              | nn.Module, building blocks, CNNs                                                                                                                 | Video, project, export functions? optimization/training tricks?                | WIP              |
| Lesson 4: Learning Representations  | What do networks learn, style transfer                                                                                                           | Video, tidy, …                                                                 | WIP              |
| Lesson 5: CLIP                      | Contrastive loss, CLIP, as a loss function                                                                                                       | everything (pull from AIAIART)                                                 | Not Started      |
| Lesson 6: Generative Modelling      | Intro to generative models, AEs and VAEs, issues                                                                                                 | everything (pull from AIAIART)                                                 | WIP              |
| Lesson 7: GANs 1                    | Intro to GANs, DC GAN, Conditioning                                                                                                              | everything                                                                     | Not Started      |
| Lesson 8: GANs 2                    | GAN training tricks, NOGAN, using modern GANs, VQGAN                                                                                             | everything                                                                     | Not Started      |
| Lesson 9: Sequence Modelling Intro  | idea, language modelling concept, transformer demo                                                                                               | everything                                                                     | Not Started      |
| Lesson 10: Transformers             | Intro to transformes, attention, from scratch or dissect existing ones? ViT?                                                                     | everything, check Niels’ repo and other resources                              | Not Started      |
| Lesson 11: Everythign is a sequence | Show whistlegen, protein, VQGAN, parti…                                                                                                          | everything                                                                     | Not Started      |
| Lesson 12: DM 1                     | Intro to diffusion models, toy example, comparison to DDPM                                                                                       | Video, tidy, second half, logs of my version training, better unet replacement | Rough Draft Done |
| Lesson 13: DM2                      | Conditioning, CFG, guiding, sampling, better training                                                                                            | everything                                                                     | WIP              |
| Lesson 14: DM3                      | SD deep dive                                                                                                                                     | link to video run-through or record smaller videos                             | Rough Draft Done |
| Lesson 15: DM4                      | Other modalities - ideally demo class-conditioned audio generation. Might not be done by the time the course launches but would be nice to have. | wavenet style diffusion for birdcalls, look at video stuff                     | Not Started      |
| Lesson 16: Going Further            | Finding your niche, exploring less common areas                                                                                                  | TODO                                                                           | Not Started      |

General TODOs - Create a page for project explanations + figure out
submission info. - Figure out what the discussions page (pages) looks
like - Ethics page but also callouts in each lesson - Pick a cloud for
cloud stuff (sponsor?) - Figure out plan for datasets - Figure out
losses+generators notebook/library bit and include in the plan somewhere
DONE - Figure out where smaller ‘bonus’ things like paper readings or
topical deep-dives go. Do we add a blog? - Make the website pretty -
Generate a bunch of images - Get Dave to make video assets + logo -
Streamline skipping tests / avoiding anything long-running during CI -
Set up org on HF for sharing models - Set up W&B projects for sharing
training runs/reports - Notebook to document the library

### Planned Bonus Notebooks:

- Datasets and Dataloaders Intro (also defines data util funcs for the
  rest of the course)
- Experiment Tracking (eg W&B)
- Pretrained networks and transfer learning
- **Ethics in generative modelling**
- Fine-tuning existing models
- Working with GPUs? Multi-GPU or TPU training?
- Sharing demos w/ Gradio
- Managing cloud machines
- Dataloaders deep dive, streaming data
- Navigating other codebases
- Version control and CI (+ NBDev)
- “Text inversion” training (https://arxiv.org/pdf/2208.01618.pdf)
- “prompt to prompt editing” https://arxiv.org/pdf/2208.01626.pdf
- Lots of paper readings / sumaries
- Neural cellular automata
- Evolutionary algorithms

### Projects:

- Create a custom generation pipeline by combining different generators
  and loss functions
- Train a GAN, explore hyperparameters
- Fine-tune a diffusion model on a custom dataset and share it with the
  community
- Create and share a final project, including a report and demo

Guest discussions: as many as we can :)

Paper explainers: as many as we can :)

Page stats: Total Hits:
[![HitCount](https://hits.dwyl.com/johnowhitaker/tglcourse.svg?style=flat-square&show=unique)](http://hits.dwyl.com/johnowhitaker/tglcourse)
Page visitors: ![visitor
badge](https://page-views.glitch.me/badge?page_id=tglcourse.index)
