Download
Implementation of DALL-E 2, OpenAI's updated text-to-image synthesis neural network, in Pytorch. The main novelty seems to be an extra layer of indirection with the prior network (whether it is an autoregressive transformer or a diffusion network), which predicts an image embedding based on the text embedding from CLIP. Specifically, this repository will only build out the diffusion prior network, as it is the best performing variant (but which incidentally involves a causal transformer as the denoising network) To train DALLE-2 is a 3 step process, with the training of CLIP being the most important. To train CLIP, you can either use x-clip package, or join the LAION discord, where a lot of replication efforts are already underway. Then, you will need to train the decoder, which learns to generate images based on the image embedding coming from the trained CLIP.
Features
- Generate the DALL-E2 images from text
- You can also train the decoder on images of greater than the size (say 512x512)
- For the layperson, no worries, training will all be automated into a CLI tool
- Training on Preprocessed CLIP Embeddings
- Alternatively, you can also use Open Clip
- Inpainting is also built into the Decoder
Project Samples
