Fairseq(-py) is a sequence modeling toolkit that allows researchers and developers to train custom models for translation, summarization, language modeling and other text generation tasks. We provide reference implementations of various sequence modeling papers. Recent work by Microsoft and Google has shown that data parallel training can be made significantly more efficient by sharding the model parameters and optimizer state across data parallel workers. These ideas are encapsulated in the new FullyShardedDataParallel (FSDP) wrapper provided by fairscale. Fairseq can be extended through user-supplied plug-ins. Models define the neural network architecture and encapsulate all of the learnable parameters. Criterions compute the loss function given the model outputs and targets. Tasks store dictionaries and provide helpers for loading/iterating over Datasets, initializing the Model/Criterion and calculating the loss.

Features

  • Multi-GPU training on one machine or across multiple machines (data and model parallel)
  • Fast generation on both CPU and GPU with multiple search algorithms implemented
  • Gradient accumulation enables training with large mini-batches even on a single GPU
  • Mixed precision training (trains faster with less GPU memory on NVIDIA tensor cores)
  • Easily register new models, criterions, tasks, optimizers and learning rate schedulers
  • Flexible configuration based on Hydra allowing a combination of code, command-line and file based configuration

Project Samples

Project Activity

See All Activity >

License

MIT License

Follow Fairseq

Fairseq Web Site

Other Useful Business Software
Our Free Plans just got better! | Auth0 Icon
Our Free Plans just got better! | Auth0

With up to 25k MAUs and unlimited Okta connections, our Free Plan lets you focus on what you do best—building great apps.

You asked, we delivered! Auth0 is excited to expand our Free and Paid plans to include more options so you can focus on building, deploying, and scaling applications without having to worry about your security. Auth0 now, thank yourself later.
Try free now
Rate This Project
Login To Rate This Project

User Reviews

Be the first to post a review of Fairseq!

Additional Project Details

Programming Language

Python

Related Categories

Python Artificial Intelligence Software, Python Neural Network Libraries

Registered

2021-05-14