PyText Files

New Features

RoBERTa and XLM-R - Integrate XLM-R into PyText (#1120) - Consolidate BERT, XLM and RobERTa Tensorizers (#1119) - Add XLM-R for joint model (#1135) - Open source Roberta (#1032) - Simple Transformer module components for RoBERTa (#1043) - RoBERTa models for document classification (#933) - Enable MLM training for RobertaEncoder (#1126) - Standardize RoBERTa Tensorizer Vocab Creation (#1113) - Make RoBERTa usable in more tasks including QA (#1017) - RoBERTa-QA JIT (#1088) - Unify GPT2BPE Tokenizer (#1110) - Adding Google SentencePiece as a Tokenizer (#1106)

TorchScript support - General torchscript module (#1134) - Support torchscriptify XLM-R (#1138) - Add support for torchscriptification of XLM intent slot models (#1167) - Script xlm tensorizer (#1118) - Refactor ScriptTensorizer with general tensorize API (#1117) - ScriptXLMTensorizer (#1123) - Add support for Torchscript export of IntentSlotOutputLayer and CRF (#1146) - Refactor ScriptTensorizor to support both text and tokens input (#1096) - Add torchscriptify API in tokenizer and tensorizer (#1055) - Add more stats in torchscript latency script (#1044) - Exported Roberta torchscript model include both traced_model and pre-processing logic (#1013) - Native Torchscript Wordpiece Tokenizer Op for BERTSquadQA, Torchscriptify BertSQUADQAModel (#879) - TorchScript-ify BERT training (#887) - Modify Return Signature of TorchScript BERT (#1058) - Implement BertTensorizer and RoBERTaTensorizer in TorchScript (#1053)

Others - FairseqModelEnsemble class (#1116) - Inverse Sqrt Scheduler (#1150) - Lazy modules (#1039) - Adopt Fairseq MemoryEfficientFP16Optimizer in PyText (#910) - Add RAdam (#952) - Add AdamW (#945) - Unify FP16&FP32 API (#1006) - Add precision at recall metric (#1079) - Added PandasDataSource (#1098) - Support testing Caffe2 model (#1097) - Add contextual feature support to export for Seq2Seq models - Convert matmuls to quantizable nn.Linear modules (#1304) - PyTorch eager mode implementation (#1072) - Implement Blockwise Sparsification (#1050) - Support Fairseq FP16Optimizer (#1008) - Make FP16OptimizerApex wrapper on Apex/amp (#1007) - Remove vocab from cuda (#955) - Add dense input to XLMModel (#997) - Replace tensorboardX with torch.utils.tensorboard (#1003) - Add mentioning of mixed precision training support (#643) - Sparsification for CRF transition matrix (#982) - Add dense feature normalization to Char-LSTM TorchScript model. (#986) - Cosine similarity support for BERT pairwise model training (#967) - Combine training data from multiple sources (#953) - Support visualization of word embeddings in Tensorboard (#969) - Decouple decoder and output layer creation in BasePairwiseModel (#973) - Drop rows with insufficient columns in TSV data source (#954) - Add use_config_from_snapshot option(load config from snapshot or current task) (#970) - Add predict function for NewTask (#936) - Use create_module to create CharacterEmbedding (#920) - Add XLM based joint model - Add ConsistentXLMModel (#913) - Optimize Gelu module for caffe2 export (#918) - Save best model's sub-modules when enabled (#912)

Documentation / Usability

• XLM-R tutorial in notebook (#1159)
• Update XLM-R OSS tutorial and add Google Colab link (#1168)
• Update "raw_text" to "text" in tutorial (#1010)
• Make tutorial more trivial (add git clone) (#1037)
• Changes to make tutorial code simpler (#1002)
• Fix datasource tutorial example (#998)
• Handle long documents in squad qa datasource and models (#975)
• Fix pytext tutorial syntax (#971)
• Use torch.equal() instead of "==" in Custom Tensorizer tutorial (#939)
• Remove and mock doc dependencies because readthedocs is OOM (#983)
• Fix Circle CI build_docs error (#959)
• Add OSS integration tests: DocNN (#1021)
• Print model into the output log (#1127)
• Migrate pytext/utils/torch.py logic into pytext/torchscript/ for long term maintainability (#1082)
• Demo datasource fix + cleanup (#994)
• Documentation on the config files and config-related commands (#984)
• Config adapter old data handler helper (#943)
• Nicer gen_config_impl (#944)

Deprecated Features

• Remove DocModel_Deprecated (#916)
• Remove RNNGParser_Deprecated, SemanticParsingTask_Deprecated, SemanticParsingCppTask_Deprecate, RnngJitTask,
• Remove QueryDocumentTask_Deprecated(#926)
• Remove LMTask_Deprecated and LMLSTM_Deprecated (#882)
• CompositionDataHandler to fb/deprecated (#963)
• Delete deprecated Word Tagging tasks, models and data handlers (#910)

Bug Fixes

• Fix caffe2 predict (#1103)
• Fix bug when tensorizer is not defined (#1169)
• Fix multitask metric reporter for lr logging (#1164)
• Fix broken gradients logging and add lr logging to tensorboard (#1158)
• Minor fix in blockwise sparsifier (#1130)
• Fix clip_grad_norm API (#1143)
• Fix for roberta squad tensorizer (#1137)
• Fix multilabel metric reporter (#1115)
• Fixed prepare_input in tensorizer (#1102)
• Fix unk bug in exported model (#1076)
• Fp16 fixes for byte-lstm and distillation (#1059)
• Fix clip_grad_norm_ if grad_norm > max_norm > 0: TypeError: '>' not supported between instances of 'float' and 'NoneType' (#1054)
• Fix context in multitask (#1040)
• Fix regression in ensemble trainer caused by recent fp16 change (#1033)
• ReadTheDocs OOM fix with CPU Torch (#1027)
• Dimension mismatch after setting max sequence length (#1154)
• Allow null learning rate (#1156)
• Don't fail on 0 input (#1104)
• Remove side effect during pickling PickleableGPT2BPEEncoder
• Set onnx==1.5.0 to fix CircleCI build temporarily (#1014)
• Complete training loop gracefully even if no timing is reported (#1128)
• Propagate min_freq for vocab correctly (#907)
• Fix gen-default-config with Model param (#917)
• Fix torchscript export for PyText modules (#1125)
• Fix label_weights in DocModel (#1081)
• Fix label_weights in bert models (#1100)
• Fix config issues with Python 3.7 (#1066)
• Temporary fix for Fairseq dependency (#1026)
• Fix MultipleData by making tensorizers able to initialize from multiple data sources (#972)
• Fix bug in copy_unk (#964)
• Division by Zero bug in MLM Metric Reporter (#968)