Download Latest Version Release 0.15.1 source code.zip (2.5 MB)
Email in envelope

Get an email when there's a new version of flair

Home / v0.13.0
Name Modified Size InfoDownloads / Week
Parent folder
README.md 2023-10-28 14.0 kB
0
Release 0.13.0 source code.tar.gz 2023-10-28 2.2 MB
0
Release 0.13.0 source code.zip 2023-10-28 2.3 MB
0
Totals: 3 Items   4.5 MB 0

This release adds several major new features such as (1) faster and more memory-efficient transformer training, (2) a new plugin system for custom logging and training, (3) new API docs for better documentation - still in beta, and (4) various new models, datasets, bug fixes and enhancements. This release also increases the minimum requirement to Python 3.8!

New Feature: Faster and more memory-efficient transformer training

This release integrates @helpmefindaname's transformer-smaller-training-vocab into the ModelTrainer. This temporarily reduces a transformer's vocabulary to only the tokens in the training dataset, and after training restores the full vocabulary. Depending on the dataset, this may effect huge savings in GPU memory and tuning speeds.

To use this feature, simply add the flag reduce_transformer_vocab=True to the fine_tune method. For example, to fine-tune a distilbert model on TREC_6, run this code (step 7 has the flag to reduce the vocabulary): 

:::python
# 1. get the corpus
corpus: Corpus = TREC_6()

# 2. what label do we want to predict?
label_type = "question_class"

# 3. create the label dictionary
label_dict = corpus.make_label_dictionary(label_type=label_type)

# 4. initialize transformer document embeddings (many models are available)
document_embeddings = TransformerDocumentEmbeddings("distilbert-base-uncased", fine_tune=True)

# 5. create the text classifier
classifier = TextClassifier(document_embeddings, label_dictionary=label_dict, label_type=label_type)

# 6. initialize trainer
trainer = ModelTrainer(classifier, corpus)

# 7. fine-tune the model, but **reduce the vocabulary** for faster training
trainer.fine_tune(
    "resources/taggers/question-classification-with-transformer",
    reduce_transformer_vocab=True,  # set this to False for slow version
)

Involved PR: add reduce transformer vocab plugin by @helpmefindaname in https://github.com/flairNLP/flair/pull/3217

New Feature: Trainer Plugins

A new "Plugin" system was added to the ModelTrainer, allowing far greater options to customize the training cycle (and slimming down the code of the ModelTrainer somewhat). For instance, it is now possible to customize logging to a far greater degree and integrate third-party logging tools.

For instance, if you want to integrate ClearML logging into the above script, simply instantiate the plugin and attach it to the trainer: 

:::python
[...]

# 6. initialize trainer
trainer = ModelTrainer(classifier, corpus)

# NEW: instantiate a special logger and attach it to the trainer before the training run
ClearmlLoggerPlugin(clearml.Task.init(project_name="test", task_name="test")).attach_to(trainer)

# 7. fine-tune the model, but **reduce the vocabulary** for faster training
trainer.fine_tune(
    "resources/taggers/question-classification-with-transformer",
    reduce_transformer_vocab=True,  # set this to False for slow version
)

Involved PRs: * Proposal: Pluggable ModelTrainer train function by @plonerma in https://github.com/flairNLP/flair/pull/3084 * Major refactoring of ModelTrainer by @alanakbik in https://github.com/flairNLP/flair/pull/3182 * Allow users to use no scheduler and use a custom scheduling plugin by @plonerma in https://github.com/flairNLP/flair/pull/3200 * Don't pickle classes & plugins in modelcard by @helpmefindaname in https://github.com/flairNLP/flair/pull/3325 * Clearml logger by @helpmefindaname in https://github.com/flairNLP/flair/pull/3259 * Add a convenience conversion for flair.device by @alanakbik in https://github.com/flairNLP/flair/pull/3350

API Docs and other documentation

We are working towards improving our documentation. A first step was the release of our tutorial page. Now, we are adding (in beta) online API docs to make navigating the code and options offered by Flair easier. To enable it, we changed all docstrings to Google docstrings. However, this process is still ongoing, so expect the API docs to improve in coming versions of Flair.

You can find the API docs here: https://flairnlp.github.io/flair/master/api/index.html

Involved PRs: * Creating a doc page with autodocs by @helpmefindaname in https://github.com/flairNLP/flair/pull/3273 * Google doc strings by @helpmefindaname in https://github.com/flairNLP/flair/pull/3164 * Add redirects to old tutorials by @alanakbik in https://github.com/flairNLP/flair/pull/3211 * Add some more documentation and (rather empty) glossary page by @helpmefindaname in https://github.com/flairNLP/flair/pull/3339 * Update README.md by @eltociear in https://github.com/flairNLP/flair/pull/3241 * Fix embedding finetuning tutorial by @helpmefindaname in https://github.com/flairNLP/flair/pull/3301 * Fix build doc page action trigger by @helpmefindaname in https://github.com/flairNLP/flair/pull/3319 * Reduce gh-actions diskspace by @helpmefindaname in https://github.com/flairNLP/flair/pull/3327 * Orange secondary color by @helpmefindaname in https://github.com/flairNLP/flair/pull/3321 * Bump Flair and Python versions by @alanakbik in https://github.com/flairNLP/flair/pull/3355

Model Refactorings

In an effort to unify class names, we now offer models that inherit from DefaultClassifier for each label type we predict, i.e.: - TokenClassifier for predicting Token labels - TextPairClassifier for predicting TextPair labels - RelationClassifier for predicting Relation labels - SpanClassifier for predicting Span labels - TextClassifier for predicting Sentence labels

An advantage of such a structure is that most functionality (such as new decoders) needs to only be implemented once in DefaultClassifier and then is immediately usable for all model classes.

To enable this, we renamed and extended WordTagger as TokenClassifier, and renamed Entity Linker to SpanClassifier. This is not a breaking change yet, as the old names are still available. But in the future, WordTagger and Entity Linker will be removed.

Involved PRs: * TokenClassifier model by @alanakbik in https://github.com/flairNLP/flair/pull/3203 * Rename EntityLinker and remove some legacy embeddings by @alanakbik in https://github.com/flairNLP/flair/pull/3295

New Models

We also add two new model classes: (1) a TextPairRegressor for regression tasks on pairs of sentences (such as STS-B), and (2) an experimental Label Encoder method for few-shot classification.

Involved PRs: * Add TextPair regression model by @plonerma in https://github.com/flairNLP/flair/pull/3202 * Add dual encoder by @whoisjones in https://github.com/flairNLP/flair/pull/3208 * Adapt LabelVerbalizer so that it also works for non-BIOES span labes by @alanakbik in https://github.com/flairNLP/flair/pull/3231

New Datasets

Build Process

Bug Fixes

Enhancements

Breaking Changes

  • Removing the following legacy embeddings, as their support was droppend long ago:
  • XLNetEmbeddings
  • XLMEmbeddings
  • OpenAIGPTEmbeddings
  • OpenAIGPT2Embeddings
  • RoBERTaEmbeddings
  • CamembertEmbeddings
  • XLMRobertaEmbeddings
  • BertEmbeddings you can use TransformerWordEmbeddings or TransformerDocumentEmbeddings instead.
  • Removing ELMoTransformerEmbeddings as allennlp is no longer maintained.
  • Removal of the flair.hyperparameter module: We recommend using the hyperparameter optimzier of your choice as external module, for example see here how to fine tune flair models with the hugginface AutoTrain SpaceRunner
  • Drop of the trainer.resume(...) functionality. Similary to the flair.hyperparameter module, this functionality was dropped due to the trainer rework.
  • Changes to the trainer.train(...) and trainer.fine_tune(...) parameters:
  • monitor_train: bool was replaced by monitor_train_sample: float: this allows you to specify the percentage of training data points used for monitoring (setting monitor_train_sample=1.0 is equivalent to the previous behaivour of monitor_train=True.
  • eval_on_train_fraction is removed in favour of monitor_train_sample see monitor_train.
  • eval_on_train_shuffle is removed.
  • anneal_with_prestarts and batch_growth_annealing have been removed.
  • num_workers has been removed, now there is always used a single worker for data loading, as it is the fastest for the inmemory datasets.
  • checkpoint has been removed as parameter. You can use the CheckpointPlugin for the same behaviour.
  • cycle_momentum has been removed, as schedulers have been moved to Plugins.
  • param_selection_mode has been removed, similar to the hyper parameter optimization.
  • optimizer_state_dict and scheduler_state_dict were removed as part of the resume functionality.
  • anneal_against_dev_loss has been dropped, as the annealing goeas always against the metric specified by main_evaluation_metric
  • use_swa has been removed
  • use_tensorboard, tensorboard_comment tensorboard_log_dir & metrics_for_tensorboard are removed in favour of the TensorboardLogger plugin.
  • amp_opt_level is removed, as we moved to the torch integration.
  • WordTagger has been deprecated as it was renamed to TokenClassifier
  • EntityLinker has been deprecated as it was renamed to SpanClassifier

New Contributors

Full Changelog: https://github.com/flairNLP/flair/compare/v0.12.2...v0.13.0

Source: README.md, updated 2023-10-28