The repository contains a scratch-implementation of the transformer network encoder and its training in the emotion classification task on emotion dataset. The bert-base-uncased tokenizer was used.
The repository contains a comparison of models:
- Logistic Regression and SVM ("classic" algorithms, trained on DistilBERT embeedings),
- LSTM and BiLSTM (custom models, written with Pytorch, trained on Fasttext embeedings),
- finetuned DistilBERT
trained in the task of emotions classification on emotion dataset.
The repository contains a comparison of DistilBERT models trained in the task of emotions classification (emotion dataset). Compared models are:
- DistilBERT with freezed first 6, 4 and 2 layers,
- DistilBERT trained with bottleneck adapters,
- unfreezed DistilBERT.
The repository contains a dataset preprocessing and XLM-RoBERTa model finetuning in the named entity recognition task on the subset ('de', 'fr', 'it, 'en') of xtreme dataset. Moreover cross-ligual transfer has been examined.
The repository contains finetuning of the distilled Pegasus model distill-pegasus-cnn-16-4 in the task of generating abstract summaries (on the Samsum dataset).
The repository contains a short study of the influence of parameters (temperature, number of beams, topk, topp) on the quality of the output generated by the trained gpt2 model.