Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

T5 support #1

Open
wants to merge 15 commits into
base: main
Choose a base branch
from
Open

T5 support #1

wants to merge 15 commits into from

Conversation

davidkaczer
Copy link
Owner

What does this PR do?

This PR adds support for pretrained LongT5 encoder-decoder models from HuggingFace.

General Changes

  • Add HuggingFacePretrainedEncoderDecoderModel class for loading LongT5
  • Add span masking collator ported from official Google implementation
  • Add test for span masking collator
  • Add example training configs for LongT5
  • Add logic to distinguish between decoder-only and encoder-decoder models when passing inputs

Breaking Changes

  • None intended

Checklist before submitting final PR

  • My PR is minimal and addresses one issue in isolation
  • I have merged the latest version of the target branch into this feature branch
  • I have reviewed my own code w.r.t. correct implementation, missing type hints, proper documentation, etc.
  • I have run a sample config for model training
  • I have checked that all tests run through (python tests/tests.py) (apparently some tests were failing already in upstream? also didn't test multi-GPU)
  • I have updated the internal changelog (CHANGELOG_DEV.md)

davidkaczer and others added 15 commits August 8, 2024 10:53
@davidkaczer
feat(t5): WIP: add support for T5 model
697d515 
- add SpanMaskingCollateFn for span denoising objective
- support loading T5 checkpoint from huggingface
- support passing decoder inputs to T5 model
- add example config for pretraining T5 checkpoint from HF
@davidkaczer
fix(t5): shift target tokens for input to decoder
be4d931
@davidkaczer
Merge branch 'Modalities:main' into t5
a90838b
@davidkaczer
chore: add span masking collator test
16b3690
@davidkaczer
Merge branch 't5' of https://github.com/davidkaczer/modalities into t5
0cc59dc
update training yaml file using T5 model and tokenizer, also create n…
8fe753a 
…ew huggingface model wrapper for encoder decoder architecture
@davidkaczer
fix(t5): encoder and decoder model handling logic
41df9bf 
- add logic to model_predict_batch
- fix example config
- cleanup HF model implementation
@davidkaczer
chore: pre-commit hook compliance
578e9e6
@davidkaczer
docs: add docstrings and type hints
e32f825
@davidkaczer
refactor: t5 config files
8699d46
@davidkaczer
fix: don't add EOS token to span masking batch
edfdc28
@davidkaczer
fix: set excluded weight groups in t5 config
6be5b34
@davidkaczer
fix: remove tokenizer dependency
1ba5006 
- remove dependency in span masking collator
- refactor t5 config
- refactor collator test
@davidkaczer
refactor: cleaner typecheck in model_predict_batch
30ffca2
@davidkaczer
refactor: change parent class of HuggingFacePretrainedEncoderDecoderM…
81f43e0 
…odel

- simplify future type checks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@davidkaczer