BloomForSequenceClassification' does not have a lm_head ... can this technique still apply?

[maadnfritz]

re the notebook :✉️ MarketMail AI ✉️ Fine tuning BLOOMZ (Completed Version).ipynb

https://colab.research.google.com/drive/1ARmlaZZaKyAg6HTi57psFLPeh0hDRcPX?usp=sharing

i tried to modify the example to use BloomForSequenceClassification instead of AutoModelForCausalLM but the "Post-processing on the model":
model.lm_head = CastOutputToFloat(model.lm_head)
fails because BloomForSequenceClassification does not have an attribute lm_head.

This is true, so i change code to try and affect the last layer of BloomForSequenceClassification:
model.ln_f = CastOutputToFloat(model.ln_f)
This also fails: AttributeError: 'BloomForSequenceClassification' object has no attribute 'ln_f'

This leaves me wondering can thus gradient accumulation work for BloomForSequenceClassification? Or only for AutoModelForCausalLM? Alternatively, does anyone know if AutoModelForCausalLM can be used for fine tuning a classification task equally well as BloomForSequenceClassification?

[chris-alexiuk-1]

https://colab.research.google.com/drive/1FOkqPZBm5H53l9Zlb15px0Dlf529TMzf?usp=sharing

This notebook should be able to explain using BloomForSequenceClassification with LoRA!