You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently, we only support training encoder-decoder models.
We might want to support encoder-only (e.g. BERT) and decoder-only models (e.g. GPTs). This could be inferred automatically from the types of sharing groups defined per task:
if the dec_sharing_groups key is absent then we do not instantiate a layer stack decoder, and directly route the encoder's output to the generator
if the enc_sharing_groups key is absent from all tasks, then we need to instantiate an encoder stack and systematically add an autoregressive mask at every forward pass.
Batched data already distinguishes between target-side and source-side inputs and target outputs (labels).
The text was updated successfully, but these errors were encountered:
Currently, we only support training encoder-decoder models.
We might want to support encoder-only (e.g. BERT) and decoder-only models (e.g. GPTs). This could be inferred automatically from the types of sharing groups defined per task:
dec_sharing_groups
key is absent then we do not instantiate a layer stack decoder, and directly route the encoder's output to the generatorenc_sharing_groups
key is absent from all tasks, then we need to instantiate an encoder stack and systematically add an autoregressive mask at every forward pass.Batched data already distinguishes between target-side and source-side inputs and target outputs (
labels
).The text was updated successfully, but these errors were encountered: