reimplement sentence based max len padding #67
Labels
Comments
Waino
added
enhancement
Waino
added a commit
that referenced
this issue
May 20, 2024
Fixed size batching entails using - "batch_type: sents" to fix the batch dimension to batch_size, and - "pad_to_max_length: true" together with "max_length" to fix the sequence length dimension. Closes #67
Waino
added a commit
that referenced
this issue
May 20, 2024
Fixed size batching entails using - "batch_type: sents" to fix the batch dimension to batch_size, and - "pad_to_max_length: true" together with "max_length" to fix the sequence length dimension. Closes #67
Waino
added a commit
that referenced
this issue
May 20, 2024
Fixed size batching entails using - "batch_type: sents" to fix the batch dimension to batch_size, and - "pad_to_max_length: true" together with "max_length" to fix the sequence length dimension. Closes #67
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
#55 implements fixed size minibatches with both a fixed number of sentences and a fixed number of tokens per sentence, with truncating/padding to enforce the length.
This seems to result in good throughput and GPU utilization under some conditions, but is sensitive to the
--max_lengthparameter: too short will fill up the batches but result in truncation and inability to translate long sentences, while too long will lead to excessive padding which is wasteful in terms of compute.The implementation in #55 is built on top of the spiral
LookAheadBucketing, which was removed in #66. It should be reimplemented as a standalone component.The text was updated successfully, but these errors were encountered: