Other generation strategies

[dachenlian]

I'm wondering if it's possible to use other generation strategies that are available in the Transformers library, like eta sampling.

[fawern]

Currently, Text Generation Inference supports commonly used generation strategies like temperature, top-k sampling, top-p sampling, and typical sampling. However, advanced strategies like eta sampling (available in the Transformers library) are not directly supported in TGI at this time.

If you want to use custom generation strategies like eta sampling, you would need to modify the TGI code or implement those strategies in a custom inference pipeline.