Flattened inf params #152
Flattened inf params #152
Conversation
Signed-off-by: gkumbhat <[email protected]>
Signed-off-by: gkumbhat <[email protected]>
Signed-off-by: gkumbhat <[email protected]>
Signed-off-by: gkumbhat <[email protected]>
Signed-off-by: gkumbhat <[email protected]>
Signed-off-by: gkumbhat <[email protected]>
Signed-off-by: gkumbhat <[email protected]>
Signed-off-by: gkumbhat <[email protected]>
Signed-off-by: gkumbhat <[email protected]>
Signed-off-by: gkumbhat <[email protected]>
gkumbhat
requested review from
alex-jw-brooks,
evaline-ju and
gabe-l-hart
as code owners
Signed-off-by: gkumbhat <[email protected]>
Signed-off-by: gkumbhat <[email protected]>
| top_p: Optional[float] = 0.0, | ||
| typical_p: Optional[float] = 0.0, | ||
| temperature: Optional[float] = 1.0, | ||
| repetition_penalty: Optional[float] = 0.0, |
There was a problem hiding this comment.
Same comments for repetition penalty and top_p here
| truncation = True | ||
|
|
||
| if repetition_penalty == 0.0: | ||
| repetition_penalty = 1.0 |
There was a problem hiding this comment.
Ah, I see now - but why is this being overridden here rather than just using 1 as the default in .run?
Is top_p supposed to be handled outside of generate like this too?
There was a problem hiding this comment.
This was to align with tgis, as per the comments in proto. https://github.com/caikit/caikit-tgis-backend/blob/main/caikit_tgis_backend/generation.proto#L103
| typical_p=0.23, | ||
| temperature=0.77, | ||
| ) | ||
| assert isinstance(pred, GeneratedTextResult) |
There was a problem hiding this comment.
It might be nice to add some kind of validation to trace the input args to generate or something like that? Since these tests aren't actually validating that greedy / sampling etc are happening
Signed-off-by: gkumbhat <[email protected]>
Signed-off-by: gkumbhat <[email protected]>
Signed-off-by: gkumbhat <[email protected]>
Signed-off-by: gkumbhat <[email protected]>
| error.type_check("<NLP84635843E>", int, allow_none=True, top_k=top_k) | ||
| error.type_check("<NLP55267523E>", float, allow_none=True, top_p=top_p) | ||
| error.type_check("<NLP13670202E>", float, allow_none=True, typical_p=typical_p) | ||
| error.type_check( |
There was a problem hiding this comment.
type check for decoding_method, temperature, max_time and exponential_decay_length_penalty? And a value check on decoding_method something like this?
… run Signed-off-by: gkumbhat <[email protected]>
Signed-off-by: gkumbhat <[email protected]>
| Amount of time in seconds that the query should take maximum. | ||
| NOTE: this does not include network overhead. | ||
| Range: 0-120.0 | ||
| exponential_decay_length_penalty: Tuple(int, float) |
There was a problem hiding this comment.
Type should also include ExponentialDecayLengthPenalty
Supports #155