Globally setting TokenSelectionBiases (logit_bias) when invoking a function #4503
Unanswered
CharlieDigital
asked this question in
Q&A
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Use Case
We have a set of plugins and functions which we would like to globally and dynamically inject a set of
TokenSelectionBiases
to ban or reduce the occurrence of specific tokens.Analysis
My first attempt was via the
KernelArguments
constructor. However, the observation here is that if I pass inOpenAIPromptExecutionSettings
to theKernelArguments
constructor, it wipes out the configuration that was loaded when the prompt is imported viaImportPluginFromPromptDirectory
.In other words, the
execution_settings
loaded via importing the plugin are replaced entirely.My second attempt was via the
FunctionInvoking
event delegate before invoking:However, the
execution_settings
are nowhere to be found and in fact, this makes sense becauseargs.Arguments.ExecutionSettings
doesn't contain theexecution_settings
in theconfig.json
.Question
Given that, is there a mechanism to globally inject a dynamically created
TokenSelectionBias
(logit_bias
) for all executions?Where are the
execution_settings
from theconfig.json
stored?Can they be modified programmatically after loading?
Beta Was this translation helpful? Give feedback.
All reactions