You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently the OpenAI integration (#577) does not send any details about the schema or data of the user in the prompt to OpenAI. We should explore sending the schema or at least database and table names along with the prompt which should result in more relevant KQL queries. This should be conditional and opt-out by default. The UX is unsolved here but maybe a checkbox in the header to include schema details works well enough?
Potential issues
Cost
The OpenAI API charges per 1k tokens sent (and more if the user uses gpt4). Including the schema or even just the table names in the prompt potentially introduces many more tokens than the user is anticipating. We need to avoid surprise charges from API use. We should at the very least warn the user about this possibility or at best estimate how many tokens will be sent before the request is actually made.
Token limits
There is a max of 4096 tokens for the API. Along with ☝️ we should estimate the amount of tokens before send and alert the user if they've hit the limit before sending.
The text was updated successfully, but these errors were encountered:
const{encode}=require('gpt-3-encoder')conststring=process.argv[2];console.log(string);constencoded=encode(string)console.log('# of tokens: ',encoded.length)
Compare results to using their tiktoken python module
importtiktokenenc=tiktoken.get_encoding("cl100k_base")
e=enc.encode("hi there bob")
print(len(e))
Currently the OpenAI integration (#577) does not send any details about the schema or data of the user in the prompt to OpenAI. We should explore sending the schema or at least database and table names along with the prompt which should result in more relevant KQL queries. This should be conditional and opt-out by default. The UX is unsolved here but maybe a checkbox in the header to include schema details works well enough?
Potential issues
Cost
The OpenAI API charges per 1k tokens sent (and more if the user uses gpt4). Including the schema or even just the table names in the prompt potentially introduces many more tokens than the user is anticipating. We need to avoid surprise charges from API use. We should at the very least warn the user about this possibility or at best estimate how many tokens will be sent before the request is actually made.
Token limits
There is a max of 4096 tokens for the API. Along with ☝️ we should estimate the amount of tokens before send and alert the user if they've hit the limit before sending.
The text was updated successfully, but these errors were encountered: