default maxTokens setting for autocomplete

[ferenci84]

Description

Default maxTokens at 256 for autocomplete if there is no overriding user setting for the model. Added autoCompleteMaxTokens

Change from this comment:
#3994 (comment)

Checklist

• The relevant docs, if any, have been updated or created - let's create a separate issue for this once the PR is approved and autoCompleteMaxTokens setting is kept in the completion options.
• The relevant tests, if any, have been updated or created - no relevant tests as far as I know

Testing instructions

I tested by directly putting a log message into the Ollama._streamFim() method.

[netlify]

✅ Deploy Preview for continuedev ready!

Name	Link
🔨 Latest commit	9d26892
🔍 Latest deploy log	https://app.netlify.com/sites/continuedev/deploys/67d73d7a954f600008b5f915
😎 Deploy Preview	https://deploy-preview-4448--continuedev.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[sestinj]

@ferenci84 rather than adding even more options to the config, I would much rather allow users to set maxTokens in the completionOptions section of their config for the model

[ferenci84]

The main part of this modification is not the new config option, but the ability to set different default for autocomplete (possibly for individual providers). autocompleteMaxTokens is a technical thing (to let us set default maxTokens for autocomplete for the model without knowing whether that model will be used for autocomplete or something different; and it also allows providers to set maxTokens setting specifically for autocomplete use), the question is whether to document or keep it hidden from users, that's why I didn't add it to the documentation without knowing whether the core team want users being able to tinker with this option or just keep it hidden and/or unavailable (there is an option for both, see my next comment).

[ferenci84]

@sestinj Please look at this: these are possible places to set defaults for maxTokens for autocomplete:

This is setting for global default:

There may also be defaults for individual models:

This is how the final maxTokens is set when making the request. As you can see, maxTokens user setting will always take precedence, the user don't even have to know about the extra key in the BaseCompletionOptions type:

There may be a misunderstanding, I think it's better if the additional key exist just in the type, but not open to users (i.e. hidden, not documented), however they should be used for setting individual defaults for providers, there may be fast and possibly more capable providers that can output more lines of completion, while slower, or those that tend to go into repetition, should be limited.

I believe that adding this key to the BaseCompletionOptions type is a good way to keep it simple for us, developers. If you want to keep it simple for users too, and limit their options, there is a way for us to ignore this additional setting that comes from the user config:

Any way, please let me know if you have an idea to make it better.