Customizing for local model

[nathasm]

I have ollama setup hosting the deepseek-v2 model. In my neovim config, I've kept the provider as openai but changed the endpoint to be my local ollama server.

What I'm finding is that while the context, instruction, and question are all being sent in the request payload, the responses I get back are just wildly wrong (e.g. Q: "What's this lua code do?" A: "Here's some python code for a web scraper"). I suspect it has to do with the message payload and/or prompts? I suspect that the prompt statements and maybe format need to be tweaked to get avante to work against other providers.

Curious what others are doing to combat this type of behavior. Do I need to configure the prompt via avanterules (for example modify the payload to match https://github.com/continuedev/continue/blob/3791774e45ae87821f6539f38da5c0752d63fe4f/core/llm/templates/edit.ts#L173)? Or do I need to create a custom vendor provider and override the payload construction functions.

Have others had success when dealing with non-standard or local models?

[peter-miroshnikov]

This is not an answer but more like a question if you had found a solution for this problem.
To me it looks like the ollama doesn't take the current file as context at all.

[nathasm]

No, I gave up and just used Claude

[jlodenius]

I had the same issue initially but not anymore, I think the solution was to remove the max_tokens argument from parse_curl_args, but tbh I'm not sure, might just have been solved by switching model to qwen-7b

[MaxOSchulte]

removing max_tokens solved it for me also. Also take a look at #850

[MaxOSchulte]

Works fine with opencoder:latest