Allow models to not have a tag, don't add :latest automatically

[rustyrazorblade]

Validations

• I believe this is a way to improve. I'll try to join the Continue Discord for questions
• I'm not able to find an open issue that requests the same enhancement

Problem

OpenWebUI allows you to create custom models, but they don't have a :tag at the end, they just have an ID. If you don't put a tag, I believe continue will automatically add :latest to the end. I think that's here, but I'm not 100% sure.

Solution

It would be great if continue could support custom models created in OpenWebUI by letting the user specify models without a tag, and it wouldn't auto add :latest when sending requests to the api.

[sestinj]

@rustyrazorblade is there an example you could share of a model that isn't working? The code linked here is used for "blocks", which are a distinct concept in Continue, described a bit more here: https://docs.continue.dev/hub/blocks/intro

You should be able to use any model you'd like by hooking it up with the OpenAI-compatible config: https://docs.continue.dev/customize/model-providers/openai#openai-compatible-servers--apis

[rustyrazorblade]

Ah, ok. I don't really know the codebase, or typescript :)

Open WebUI also has open api compatible endpoints, they have a little tutorial here how to set it up with continue: https://docs.openwebui.com/tutorials/integrations/continue-dev

I created a model in Open Web UI called "Tech Writer". It automatically got assigned the id "tech-writer", I don't see a way this can be set manually. Hitting the /api/models endpoint, i see this as one of them:

{
      "id": "tech-writer",
      "name": "Tech Writer",
      "object": "model",
      "created": 1740635014,
      "owned_by": "ollama",

In my continue config.yaml, this is my setting:

models:
  - name: Writing
    provider: openai
    model: tech-writer
    useLegacyCompletionsEndpoint: false
    apiBase: http://localhost:3000/ollama/v1
    apiKey: "XXX"
    roles: [chat]

When I try to use the chat, I get a 400 error.

Note: Before I updated to the latest continue, I saw it mentioned that it couldn't find tech-writer:latest. The latest continue plugin doesn't show the specific error anymore, but I assume it's the same problem given the error code is the same. I'm not sure how to get the details on what went wrong in the latest request. I'm watching the error logs in Open WebUI and it is showing the 400 there too.

I ran another test, this time using the name of one of the ollama models, still going through Open WebUI. Using llama3.2:1b, everything started working.

I can use the model if I hit the

curl -X POST http://localhost:3000/api/chat/completions \
-H "Authorization: Bearer XXXXXX" \
-H "Content-Type: application/json" \
-d '{
      "model": "tech-writer",
      "messages": [
        {
          "role": "user",
          "content": "Write one sentence describing Apache Cassandra"
        }
      ]
    }'

So the model can be hit via the API (and works within their UI). I believe the error is most likely due to what I saw a few days ago, where ":latest" is being appended to the model's name.

[rustyrazorblade]

Ah, I found core.log, and it confirms that :latest is being appended to my model name before being sent to Open WebUI:

{"detail":"Model 'tech-writer:latest' was not found"}
    at customFetch (/snapshot/continue/binary/out/index.js:531387:17)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async withExponentialBackoff (/snapshot/continue/binary/out/index.js:531128:22)
    at async OpenAI._streamChat (/snapshot/continue/binary/out/index.js:532001:22)
    at async OpenAI.streamChat (/snapshot/continue/binary/out/index.js:531604:26)
    at async llmStreamChat (/snapshot/continue/binary/out/index.js:579138:19)
    at async /snapshot/continue/binary/out/index.js:579826:30