Autodetect Feature?

[changchiyou]

Hello, a quick question here:

Does Litellm support something like autodetect? For instance, is it possible to set configs like below?

model_list:
  - model_prefix: "[openai] "
    litellm_params:
      model: openai/autodetect
      api_key: "myKey"
      api_base: "myBase"

[ishaan-jaff]

yes - wildcard routing @changchiyou https://docs.litellm.ai/docs/wildcard_routing

[changchiyou]

My config.yaml setup:

model_list:
  - model_name: "[trendmicro] *"
    litellm_params:
      model: openai/*
      api_key: os.environ/TRENDMICRO_API_KEY
      api_base: https://api.rdsec.trendmicro.com/prod/aiendpoint/v1

litellm_settings:
  drop_params: True
  success_callback: ["langfuse"]
  failure_callback: ["langfuse"]

general_settings:
  master_key: "os.environ/MASTER_KEY"

What I get from Open WebUI:

What I get	What I want

Is it possible?

[changchiyou]

I apologize for not explaining my earlier question clearly. My current setup involves using the pipelines framework provided by Open WebUI, where I write Python code and integrate it with the Open WebUI core:

python script:

"""
title: LiteLLM Manifold Pipeline
author: open-webui
date: 2024-05-30
version: 1.0.1
license: MIT
description: A manifold pipeline that uses LiteLLM.
"""

from typing import List, Union, Generator, Iterator
from schemas import OpenAIChatMessage
from pydantic import BaseModel
import requests
import os


class Pipeline:

    class Valves(BaseModel):
        LITELLM_BASE_URL: str = ""
        LITELLM_API_KEY: str = ""
        LITELLM_PIPELINE_DEBUG: bool = False

    def __init__(self):
        # You can also set the pipelines that are available in this pipeline.
        # Set manifold to True if you want to use this pipeline as a manifold.
        # Manifold pipelines can have multiple pipelines.
        self.type = "manifold"

        # Optionally, you can set the id and name of the pipeline.
        # Best practice is to not specify the id so that it can be automatically inferred from the filename, so that users can install multiple versions of the same pipeline.
        # The identifier must be unique across all pipelines.
        # The identifier must be an alphanumeric string that can include underscores or hyphens. It cannot contain spaces, special characters, slashes, or backslashes.
        # self.id = "litellm_manifold"

        # Optionally, you can set the name of the manifold pipeline.
        # self.name = "LiteLLM: "
        self.name = ""

        # Initialize rate limits
        self.valves = self.Valves(
            **{
                "LITELLM_BASE_URL": os.getenv(
                    "LITELLM_BASE_URL", "http://localhost:4001"
                ),
                "LITELLM_API_KEY": os.getenv("LITELLM_API_KEY", "your-api-key-here"),
                "LITELLM_PIPELINE_DEBUG": os.getenv("LITELLM_PIPELINE_DEBUG", True),
            }
        )
        # Get models on initialization
        self.pipelines = self.get_litellm_models()
        pass

    async def on_startup(self):
        # This function is called when the server is started.
        print(f"on_startup:{__name__}")
        # Get models on startup
        self.pipelines = self.get_litellm_models()
        pass

    async def on_shutdown(self):
        # This function is called when the server is stopped.
        print(f"on_shutdown:{__name__}")
        pass

    async def on_valves_updated(self):
        # This function is called when the valves are updated.

        self.pipelines = self.get_litellm_models()
        pass

    def get_litellm_models(self):

        headers = {}
        if self.valves.LITELLM_API_KEY:
            headers["Authorization"] = f"Bearer {self.valves.LITELLM_API_KEY}"

        if self.valves.LITELLM_BASE_URL:
            try:
                r = requests.get(
                    f"{self.valves.LITELLM_BASE_URL}/v1/models", headers=headers
                )
                models = r.json()
                return [
                    {
                        "id": model["id"],
                        "name": model["name"] if "name" in model else model["id"],
                    }
                    for model in models["data"]
                ]
            except Exception as e:
                print(f"Error fetching models from LiteLLM: {e}")
                return [
                    {
                        "id": "error",
                        "name": "Could not fetch models from LiteLLM, please update the URL in the valves.",
                    },
                ]
        else:
            print("LITELLM_BASE_URL not set. Please configure it in the valves.")
            return []

    def pipe(
        self, user_message: str, model_id: str, messages: List[dict], body: dict
    ) -> Union[str, Generator, Iterator]:
        if "user" in body:
            print("######################################")
            print(f'# User: {body["user"]["name"]} / {body["user"]["email"]}')
            print(f"# Message: {user_message}")
            print("######################################")

        headers = {}
        if self.valves.LITELLM_API_KEY:
            headers["Authorization"] = f"Bearer {self.valves.LITELLM_API_KEY}"

        try:
            payload = {**body, "model": model_id}

            payload.pop("chat_id", None)
            payload.pop("user", None)
            payload.pop("title", None)
            payload.pop("custom_metadata", None)

            if body.get('custom_metadata'):
                payload["metadata"] = body["custom_metadata"]

            r = requests.post(
                url=f"{self.valves.LITELLM_BASE_URL}/v1/chat/completions",
                json=payload,
                headers=headers,
                stream=True,
            )

            r.raise_for_status()

            if body["stream"]:
                return r.iter_lines()
            else:
                return r.json()
        except Exception as e:
            return f"Error: {e}"

However, even after calling the API f"{self.valves.LITELLM_BASE_URL}/v1/models" to retrieve all available APIs, I only receive [trendmicro] * instead of the actual APIs. Does this mean that even though I have referred to https://docs.litellm.ai/docs/wildcard_routing and configured wildcard routing, I still need to know the available endpoints in advance to call the corresponding models?

Maybe it's a good idea to key the api endpoint directly into Open Webui, but I really want the records in langfuse which is genereated by the callback from litellm. 😢

[changchiyou]

Seems related to #4903

[changchiyou]

@ishaan-jaff I quickly skimmed through the code and tested the current version of litellm (I personally believe that the documentation at Provider specific Wildcard routing is not detailed enough; developers must browse the source code to get a general understanding of the scope of this functionality). I found that whether setting model to * or {provider}/*, a corresponding provider API key must be set in order for litellm to list all models that the provider is expected to offer. However, this does not meet the requirement I mentioned earlier (I apologize again for not being more detailed in my question).

What I hope to achieve is to connect one litellm to another litellm and, after configuring it similarly to *, be able to obtain and display all available models listed in model_list using a request to /v1/models, rather than having to manually input each model into the configuration file one by one.