Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEATURE] Support remote inference on LTR plugin #27

Open
noCharger opened this issue Dec 13, 2023 · 0 comments
Open

[FEATURE] Support remote inference on LTR plugin #27

noCharger opened this issue Dec 13, 2023 · 0 comments
Labels
enhancement New feature or request

Comments

@noCharger
Copy link

noCharger commented Dec 13, 2023

Is your feature request related to a problem?

A clear and concise description of what the problem is, e.g. I'm always frustrated when [...].

As a developer, I want to have a large ML model (even LLMs) supported for remote inference purpose.

What solution would you like?

A clear and concise description of what you want to happen.

One direction is to integrate with the ml-common plugin, which already have such capability.

What alternatives have you considered?

A clear and concise description of any alternative solutions or features you've considered.

  1. Native support on LTR plugin to connect remote models directly. (Not just HTTP connections, could be RPC calls too)
  2. A ML node specifically for training and inference purpose within the cluster

Do you have any additional context?

Add any other context or screenshots about the feature request here.

Ref #26
Public doc https://opensearch.org/docs/latest/ml-commons-plugin/remote-models/index/

Trade-off understanding:

  • We understand that remote inference will involve extra latency and the use case is to support inference on very large ML models
@noCharger noCharger added enhancement New feature or request untriaged labels Dec 13, 2023
@getsaurabh02 getsaurabh02 moved this from 🆕 New to Later (6 months plus) in Search Project Board Aug 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
Status: Later (6 months plus)
Development

No branches or pull requests

1 participant