Add support for other base LLM models

[neubig]

Currently prompt2model only supports using OpenAI as the base LLM that is used for distillation, etc. It would be good to allow other models, including open models, as the base LLM.

One way we might be able to do this is by re-using general purpose inference code in libraries such as Zeno Build or litellm.

[Anindyadeep]

There are two ways to do this:

1. Either we host our base LLM and access it through a similar open ai like interface
2. Or, we directly access our LLMs. (which would require more memory)
   Which are we considering here (both?)

[neubig]

I think the answer is "both", and I have specific suggestions about how we could do so:

1. For the API-based generation, we transition to using litellm, which supports many different providers (including hitting hugging face APIs).
2. We create a meta-function for generating from LLMs that supports generating from either APIs or locally, like this one here (see generate_from_huggingface): https://github.com/zeno-ml/zeno-build/blob/d349ff54cb9eb984b8fdb198e8d09041ea9995e4/zeno_build/models/chat_generate.py#L30

[ishaan-jaff]

@neubig thanks for mentioning LiteLLM - i'm one of the maintainers of LiteLLM. Happy to make a PR for integrating😊. Will make one on the next 48 hrs

[saum7800]

Hey @ishaan-jaff ! Thank you so much for your comment. I had started working on this a day back. Happy to see you would like to contribute. Will you be integrating the LiteLLM part of the solution? Then I can focus on structuring the generation from huggingface models on top of that. Let me know, thanks!

[ishaan-jaff]

Pr here for tracking: #324

thanks @krrishdholakia!

[bilal-aamer]

Looks like there's a merged PR for this ticket, is this still standing for the integration of recent base LLMs?

[saum7800]

Hey @bilal-aamer ! We merged in LiteLLM to take care of this. Should we close the issue @neubig ?

[neubig]

Yes, I think this can be closed!