diff --git a/docs/odh-rhoai/anythingllm/anythingllm-configure.png b/docs/odh-rhoai/anythingllm/anythingllm-configure.png new file mode 100644 index 00000000..3a2dbb46 Binary files /dev/null and b/docs/odh-rhoai/anythingllm/anythingllm-configure.png differ diff --git a/docs/odh-rhoai/anythingllm/anythingllm-first-workspace.png b/docs/odh-rhoai/anythingllm/anythingllm-first-workspace.png new file mode 100644 index 00000000..c3cc7e0c Binary files /dev/null and b/docs/odh-rhoai/anythingllm/anythingllm-first-workspace.png differ diff --git a/docs/odh-rhoai/anythingllm/anythingllm-get-started.png b/docs/odh-rhoai/anythingllm/anythingllm-get-started.png new file mode 100644 index 00000000..e7fc1d37 Binary files /dev/null and b/docs/odh-rhoai/anythingllm/anythingllm-get-started.png differ diff --git a/docs/odh-rhoai/anythingllm/anythingllm-just-me.png b/docs/odh-rhoai/anythingllm/anythingllm-just-me.png new file mode 100644 index 00000000..f65b531f Binary files /dev/null and b/docs/odh-rhoai/anythingllm/anythingllm-just-me.png differ diff --git a/docs/odh-rhoai/anythingllm/anythingllm-review.png b/docs/odh-rhoai/anythingllm/anythingllm-review.png new file mode 100644 index 00000000..ec0a769a Binary files /dev/null and b/docs/odh-rhoai/anythingllm/anythingllm-review.png differ diff --git a/docs/odh-rhoai/anythingllm/anythingllm-set-password.png b/docs/odh-rhoai/anythingllm/anythingllm-set-password.png new file mode 100644 index 00000000..3c2ed986 Binary files /dev/null and b/docs/odh-rhoai/anythingllm/anythingllm-set-password.png differ diff --git a/docs/odh-rhoai/anythingllm/anythingllm-skip-survey.png b/docs/odh-rhoai/anythingllm/anythingllm-skip-survey.png new file mode 100644 index 00000000..5a86eeb5 Binary files /dev/null and b/docs/odh-rhoai/anythingllm/anythingllm-skip-survey.png differ diff --git a/docs/odh-rhoai/anythingllm/anythingllm-start-chatting.png b/docs/odh-rhoai/anythingllm/anythingllm-start-chatting.png new file mode 100644 index 00000000..aa372550 Binary files /dev/null and b/docs/odh-rhoai/anythingllm/anythingllm-start-chatting.png differ diff --git a/docs/odh-rhoai/anythingllm/import-workbench-image.png b/docs/odh-rhoai/anythingllm/import-workbench-image.png new file mode 100644 index 00000000..d5f9d7c0 Binary files /dev/null and b/docs/odh-rhoai/anythingllm/import-workbench-image.png differ diff --git a/docs/odh-rhoai/custom-workbench-anythingllm.md b/docs/odh-rhoai/custom-workbench-anythingllm.md new file mode 100644 index 00000000..0b6a57f3 --- /dev/null +++ b/docs/odh-rhoai/custom-workbench-anythingllm.md @@ -0,0 +1,104 @@ +# Using AnythingLLM as Custom Workbench in ODH/RHOAI + +[AnythingLLM](https://anythingllm.com/) is the "All-in-one AI application. Any LLM, any document, any agent, fully private." It is a fully open-source, customizable chatbot that allows users to interact with various documents locally or on a hosted platform. It enables conversations with documents in formats like PDFs, TXT, and CSVs. The tool is particularly useful for creating private, localized versions of ChatGPT that can process and answer questions based on uploaded files. + +In this post, we'll explore how to deploy AnythingLLM on your Red Hat OpenShift AI instance to create a private chatbot for your internal users. AnythingLLM allows teams to securely interact with documents and knowledge bases by integrating large language models (LLMs) with private data in a controlled environment. Combining it with OpenShift’s robust platform for AI and its security features, we can set up AnythingLLM as an efficient internal chatbot. + +Whether your team needs to query PDFs, CSVs, or other document types, this guide will help you seamlessly bring AnythingLLM into your OpenShift AI infrastructure. + + +## Prerequisites + + +### 1. Running a Large Language Model + +To get started with AnythingLLM, you'll need an endpoint for a large language model (LLM) to interact with. AnythingLLM supports various models, so you're not limited to a single provider. You can deploy open-source models like Granite or Mistral, or use your own custom-trained LLM. To integrate the LLM, provide the endpoint URL and, if needed, an authentication token. +Alternatively, you can adopt a "Models as a Service (MaaS)" approach, allowing users to experiment with different models to find the best fit. For MaaS setup guidance, refer [here](https://github.com/rh-aiservices-bu/models-aas/). + + +### 2. Custom Workbench for AnythingLLM in OpenShift AI + +A workbench is your development and experimentation environment, similar to an IDE. OpenShift AI provides several default workbench images, but to use AnythingLLM, you'll need to add it as a custom workbench. This involves making the AnythingLLM container image available in your OpenShift AI environment for users to select in their projects. + +The AnythingLLM image is hosted on [Quay](http://quay.io/rh-aiservices-bu/anythingllm-workbench). You can view its details [here](https://github.com/rh-aiservices-bu/llm-on-openshift/tree/main/llm-clients/anythingllm). + +To make this image available cluster-wide by using the OpenShift AI user interface: + +- Log in to your OpenShift AI instance as `rhoai-admin`. +- From the left menu, navigate to Settings > Notebook Images > Import New Image. +- Enter `quay.io/rh-aiservices-bu/anythingllm-workbench` as the image location. + +For more guidance on building custom notebook images, check out [this blog post](https://ai-on-openshift.io/odh-rhoai/custom-notebooks/). + +## Configuration of AnythingLLM Workbench + +Now that you have the prerequisites set up, it’s time to dive in! In the following steps, we’ll walk you through creating a custom AnythingLLM workbench, configuring it to connect to an LLM, and getting hands-on with it. Let’s bring your ideas to life and explore AnythingLLM! + +### Create the Workbench + +Go to your Data Science project and click `Create a Workbench`. + +- **Name:** Choose a name, such as “AnythingLLM”. +- **Image selection:** Select your `custom-anythingLLM-image`. +- **Deployment size:** Choose `Small` (no GPU is needed for this setup). +- **Cluster storage:** It is to save your data for retrieval-augmented generation (RAG). Adjust the size based on your data needs. + +You can leave the rest of the settings as default and click Create. + +![import-workbench-image.png](./anythingllm/import-workbench-image.png) + +### Launch the Workbench + +Once the workbench is running, click `Open`. On the first launch, you’ll see a splash screen. Click on Get started. + +![anythingllm-get-started.png](./anythingllm/anythingllm-get-started.png) + +### Configure the LLM Endpoint + +We’ll connect the workbench to an LLM running in your environment. + +- **Provider:** Choose Generic OpenAI from the available options. +- **Base URL:** Enter the endpoint of your model, ending with /v1 (e.g., https://mistral-7b-instruct-v0-3-mycluster.com:443/v1). +- **API Key & Model Name:** Enter the required API key and specify the model name. +- **Token Context Window:** Set the context window size. This value may vary depending on the model you’re using. +- **Max Tokens:** Specify the maximum number of tokens to be generated by default. 1024 is good to start, and you can adjust this later in your workspaces. + +![anythingllm-configure.png](./anythingllm/anythingllm-configure.png) + +### Set Up User Access + +On the next screen, select `Just me`. OpenShift’s authentication ensures that only you can access your workbench anyhow, hence we select Just me. + +![anythingllm-just-me.png](./anythingllm/anythingllm-just-me.png) + +Then you’ll be prompted to set up a secondary password. Normally it’s not necessary as access to the workbench is already secured with OpenShift authentication. + +![anythingllm-set-password.png](./anythingllm/anythingllm-set-password.png) + +### Review Configuration + +You will see a summary screen displaying your settings. Confirm that everything looks correct. + +![anythingllm-review.png](./anythingllm/anythingllm-review.png) + +You might encounter a brief survey, which you can skip if you prefer. + +![anythingllm-skip-survey.png](./anythingllm/anythingllm-skip-survey.png) + +### Create Your First Workspace + +Next, you’ll set up a workspace, which acts as a project area within AnythingLLM. Each workspace can have its own settings and data, allowing you to organize different tasks or experiments separately. + +Click on `Create Workspace` to proceed. + +![anythingllm-first-workspace.png](./anythingllm/anythingllm-first-workspace.png) + +### Start Chatting + +The initial view will provide you with various information, but you can go straight to your workspace and begin interacting with the LLM. + +![anythingllm-start-chatting.png](./anythingllm/anythingllm-start-chatting.png) + + +There’s a lot you can do with AnythingLLM, so be sure to explore its features! For more details, check out the [documentation](https://docs.anythingllm.com/). + diff --git a/docs/whats-new/whats-new.md b/docs/whats-new/whats-new.md index 76ebdd46..e39cd193 100644 --- a/docs/whats-new/whats-new.md +++ b/docs/whats-new/whats-new.md @@ -1,5 +1,7 @@ # What's new? +**2024-10-14**: Add [Using AnythingLLM as custom workbench in ODH/RHOAI](../odh-rhoai/custom-workbench-anythingllm.md) instructions. + **2024-08-21**: Add [Model serving type modification](../odh-rhoai/model-serving-type-modification.md) instructions. **2024-07-29**: Add [Kueue preemption](../odh-rhoai/kueue-preemption/readme.md) example. diff --git a/mkdocs.yml b/mkdocs.yml index ed909b58..8f8c1c59 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -122,6 +122,7 @@ nav: - KServe Timeout Issues: odh-rhoai/kserve-timeout.md - RHOAI Metrics Dashboard for Model Serving: odh-rhoai/kserve-uwm-dashboard-metrics.md - Connect to RHOAI Workbench Kernel from VS Code: odh-rhoai/connect-vscode-to-rhoai-wb.md + - AnythingLLM as Custom Workbench: odh-rhoai/custom-workbench-anythingllm.md - Tools: - GPU pruner: odh-rhoai/gpu-pruner.md - ODH Tools and Extensions Companion: odh-rhoai/odh-tools-and-extensions-companion.md