[Security Solution] AI Assistant: LLM Connector model chooser bug. New chat does not use connector's model (#199303)

[e40pud]

Summary

The PR fixes this bug

The issue happens with some of the locally setup LLMs (like Ollama) which requires the correct model to be passed as part of the chat completions API.

We had a bug in our code when on new conversation creation we would not pass all the connectors configuration and only connectorId and actionTypeId would be passed. Here is the old code implementation:

const newConversation = await createConversation({
  title: NEW_CHAT,
  ...(currentConversation?.apiConfig != null &&
  currentConversation?.apiConfig?.actionTypeId != null
    ? {
          apiConfig: {
            connectorId: currentConversation.apiConfig.connectorId,
            actionTypeId: currentConversation.apiConfig.actionTypeId,
            ...(newSystemPrompt?.id != null ? { defaultSystemPromptId: newSystemPrompt.id } : {}),
          },
        }
      : {}),
});

and thus the new conversation would not have the complete connector configuration which would cause to use default model (gpt-4o) as a model we pass to the LLM.

Also, I updated the default body that we use on the Test connector page, to make sure that we send a model parameter to the LLM in case of Open AI > Other (OpenAI Compatible Service) kind of connectors.

Testing notes

Steps to reproduce:

1. Install Ollama locally
2. Setup an OpenAI connector using Other (OpenAI Compatible Service) provider
3. Open AI Assistant and select created Ollama connector to chat
4. Create a "New Chat"
5. The Ollama connector should be selected
6. Send a message to LLM (for example "hello world")

Expected: there should be no errors saying ActionsClientChatOpenAI: an error occurred while running the action - Unexpected API Error: - 404 model "gpt-4o" not found, try pulling it first

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[neptunian]

Hey, I tried testing this locally (serverless security) using the directions provided and I get this when I chat with the assistant:

Error in handler TelemetryTracer, handleChainEnd: Error: Failed to validate payload coming from "Event Type 'invoke_assistant_success'": - []: excess key 'toolsInvoked' found

[e40pud]

Hey, I tried testing this locally (serverless security) using the directions provided and I get this when I chat with the assistant:

Error in handler TelemetryTracer, handleChainEnd: Error: Failed to validate payload coming from "Event Type 'invoke_assistant_success'": - []: excess key 'toolsInvoked' found

@neptunian does it happen consistently? Do you see same error when you use other LLMs?

@stephmilovic do you know what could be an issue here? It does not look like related to the changes in this PR, but maybe I'm wrong. Seems like that issue is related to this code?

[stephmilovic]

@stephmilovic do you know what could be an issue here? It does not look like related to the changes in this PR, but maybe I'm wrong. Seems like that issue is related to this code?

Yes, I've seen that on other branches. I have a note on my to do list to fix it, we can create an issue though.

[neptunian]

@neptunian does it happen consistently? Do you see same error when you use other LLMs?

I only tried it using llama3.2:latest. It was consistent and the only response I ever received when using the security assistant with it. If it's of any use, the observability AI assistant, after making local changes to pass in the model, did not have that issue, but its using the inference plugin.

[e40pud]

Thanks @neptunian, it seems we have some issue in security assistant.

cc @elastic/security-generative-ai For some reason llama3.2:latest returns a weird answer. I tested with llama:3.1 and everything works well. Maybe our prompts do not work well with that model?

When I send simple hello, it returns a next response:

{
  "action": "Final Answer",
  "action_input": {
    "query": "hello"
  }
}

which then being converted into "[object Object]" as an output result.

Here is the example of such a behaviour https://smith.langchain.com/o/b739bf24-7ba4-4994-b632-65dd677ac74e/projects/p/b9ebe1df-f5ad-4c26-bc57-e5e65994b91e?timeModel=%7B%22duration%22%3A%227d%22%7D&runtab=0&peek=5f3962c7-18fa-46de-9be6-57f56f8760e4

[e40pud]

After discussing this issue with the team, we decided to postpone the fix. Here is the ticket to track it #204261

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: c61b8f7

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
securitySolution	14.7MB	14.7MB	-57.0B
stackConnectors	659.2KB	659.5KB	+312.0B
total			+255.0B

History

• 💚 Build #259975 succeeded 5085ab5
• 💔 Build #259904 failed 2e45bf2
• 💔 Build #259790 failed 27d1d86
• 💔 Build #259768 failed a4ba1da

cc @e40pud

[stephmilovic]

@neptunian @e40pud

The excess key 'toolsInvoked' found error is fixed in this PR: #204280

[kibanamachine]

Starting backport for target branches: 8.16, 8.17, 8.x

https://github.com/elastic/kibana/actions/runs/12328503531

[kibanamachine]

💚 All backports created successfully

Status	Branch	Result
✅	8.16
✅	8.17
✅	8.x

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation