Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug]: FinishTool doesn't have a tool response so the agent loses its content #5154

Open
1 task done
enyst opened this issue Nov 21, 2024 · 2 comments · May be fixed by #5231
Open
1 task done

[Bug]: FinishTool doesn't have a tool response so the agent loses its content #5154

enyst opened this issue Nov 21, 2024 · 2 comments · May be fixed by #5231
Labels
bug Something isn't working fix-me-experimental

Comments

@enyst
Copy link
Collaborator

enyst commented Nov 21, 2024

Is there an existing issue for the same bug?

  • I have checked the existing issues.

Describe the bug and reproduction steps

Reported on slack. Regression from function calling.

FinishAction with source=agent is not the last thing that happens when running with UI or restoring session in CLI, but our backend doesn't give it a tool response. CodeAct "forgets" it, which makes other things not play nice.

Note: it happens with native function calling, but it probably happens with non-native too after we started converting them back and forth.

IMPORTANT:

  • find in the codebase and read codeact_agent.py, function_calling.py, fn_call_converter.py, llm.py.
  • the problem is in codeact agent, the other files are only for context on the processing that happens in the agent
  • the FinishAction with source=user doesn't have a tool response, because it just returns (which will end execution). Find and read the relevant code in codeact that checks for this action. Hint: at the start of step().
  • the FinishAction with source=agent is a tool to execute, but its execution is the same: it returns, and it doesn't get a tool response.

With this context, fix this:

  • either add a tool response, correctly linked to the tool call by id
  • or make sure its thought is sent to the llm as if it was a message action, and its tool id is never sent
  • note that in get_action_message() there are other odd cases; we can add these two.

IMPORTANT INFO:
We are fixing an AI agent. The agent sends multiple messages to an LLM API. That API and how it works are impossible for us to change! It is important that you do NOT hallucinate their behavior.

One requirement for tools is, for example, that a message that is a tool call, has a tool id, and is immediately followed by a tool response, which is linked to the previous message by tool id.

Additional Context

It will be addressed by this refactoring or before.

@enyst enyst added bug Something isn't working fix-me-experimental labels Nov 21, 2024
Copy link
Contributor

OpenHands started fixing the issue! You can monitor the progress here.

Copy link
Contributor

A potential fix has been generated and a draft PR #5231 has been created. Please review the changes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working fix-me-experimental
Projects
None yet
1 participant