Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Resolver] Support custom env vars for Openhands agents #5156

Open
wants to merge 16 commits into
base: main
Choose a base branch
from

Conversation

malhotra5
Copy link
Contributor

@malhotra5 malhotra5 commented Nov 21, 2024

End-user friendly description of the problem this fixes or functionality that this introduces

  • Include this change in the Release Notes. If checked, you must provide an end-user friendly description for your change below

Give a summary of what the PR does, explaining any non-trivial design decisions

This PR allows users to pass custom env variables to the openhands runtime. This is tough to achieve in Github actions natively as Github doesn't allow for dynamic manipulation of secrets. How this PR works -

  1. User creates a repo secret with the prefix OPENHANDS_ENV_
  2. ALL secrets are passed to the reusable workflow
  3. Secrets are filtered by prefix OPENHANDS_ENV_ and replaced with SANDBOX_ENV_ (this allows openhands agents to have access to env vars in its runtime)

Pros/Cons

This is quite simple for the user to setup. The downside is ALL secrets are passed to the reusable workflow, which isn't ideal in terms of granularity.

Sample Test

Issue addressed
- This issue requires openhands to run an api with the custom env variable. Openhands would fail unless the env var exists in its runtime.
Workflow logs

Alternative approaches

1: Filter before sending

  1. The caller workflow would require a separate job to filter secrets by prefix (the "env job")
  2. The caller would encrypt the secrets
  3. env job would pass the encrypted value (through its output, which makes the encrypted value visible) to the job which invokes the reusable workflow
  4. The reusable workflow is invoked with the encrypted secrets
  5. Reusable workflow will decrypt the secrets and pass them to Openhands

We need an encryption scheme as github actions don't support a native way to pass secrets between jobs (and filtering secrets cannot be performed inside the job calling the reusable workflow).

I'm thinking symmetric key encryption using GNU Privacy Guard would work to achieve this. The user would need to perform just one additional step to setup a PASSPHRASE in their repo secrets.

2: Key store

Add support for custom env vars via integrations to external key stores. For example, AWS Secrets Key manager.


Link of any specific issues this addresses
#5132

@malhotra5 malhotra5 changed the title [Resolver] Support custom environment variables for Openhands agents [Resolver] Support custom env vars for Openhands agents Nov 21, 2024
@neubig neubig self-requested a review November 21, 2024 03:40
@neubig neubig self-assigned this Nov 21, 2024
Copy link
Contributor

@neubig neubig left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you please document this functionality?

Also, what do you mean by "the downside is all secrets are passed to the reusable workflow"? I thought that only ones that start with OPENHANDS_ENV are passed.

@malhotra5
Copy link
Contributor Author

Could you please document this functionality?

Got it will update this PR with documentation.

Also, what do you mean by "the downside is all secrets are passed to the reusable workflow"? I thought that only ones that start with OPENHANDS_ENV are passed.

The changes introduced in openhands/resolver/examples/openhands-resolver.yml passes toJSON{{ secrets }} (the stringified representation of ALL repo secrets) for the ENV_CONFIG parameter in the reusable workflow. We filter every secret contained inside ENV_CONFIG by the prefix OPENHANDS_ENV in the reusable workflow.

Ideally, the filtering is performed inside openhands/resolver/examples/openhands-resolver.yml before it is passed to the reusable workflow (but it's tough to achieve).

Now that I think about it, this PR doesn't address the fact that the resolver has access to ALL repo secrets at some point (either in the example or reusable workflow). It would make sense to migrate to using environment secrets. This will ensure openhands/resolver/examples/openhands-resolver.yml only has access to secrets inside its environment, as opposed to having access to all repo secrets.

@malhotra5 malhotra5 marked this pull request as draft November 21, 2024 22:57
@malhotra5
Copy link
Contributor Author

malhotra5 commented Nov 22, 2024

Correction: environment secrets don't solve the problem of guaranteeing openhands-resolver doesn't receive access to any secrets other than the few it needs.

If the user doesn't want openhands-resolver to have access to certain secrets, it becomes their responsibility to store it inside an environment.

Thus, it also doesn't matter whether we process OPENHANDS_ENV prefix in the caller or called workflow (they're functionally the same, the latter being easier to implement with guarantee that secrets aren't exposed in workflow logs ).

I believe the approach on this PR should suffice; will update with documentation shortly

@malhotra5 malhotra5 marked this pull request as ready for review November 22, 2024 01:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants