[BUG] Intermittent Response code 403 (Forbidden)

[fababs]

Describe the bug
We are seeing that intermittently using the latest version, we are getting 403 error. We were in touch with Akeyless who say they do not see 403 errors in their logs. Mostly it is working but occasionally we see this error.

To Reproduce
un hashicorp/[email protected]
with:
url: ***
tlsSkipVerify: true
jwtTtl: 18000
token: ***
secrets: secret/data/Engineering_VideoPlatform/rnd-alpine/compliance/code_repos_credentials/generic_token Engineering_VideoPlatform/rnd-alpine/compliance/code_repos_credentials/generic_token | repo_access_token;

method: token
kubernetesTokenPath: /var/run/secrets/kubernetes.io/serviceaccount/token
exportEnv: true
exportToken: false
outputToken: false

env:
AKEYLESS_GW: ***
AKEYLESS_ECM_TOKEN: ***

Expected behavior
I would expect consistent behavior where the secret is systematically returned.

Log Output
##[debug]Loading inputs
##[debug]Evaluating: format('token={0}
##[debug]akeyless_token_username="${{token//./}}"
##[debug]echo "AKEYLESS_TOKEN_USERNAME=$akeyless_token_username" >> $GITHUB_ENV
##[debug]', inputs.token_username)
##[debug]Evaluating format:
##[debug]..Evaluating String:
##[debug]..=> 'token={0}
##[debug]akeyless_token_username="${{token//./}}"
##[debug]echo "AKEYLESS_TOKEN_USERNAME=$akeyless_token_username" >> $GITHUB_ENV
##[debug]'
##[debug]..Evaluating Index:
##[debug]....Evaluating inputs:
##[debug]....=> Object
##[debug]....Evaluating String:
##[debug]....=> 'token_username'
##[debug]..=> 'generic_token'
##[debug]=> 'token=generic_token
##[debug]akeyless_token_username="${token//./}"
##[debug]echo "AKEYLESS_TOKEN_USERNAME=$akeyless_token_username" >> $GITHUB_ENV
##[debug]'
##[debug]Result: 'token=generic_token
##[debug]akeyless_token_username="${token//./}"
##[debug]echo "AKEYLESS_TOKEN_USERNAME=$akeyless_token_username" >> $GITHUB_ENV
##[debug]'
##[debug]Loading env
Run token=generic_token
##[debug]/bin/bash --noprofile --norc -e -o pipefail /home/action-runner/_work/_temp/027df708-d8d0-4268-b684-f35543c4570b.sh
##[debug]AKEYLESS_TOKEN_USERNAME='generic_token'
##[debug]Finished: run
##[debug]Evaluating condition for step: 'run'
##[debug]Evaluating: (success() && (inputs.token_username != ''))
##[debug]Evaluating And:
##[debug]..Evaluating success:
##[debug]..=> true
##[debug]..Evaluating NotEqual:
##[debug]....Evaluating Index:
##[debug]......Evaluating inputs:
##[debug]......=> Object
##[debug]......Evaluating String:
##[debug]......=> 'token_username'
##[debug]....=> 'generic_token'
##[debug]....Evaluating String:
##[debug]....=> ''
##[debug]..=> true
##[debug]=> true
##[debug]Expanded: (true && ('generic_token' != ''))
##[debug]Result: true
##[debug]Starting: run
##[debug]Loading inputs
##[debug]Evaluating: format('echo {0}
##[debug]', env.AKEYLESS_TOKEN_USERNAME)
##[debug]Evaluating format:
##[debug]..Evaluating String:
##[debug]..=> 'echo {0}
##[debug]'
##[debug]..Evaluating Index:
##[debug]....Evaluating env:
##[debug]....=> Object
##[debug]....Evaluating String:
##[debug]....=> 'AKEYLESS_TOKEN_USERNAME'
##[debug]..=> 'generic_token'
##[debug]=> 'echo generic_token
##[debug]'
##[debug]Result: 'echo generic_token
##[debug]'
##[debug]Loading env
Run echo generic_token
##[debug]/bin/bash --noprofile --norc -e -o pipefail /home/action-runner/_work/_temp/3143d640-649f-4fd0-a279-2805c5152571.sh
generic_token
##[debug]Finished: run
##[debug]Evaluating condition for step: 'run'
##[debug]Evaluating: (success() && (inputs.token_username != ''))
##[debug]Evaluating And:
##[debug]..Evaluating success:
##[debug]..=> true
##[debug]..Evaluating NotEqual:
##[debug]....Evaluating Index:
##[debug]......Evaluating inputs:
##[debug]......=> Object
##[debug]......Evaluating String:
##[debug]......=> 'token_username'
##[debug]....=> 'generic_token'
##[debug]....Evaluating String:
##[debug]....=> ''
##[debug]..=> true
##[debug]=> true
##[debug]Expanded: (true && ('generic_token' != ''))
##[debug]Result: true
##[debug]Starting: run
##[debug]Loading inputs
##[debug]Evaluating: env.AKEYLESS_GW
##[debug]Evaluating Index:
##[debug]..Evaluating env:
##[debug]..=> Object
##[debug]..Evaluating String:
##[debug]..=> 'AKEYLESS_GW'
##[debug]=> ''
##[debug]Result: ''
##[debug]Evaluating: env.AKEYLESS_ECM_TOKEN
##[debug]Evaluating Index:
##[debug]..Evaluating env:
##[debug]..=> Object
##[debug]..Evaluating String:
##[debug]..=> 'AKEYLESS_ECM_TOKEN'
##[debug]=> ''
##[debug]Result: ''
##[debug]Evaluating: format('secret/data/Engineering_VideoPlatform/rnd-alpine/compliance/code_repos_credentials/{0} Engineering_VideoPlatform/rnd-alpine/compliance/code_repos_credentials/{1} | repo_access_token;
##[debug]', env.AKEYLESS_TOKEN_USERNAME, env.AKEYLESS_TOKEN_USERNAME)
##[debug]Evaluating format:
##[debug]..Evaluating String:
##[debug]..=> 'secret/data/Engineering_VideoPlatform/rnd-alpine/compliance/code_repos_credentials/{0} Engineering_VideoPlatform/rnd-alpine/compliance/code_repos_credentials/{1} | repo_access_token;
##[debug]'
##[debug]..Evaluating Index:
##[debug]....Evaluating env:
##[debug]....=> Object
##[debug]....Evaluating String:
##[debug]....=> 'AKEYLESS_TOKEN_USERNAME'
##[debug]..=> 'generic_token'
##[debug]..Evaluating Index:
##[debug]....Evaluating env:
##[debug]....=> Object
##[debug]....Evaluating String:
##[debug]....=> 'AKEYLESS_TOKEN_USERNAME'
##[debug]..=> 'generic_token'
##[debug]=> 'secret/data/Engineering_VideoPlatform/rnd-alpine/compliance/code_repos_credentials/generic_token Engineering_VideoPlatform/rnd-alpine/compliance/code_repos_credentials/generic_token | repo_access_token;
##[debug]'
##[debug]Result: 'secret/data/Engineering_VideoPlatform/rnd-alpine/compliance/code_repos_credentials/generic_token Engineering_VideoPlatform/rnd-alpine/compliance/code_repos_credentials/generic_token | repo_access_token;
##[debug]'
##[debug]Loading env
Run hashicorp/[email protected]
with:
url: ***
tlsSkipVerify: true
jwtTtl: 18000
token: ***
secrets: secret/data/Engineering_VideoPlatform/rnd-alpine/compliance/code_repos_credentials/generic_token Engineering_VideoPlatform/rnd-alpine/compliance/code_repos_credentials/generic_token | repo_access_token;

method: token
kubernetesTokenPath: /var/run/secrets/kubernetes.io/serviceaccount/token
exportEnv: true
exportToken: false
outputToken: false

env:
AKEYLESS_GW: ***
AKEYLESS_ECM_TOKEN: ***
RELEASE_VERSION: 23.3.2-0
EXTERNAL_REPO_SSH_KEY:
AKEYLESS_TOKEN_USERNAME: generic_token
ACTIONS_ALLOW_UNSECURE_COMMANDS: true
::group::Get Vault Secrets
Get Vault Secrets
::add-mask::***
::endgroup::
Error: Response code 403 (Forbidden)

Additional context
Add any other context about the problem here.

[fairclothjm]

Hello @fababs! I am sorry you are having trouble.

We were in touch with Akeyless who say they do not see 403 errors in their logs

I find this quite perplexing. Unfortunately, without more information we are unable to help you debug the issue.

That being said, the most common reason for a 403 status code is that the user lacks the necessary permissions to access the requested resource. This can mean that the user is not logged in, has not provided valid credentials, or does not belong to the appropriate user group to access the resource.

I see you are using the default token authentication for vault-action. Maybe you can try another auth type and see if that resolves the issue?

[fababs]

Thanks for the response.
So I'm sure that the user and the credentials are ok because the same code, same user is usually working, except that sometimes we are seeing these 403 errors. What I think is that it is connected to load and when we have a few workflows running in parallel on the same workers. We are seeing these errors particularly when there is more load.

[fairclothjm]

Do you see the same issue when you take Vault Action out of the picture and directly query Akeyless under load?

[fababs]

We are only using the vault action but I could write something and try and see if it is the aleyless server. Please just tell me which API on akeyless.

[fairclothjm]

Hi, unfortunately, I don't have any knowledge of akeyless or their API.

[fairclothjm]

Due to inactivity, we will close this issue. Please feel free to reopen if you feel this was a mistake, or if there is additional information to add. Thanks!

[Tjitse-E]

@fababs did you ever receive more info from Akeyless? We're dealing with the exact same problem, also using Akeyless.

[fababs]

@Tjitse-E So what seemed to work for us was to lengthen the expiration of the session token in Akeyless UI. Find your API Key and increase the JWT TTL (in minutes). Since we did this, we have not seen this issue.