feat: Enable multiple token caching and add option to disable caching…

[DaThumpingRabbit]

Description

Due to the previous PR #544 fixing the cache behavior, users are not able to use multiple credentials anymore with the same AVP

This new PR's goal is to fix that issue by adding the following features

• Tokens will now be cached on separate files identified by the connection method. Additionally, the approle and userpass method will have an additional identifier containing either the roleID or the username to allow users to reference multiple credentials if needed
• Addition of a command-line parameter to disable completely the token caching feature if the user needs to make a fresh login at every call

Fixes: #658

Checklist

Please make sure that your PR fulfills the following requirements:

• Reviewed the guidelines for contributing to this repository
• The commit message follows the Conventional Commits Guidelines.
• Tests for the changes have been updated
• Are you adding dependencies? If so, please run go mod tidy -compat=1.21 to ensure only the minimum is pulled in.
• Docs have been added / updated
• Optional. My organization is added to USERS.md.

Type of Change

• Bugfix
• Feature
• Code style update (formatting, local variables)
• Refactoring (no functional changes, no api changes)
• New tests
• Build/CI related changes
• Documentation content changes
• Other (please describe)

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 53.44828% with 27 lines in your changes missing coverage. Please review.

Project coverage is 72.91%. Comparing base (42a43f0) to head (b5c0bad).
Report is 16 commits behind head on main.

Files with missing lines	Patch %	Lines
pkg/utils/util.go	44.68%	17 Missing and 9 partials ⚠️
pkg/auth/vault/userpass.go	50.00%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #660      +/-   ##
==========================================
+ Coverage   72.16%   72.91%   +0.74%     
==========================================
  Files          26       26              
  Lines        1994     1606     -388     
==========================================
- Hits         1439     1171     -268     
+ Misses        458      326     -132     
- Partials       97      109      +12     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[pre]

Can't wait to see this merged soon!

[DaThumpingRabbit]

@werne2j @jkayani
Would you have time to look into that PR ? I think that lots of people might have been impacted by my last one to enable the cache feature

[sabinayakc]

@DaThumpingRabbit Just a question, does this improve the performance of the plugin? We are experiencing issues in our vault using approle method and vault seals itself. We have around 500 apps using this plugin via Argo. And we see millions of info logs shown below.

Wondering if token caching would help here.

2025-01-14T23:18:34.080Z [INFO]  expiration: revoked lease: lease_id=auth/approle/login/hb5224a3b526cf46546abe77d6e43d94efa5183fca0b38c
2025-01-14T23:18:34.080Z [INFO]  expiration: revoked lease: lease_id=auth/approle/login/h15ada71de47d5c65793165fd92f2774d313d9b81cf05

[DaThumpingRabbit]

@sabinayakc It should definitely improve performance as you will contact the vault instance less (as long as there is a valid token for the auth method you are using, it will be used instead of performing a new login)
This should improve both the performance of argo and the load on the vault instance

However, it will also depend on your configuration, if you have one approle per app (taking an extreme use case) with a very short TTL, it will do nothing for you
But if you have argocd refreshing 100 apps with the same approle during the time your token is valid, you will reuse it instead of making 100 logins

(please note that this PR is not available in any current release of the plugin for now, cf #666)

[jakuboskera]

We are experiencing this issue as well. When will be a new release available containing this fix? It has been merged 30th of September 2024 and not release yet. Thanks

[pre]

related

• Create a new release which includes PR #660 #666

[sabinayakc]

@jakuboskera @DaThumpingRabbit I went ahead and created a release on my fork since its a bit urgent.

https://github.com/sabinayakc/argocd-vault-plugin/releases/tag/v1.18.2

Update the following if you want to use it while we wait for this repo to update.

args:
          - >-
            wget -O argocd-vault-plugin
            https://github.com/sabinayakc/argocd-vault-plugin/releases/download/v${AVP_VERSION}/argocd-vault-plugin_${AVP_VERSION}_linux_amd64 &&
            chmod +x argocd-vault-plugin &&
            mv argocd-vault-plugin /custom-tools/

[jakuboskera]

@jakuboskera @DaThumpingRabbit I went ahead and created a release on my fork since its a bit urgent.

https://github.com/sabinayakc/argocd-vault-plugin/releases/tag/v1.18.2

Update the following if you want to use it while we wait for this repo to update.

args:
          - >-
            wget -O argocd-vault-plugin
            https://github.com/sabinayakc/argocd-vault-plugin/releases/download/v${AVP_VERSION}/argocd-vault-plugin_${AVP_VERSION}_linux_amd64 &&
            chmod +x argocd-vault-plugin &&
            mv argocd-vault-plugin /custom-tools/

Hmm I just tried it and it is not working as expected. My setup is Vault Enterprise with multiple namespaces (in this example /dev/foo and /dev/bar), JWT Kubernetes auth, one sidecar plugin.

Error:

# ConfigMap where should be replaced value of .data.title
argocd-repo-server-5c66cb68f5-2tb8q:~$ cat cm.yaml
apiVersion: v1
data:
  title: <title>
kind: ConfigMap
metadata:
  annotations:
    avp.kubernetes.io/path: kv/data/my-app
  name: my-app

# no cached token
argocd-repo-server-5c66cb68f5-2tb8q:~$ ls -al /tmp/.avp
total 8
drwxr-xr-x    2 999      root          4096 Feb 13 11:12 .
drwxrwxrwt    1 root     root          4096 Feb 13 11:08 ..

# First generate for /dev/foo namespace is ok as there is no cached vault token
argocd-repo-server-5c66cb68f5-2tb8q:~$ cat cm.yaml | VAULT_NAMESPACE=/dev/foo argocd-vault-plugin generate -
apiVersion: v1
data:
  title: /dev/foo
kind: ConfigMap
metadata:
  annotations:
    avp.kubernetes.io/path: kv/data/my-app
  name: my-app
---

# vault token cached for /dev/foo
argocd-repo-server-5c66cb68f5-2tb8q:~$ ls -al /tmp/.avp
total 12
drwxr-xr-x    2 999      root          4096 Feb 13 11:13 .
drwxrwxrwt    1 root     root          4096 Feb 13 11:08 ..
-rw-r--r--    1 999      root           133 Feb 13 11:13 kubernetes_config.json

# Second generate for /dev/bar namespace is not ok as cached token is used but this token was issued for /dev/foo namespace
argocd-repo-server-5c66cb68f5-2tb8q:~$ cat cm.yaml | VAULT_NAMESPACE=/dev/bar argocd-vault-plugin generate -
Error: Error making API request.

Namespace: /dev/bar
URL: GET https://vault/v1/kv/data/my-app
Code: 403. Errors:

* 1 error occurred:
	* permission denied


Usage:
  argocd-vault-plugin generate <path> [flags]

Flags:
  -c, --config-path string         path to a file containing Vault configuration (YAML, JSON, envfile) to use
      --disable-token-cache        disable the automatic token cache feature that store tokens locally
  -h, --help                       help for generate
  -s, --secret-name string         name of a Kubernetes Secret in the argocd namespace containing Vault configuration data in the argocd namespace of your ArgoCD host (Only available when used in ArgoCD). The namespace can be overridden by using the format <namespace>:<name>
      --verbose-sensitive-output   enable verbose mode for detailed info to help with debugging. Includes sensitive data (credentials), logged to stderr

# still cached vault token for /dev/foo
argocd-repo-server-5c66cb68f5-2tb8q:~$ ls -al /tmp/.avp
total 12
drwxr-xr-x    2 999      root          4096 Feb 13 11:13 .
drwxrwxrwt    1 root     root          4096 Feb 13 11:08 ..
-rw-r--r--    1 999      root           133 Feb 13 11:13 kubernetes_config.json
argocd-repo-server-5c66cb68f5-2tb8q:~$

# remove cached token for /dev/foo namespace
argocd-repo-server-5c66cb68f5-2tb8q:~$ rm -v /tmp/.avp/kubernetes_config.json
removed '/tmp/.avp/kubernetes_config.json'

# Third generate for /dev/bar namespace when cached token for /dev/foo was removed
argocd-repo-server-5c66cb68f5-2tb8q:~$ cat cm.yaml | VAULT_NAMESPACE=/dev/bar argocd-vault-plugin generate -
apiVersion: v1
data:
  title: /dev/bar
kind: ConfigMap
metadata:
  annotations:
    avp.kubernetes.io/path: kv/data/my-app
  name: my-app
---

# vault token cached for /dev/foo
argocd-repo-server-5c66cb68f5-2tb8q:~$ ls -al /tmp/.avp
total 12
drwxr-xr-x    2 999      root          4096 Feb 13 11:22 .
drwxrwxrwt    1 root     root          4096 Feb 13 11:08 ..
-rw-r--r--    1 999      root           133 Feb 13 11:22 kubernetes_config.json

However when I use --disable-token-cache it is working as expected, but I don't want to use it as it will slow down the performance.

[DaThumpingRabbit]

@jakuboskera indeed the kubernetes auth is stored in a single cache with that update because I did not see on what parameter to build the filename

I could work on another PR that would look for the VAULT_NAMESPACE environment variable and use that to build different cache files, it should solve your issue

I will try to work on this soon and I'll let you know

[jakuboskera]

Thanks @DaThumpingRabbit. For now I will use App Role which create separated cache files according ROLE ID which is for me ok now.

[DaThumpingRabbit]

@jakuboskera
I created this new PR #695 to configure caching per namespace when it is provided
You will have one file per auth method, per namespace and per identifier when possible (approle and user/pass)