Issues creating and revoking ServiceId

[jaleon-ot]

Hello,

Some error logs are gotten on the Vault server by creating and revoking ServiceIds. Having said this, a couple of questions:

1. Is there any extra permission missing on the API KEY configured?
2. Is there any path/endpoint used or available for revoking ServiceIds?
3. Is there any way we can know these issues comes from ibm cloud api unavailability? or Could this be from the network where Vault runs? Can we identify both scenarios?
4. Is there any retry setup for this kind of errors?

Extra information:
Roles are created using access_group_ids.
Error log by creating:
{"time":"2022-12-05T16:25:03.382879054Z","type":"response","auth":{"client_token":"hmac-...","accessor":"hmac-...","display_name":"cert-vault-auth-pcld-ctrlsv-o-pre","policies":["apps/pcld/ctrlsv/o/pre","default"],"token_policies":["apps/pcld/ctrlsv/o/pre","default"],"metadata":{"authority_key_id":"ed:65:22:b1:6a:29:7a:07:99:2f:d6:4b:99:85:b5:e0:f1:72:da:6f","cert_name":"pcld-ctrlsv-o-pre","common_name":"pcld-ctrlsv-o-pre","serial_number":"423517596186296840021220581620933172338016438562","subject_key_id":"e3:86:14:36:2c:ac:d2:09:59:0f:3a:b3:06:25:6c:04:a1:2e:29:35"},"entity_id":"83cc92a8-4f65-1873-9db8-c20cf7fc5c46","token_type":"service","token_ttl":2764800,"token_issue_time":"2022-12-05T16:25:02Z"},"request":{"id":"eb8dc720-7a5c-232a-f37b-cd30ebc317e3","operation":"read","mount_type":"vault-plugin-secrets-ibmcloud","client_token":"hmac-sha256:12584a695ad939f2e13efd00608452c93284403c4fa59a1f6b80130f3e61a6cb","client_token_accessor":"hmac-sha256:cbd2b23c22752af4eac5486cd6ee12ee3d2b12972c2ecd23e8d36a005bc52e38","namespace":{"id":"root"},"path":"ibmcloud/cxb-shs-dev_tst_pre/creds/pctrlsv-ibmcl-rg01-pre","remote_address":"7.153.11.196","remote_port":41228},"response":{"mount_type":"vault-plugin-secrets-ibmcloud"},"error":"1 error occurred:\n\t* Post \"https://iam.cloud.ibm.com/v1/serviceids\": read tcp 7.153.28.240:42122-\u003e10.119.255.228:8080: read: connection reset by peer\n\n"}

Error log by revoking:
{"@level":"error","@message":"An error occurred removing a service ID while revoking a secret lease. The service ID may have been manually deleted in IBM Cloud. The administrator should verify the service ID is removed.","@module":"secrets.vault-plugin-secrets-ibmcloud.vault-plugin-secrets-ibmcloud_af1a7fe0.vault-plugin-secrets-ibmcloud.vault-plugin-secrets-ibmcloud","@timestamp":"2022-12-07T09:16:17.819351Z","accountID":"d7c0049ab5b44b7797bbcccb0dc37b8e","deleteError":"Delete \"https://iam.cloud.ibm.com/v1/serviceids/ServiceId-5a69xxxx-xxxx-xxxx-xxxx-xxxx101cd1a7\": read tcp 7.153.28.240:33430-\u003e10.119.255.228:8080: read: connection reset by peer","serviceID":"ServiceId-5a69xxxx-xxxx-xxxx-xxxx-xxxx101cd1a7","timestamp":"2022-12-07T09:16:17.819Z","vaultRole":"ptophub-icmdb-rg01-pre"}

Thank you!

[smatzek]

Is there any extra permission missing on the API KEY configured?

The documented permissions are the only permissions required on the API key configured with the plugin. We test using a service ID granted only these permissions.

Is there any path/endpoint used or available for revoking ServiceIds?

I'm not sure if you are asking for paths/endpoints in Vault or IBM Cloud. There are APIs and CLIs for removing service IDs in IBM Cloud. For Vault you can use the CLI vault lease revoke and its corresponding API.

Is there any way we can know these issues comes from ibm cloud api unavailability? or Could this be from the network where Vault runs? Can we identify both scenarios?

I have not seen IBM Cloud API unavailability in dev/test of this. I have also tested this by running Vault locally from my home network without seeing connection resets. I'm not sure how to easily diagnose what hop in the network connection between your Vault and IBM Cloud is causing the connection reset.

Is there any retry setup for this kind of errors?

The Vault plugins, this one as well as secret engines for other clouds, generally don't do re-tries on their API calls back to their clouds.
In terms of Vault lease revokes, if I remember correctly Vault itself triggers retries on lease revokes if revocations fail during lease expiration.