Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Secrets cannot be pushed to the vault (external) in the data transfer request #1826

Open
FaheemBhatti opened this issue Mar 6, 2025 · 4 comments
Open

Secrets cannot be pushed to the vault (external) in the data transfer request #1826

FaheemBhatti opened this issue Mar 6, 2025 · 4 comments
Labels
bug Something isn't working

Comments

@FaheemBhatti
Copy link

FaheemBhatti commented Mar 6, 2025 •
edited by ndr-brt
Loading

Description

We have configured a central vault that is integrated with multiple TX-EDC instances. They successfully read and write secrets in the provider-case. However, in a transfer scenario, when the consumer attempts to push the relevant generated secret (EDR-tokens) to the vault, it fails with the logs below [1].

Even when roles are reversed (i.e., the provider is made the consumer), the same issue persists. The producer can push secrets, but the consumer is unable to do so.

This suggests that either a configuration setting or an issue in the EDC implementation is preventing the consumer from writing secrets to the vault. Extensive testing confirms that our vault configuration does not contain any errors or misconfigurations, indicating the issue lies within the EDC connector behavior or setup.

Steps to Reproduce

  1. Configure the central vault and verify that it is recognized by the EDC connector [2].
  2. Set up a provider connector and ensure it can push secrets to the vault.
  3. Initiate a transfer request where the consumer needs to push a generated secret to the vault.
  4. Observe that the consumer fails to push the secret, while the provider succeeds.
  5. Swap roles (make the provider a consumer) and repeat the process.
  6. Note that the issue persists regardless of role reversal.

Expected Behavior

The provider and consumer connectors should be able to push secrets to the vault as required.

Actual Behavior

The consumer cannot push secrets to the vault, while the provider can do so without issues.
Role reversal does not resolve the problem.

Impact

The issue prevents the successful completion of transfer requests, potentially blocking further development.
Security risks may arise if the required secrets cannot be stored appropriately.

Possible Causes

A configuration setting within the EDC connector may restrict the consumer’s ability to push secrets.
An underlying bug in the EDC implementation affects secret storage for consumer connectors.

Attachments

[1]

DEBUG 2025-02-26T09:59:37.361257198 TransferProcess 0656575f-bc80-4bd9-bf2b-35ffba6b2556 is now in state STARTED
SEVERE 2025-02-26T09:59:37.496805894 [EDR Receiver] Failed to process event TransferProcessStarted: Failed to set secret with status 404

[2]
Vault Configuration

  injector:
    enabled: true
  server:
    dev:
      enabled: false
      devRootToken: ""
    postStart: # must be set externally!
  hashicorp:
    url: "${vaultUrl}"
    token: ""
    timeout: 30
    healthCheck:
      enabled: false
      standbyOk: false
    paths:
      secret: /v1/secret/data/ssh-internal-test-entity-1
      health: /v1/sys/health

[3]
Provider Logs:

DEBUG 2025-02-27T09:19:12.522956157 Policy Definition created trial-usage-policy-v1
DEBUG 2025-02-27T09:19:24.563750522 [DataPlaneSelectorManagerImpl] DataPlaneInstance ssh-internal-test-entity-1-tractusx-connector-dataplane is now in state AVAILABLE
DEBUG 2025-02-27T09:19:44.014331975 Policy Definition created trial-access-policy-v1
DEBUG 2025-02-27T09:20:25.026951524 [DataPlaneSelectorManagerImpl] DataPlaneInstance ssh-internal-test-entity-1-tractusx-connector-dataplane is now in state AVAILABLE
DEBUG 2025-02-27T09:21:12.078014221 DSP: Incoming CatalogRequestMessage for class org.eclipse.edc.connector.controlplane.catalog.spi.Catalog process
DEBUG 2025-02-27T09:21:12.080009544 DIM Token expired, need to refresh.
DEBUG 2025-02-27T09:21:25.495155716 [DataPlaneSelectorManagerImpl] DataPlaneInstance ssh-internal-test-entity-1-tractusx-connector-dataplane is now in state AVAILABLE
DEBUG 2025-02-27T09:22:25.927895356 [DataPlaneSelectorManagerImpl] DataPlaneInstance ssh-internal-test-entity-1-tractusx-connector-dataplane is now in state AVAILABLE
DEBUG 2025-02-27T09:22:29.518181665 DSP: Incoming ContractRequestMessage for class org.eclipse.edc.connector.controlplane.contract.spi.types.negotiation.ContractNegotiation process
DEBUG 2025-02-27T09:22:29.846457222 [PROVIDER] ContractNegotiation 24041fab-3c07-47fd-98c4-9d2230a3d8e7 is now in state REQUESTED.
DEBUG 2025-02-27T09:22:30.387115126 [ProviderContractNegotiationManagerImpl] ContractNegotiation 24041fab-3c07-47fd-98c4-9d2230a3d8e7 is now in state AGREEING
DEBUG 2025-02-27T09:22:30.395427177 ContractNegotiation: ID 24041fab-3c07-47fd-98c4-9d2230a3d8e7. [Provider] send agreement
DEBUG 2025-02-27T09:22:32.595735755 ContractNegotiation: ID 24041fab-3c07-47fd-98c4-9d2230a3d8e7. [Provider] send agreement
DEBUG 2025-02-27T09:22:32.601472117 [ProviderContractNegotiationManagerImpl] ContractNegotiation 24041fab-3c07-47fd-98c4-9d2230a3d8e7 is now in state AGREED
DEBUG 2025-02-27T09:22:33.990168902 DSP: Incoming ContractAgreementVerificationMessage for class org.eclipse.edc.connector.controlplane.contract.spi.types.negotiation.ContractNegotiation process: 24041fab-3c07-47fd-98c4-9d2230a3d8e7
DEBUG 2025-02-27T09:22:34.206582332 [PROVIDER] ContractNegotiation 24041fab-3c07-47fd-98c4-9d2230a3d8e7 is now in state VERIFIED.
DEBUG 2025-02-27T09:22:34.529865523 [ProviderContractNegotiationManagerImpl] ContractNegotiation 24041fab-3c07-47fd-98c4-9d2230a3d8e7 is now in state FINALIZING
DEBUG 2025-02-27T09:22:34.533286215 ContractNegotiation: ID 24041fab-3c07-47fd-98c4-9d2230a3d8e7. [Provider] send finalization
DEBUG 2025-02-27T09:22:35.200038664 ContractNegotiation: ID 24041fab-3c07-47fd-98c4-9d2230a3d8e7. [Provider] send finalization
DEBUG 2025-02-27T09:22:35.204445312 [ProviderContractNegotiationManagerImpl] ContractNegotiation 24041fab-3c07-47fd-98c4-9d2230a3d8e7 is now in state FINALIZED
DEBUG 2025-02-27T09:22:35.800261802 DSP: Incoming TransferRequestMessage for class org.eclipse.edc.connector.controlplane.transfer.spi.types.TransferProcess process
DEBUG 2025-02-27T09:22:36.021637359 TransferProcess 5746ff7f-650d-4aaa-bb34-c6855a8bc9ba is now in state INITIAL
DEBUG 2025-02-27T09:22:36.629228066 [TransferProcessManagerImpl] TransferProcess 5746ff7f-650d-4aaa-bb34-c6855a8bc9ba is now in state PROVISIONING
DEBUG 2025-02-27T09:22:36.632513557 TransferProcess: ID 5746ff7f-650d-4aaa-bb34-c6855a8bc9ba. Provisioning
DEBUG 2025-02-27T09:22:36.635333955 [TransferProcessManagerImpl] TransferProcess 5746ff7f-650d-4aaa-bb34-c6855a8bc9ba is now in state PROVISIONED
DEBUG 2025-02-27T09:22:36.640538378 [TransferProcessManagerImpl] TransferProcess 5746ff7f-650d-4aaa-bb34-c6855a8bc9ba is now in state STARTING
DEBUG 2025-02-27T09:22:36.644220354 TransferProcess: ID 5746ff7f-650d-4aaa-bb34-c6855a8bc9ba. Initiate data flow
DEBUG 2025-02-27T09:22:37.124132025 TransferProcess: ID 5746ff7f-650d-4aaa-bb34-c6855a8bc9ba. send transfer start to https://ssh-internal-test-entity-2.c-27d7c36.kyma.ondemand.com/api/v1/dsp
DEBUG 2025-02-27T09:22:37.721826342 TransferProcess: ID 5746ff7f-650d-4aaa-bb34-c6855a8bc9ba. send transfer start to https://ssh-internal-test-entity-2.c-27d7c36.kyma.ondemand.com/api/v1/dsp
DEBUG 2025-02-27T09:22:37.726254402 [TransferProcessManagerImpl] TransferProcess 5746ff7f-650d-4aaa-bb34-c6855a8bc9ba is now in state STARTED
DEBUG 2025-02-27T09:22:37.729009535 [PolicyMonitorManagerImpl] PolicyMonitorEntry 5746ff7f-650d-4aaa-bb34-c6855a8bc9ba is now in state STARTED
DEBUG 2025-02-27T09:23:26.421139588 [DataPlaneSelectorManagerImpl] DataPlaneInstance ssh-internal-test-entity-1-tractusx-connector-dataplane is now in state AVAILABLE

Consumer Logs

DEBUG 2025-02-27T09:22:28.502433365 [ConsumerContractNegotiationManagerImpl] ContractNegotiation 0884e9b8-2324-485e-a1e1-83be6b018137 is now in state INITIAL
DEBUG 2025-02-27T09:22:29.116193541 [ConsumerContractNegotiationManagerImpl] ContractNegotiation 0884e9b8-2324-485e-a1e1-83be6b018137 is now in state REQUESTING
DEBUG 2025-02-27T09:22:29.118479353 ContractNegotiation: ID 0884e9b8-2324-485e-a1e1-83be6b018137. [Consumer] send request
DEBUG 2025-02-27T09:22:29.853328119 ContractNegotiation: ID 0884e9b8-2324-485e-a1e1-83be6b018137. [Consumer] send request
DEBUG 2025-02-27T09:22:29.857236556 [ConsumerContractNegotiationManagerImpl] ContractNegotiation 0884e9b8-2324-485e-a1e1-83be6b018137 is now in state REQUESTED
DEBUG 2025-02-27T09:22:31.935490343 DSP: Incoming ContractAgreementMessage for class org.eclipse.edc.connector.controlplane.contract.spi.types.negotiation.ContractNegotiation process: 0884e9b8-2324-485e-a1e1-83be6b018137
DEBUG 2025-02-27T09:22:32.589113221 [CONSUMER] ContractNegotiation 0884e9b8-2324-485e-a1e1-83be6b018137 is now in state AGREED.
DEBUG 2025-02-27T09:22:33.526501566 [ConsumerContractNegotiationManagerImpl] ContractNegotiation 0884e9b8-2324-485e-a1e1-83be6b018137 is now in state VERIFYING
DEBUG 2025-02-27T09:22:33.528853606 ContractNegotiation: ID 0884e9b8-2324-485e-a1e1-83be6b018137. [consumer] send verification
DEBUG 2025-02-27T09:22:34.209714199 ContractNegotiation: ID 0884e9b8-2324-485e-a1e1-83be6b018137. [consumer] send verification
DEBUG 2025-02-27T09:22:34.213095443 [ConsumerContractNegotiationManagerImpl] ContractNegotiation 0884e9b8-2324-485e-a1e1-83be6b018137 is now in state VERIFIED
DEBUG 2025-02-27T09:22:34.917026506 DSP: Incoming ContractNegotiationEventMessage for class org.eclipse.edc.connector.controlplane.contract.spi.types.negotiation.ContractNegotiation process: 0884e9b8-2324-485e-a1e1-83be6b018137
DEBUG 2025-02-27T09:22:35.196518122 [CONSUMER] ContractNegotiation 0884e9b8-2324-485e-a1e1-83be6b018137 is now in state FINALIZED.
DEBUG 2025-02-27T09:22:35.198020827 [TransferProcessManagerImpl] TransferProcess 33e9af61-db7b-4e07-b6eb-185f889a923e is now in state INITIAL
DEBUG 2025-02-27T09:22:35.198266064 Transfer with id TransferProcess{id='33e9af61-db7b-4e07-b6eb-185f889a923e', state=INITIAL, stateTimestamp=2025-02-27T09:22:35.197Z} initiated
DEBUG 2025-02-27T09:22:35.455266474 [TransferProcessManagerImpl] TransferProcess 33e9af61-db7b-4e07-b6eb-185f889a923e is now in state PROVISIONING
DEBUG 2025-02-27T09:22:35.459572171 TransferProcess: ID 33e9af61-db7b-4e07-b6eb-185f889a923e. Provisioning
DEBUG 2025-02-27T09:22:35.462380725 [TransferProcessManagerImpl] TransferProcess 33e9af61-db7b-4e07-b6eb-185f889a923e is now in state PROVISIONED
DEBUG 2025-02-27T09:22:35.467384535 [TransferProcessManagerImpl] TransferProcess 33e9af61-db7b-4e07-b6eb-185f889a923e is now in state REQUESTING
DEBUG 2025-02-27T09:22:35.470194244 TransferProcess: ID 33e9af61-db7b-4e07-b6eb-185f889a923e. send transfer request to https://ssh-internal-test-entity-1.c-27d7c36.kyma.ondemand.com/api/v1/dsp
DEBUG 2025-02-27T09:22:36.027359127 TransferProcess: ID 33e9af61-db7b-4e07-b6eb-185f889a923e. send transfer request to https://ssh-internal-test-entity-1.c-27d7c36.kyma.ondemand.com/api/v1/dsp
DEBUG 2025-02-27T09:22:36.029982617 [TransferProcessManagerImpl] TransferProcess 33e9af61-db7b-4e07-b6eb-185f889a923e is now in state REQUESTED
DEBUG 2025-02-27T09:22:37.483675107 DSP: Incoming TransferStartMessage for class org.eclipse.edc.connector.controlplane.transfer.spi.types.TransferProcess process: 33e9af61-db7b-4e07-b6eb-185f889a923e
DEBUG 2025-02-27T09:22:37.719637502 TransferProcess 33e9af61-db7b-4e07-b6eb-185f889a923e is now in state STARTED
SEVERE 2025-02-27T09:22:37.72420183 [EDR Receiver] Failed to process event TransferProcessStarted: Failed to set secret with status 404
DEBUG 2025-02-27T09:23:09.922612763 [DataPlaneSelectorManagerImpl] DataPlaneInstance ssh-internal-test-entity-2-tractusx-connector-dataplane is now in state AVAILABLE

[4]
Also, a new secret is added to the vault in the provider's secret data space

Image
@FaheemBhatti FaheemBhatti added bug Something isn't working triage all new issues awaiting classification labels Mar 6, 2025
@lgblaumeiser
Copy link
Contributor

Hi @FaheemBhatti , can you give some information on which version you observed the behavior?

@lgblaumeiser lgblaumeiser removed the triage all new issues awaiting classification label Mar 11, 2025
@FaheemBhatti
Copy link
Author

FaheemBhatti commented Mar 11, 2025 •
edited
Loading

@lgblaumeiser The edc version is 0.9.0-rc2, and the vault configurations in the values file is

vault:
  injector:
    enabled: true
  server:
    dev:
      enabled: false
      devRootToken: "{{VAULT_TOKEN}}"
    postStart: # must be set externally!
  hashicorp:
    url: "https://vault.c-27d7c36.kyma.ondemand.com"
    token: "{{VAULT_TOKEN}}"
    timeout: 30
    healthCheck:
      enabled: false
      standbyOk: false
    paths:
      secret: /v1/secret
      folder: ssh-internal-test-entity-1
      health: /v1/sys/health

We have also tried it with this configurations

vault:
  injector:
    enabled: true
  server:
    dev:
      enabled: false
      devRootToken: "{{VAULT_TOKEN}}"
    postStart: # must be set externally!
  hashicorp:
    url: "https://vault.c-27d7c36.kyma.ondemand.com"
    token: "{{VAULT_TOKEN}}"
    timeout: 30
    healthCheck:
      enabled: false
      standbyOk: false
    paths:
      secret: /v1/secret/data/ssh-internal-test-entity-1
      health: /v1/sys/health

@ndr-brt
Copy link
Contributor

ndr-brt commented Mar 11, 2025 •
edited
Loading

@FaheemBhatti could you share also the two connectors hashicorp related settings (edc.vault.hashicorp.* ones)

@FaheemBhatti
Copy link
Author

@ndr-brt Yess ofcourse,
The vault configurations for the connectors are

The producer connector

vault:
  injector:
    enabled: true
  server:
    dev:
      enabled: false
      devRootToken: "{{VAULT_TOKEN}}"
    postStart: # must be set externally!
  hashicorp:
    url: "https://vault.c-27d7c36.kyma.ondemand.com"
    token: "{{VAULT_TOKEN}}"
    timeout: 30
    healthCheck:
      enabled: false
      standbyOk: false
    paths:
      secret: /v1/secret/data/ssh-internal-test-entity-1
      health: /v1/sys/health

The consumer connector

vault:
  injector:
    enabled: true
  server:
    dev:
      enabled: false
      devRootToken: "{{VAULT_TOKEN}}"
    postStart: # must be set externally!
  hashicorp:
    url: "https://vault.c-27d7c36.kyma.ondemand.com"
    token: "{{VAULT_TOKEN}}"
    timeout: 30
    healthCheck:
      enabled: false
      standbyOk: false
    paths:
      secret: /v1/secret/data/ssh-internal-test-entity-2
      health: /v1/sys/health

With this configuration, the EDC can read and write the secret inside the vault, but the data transfer request somehow fails and provides the vault error in the logs.

Also, in our vault, the secret paths are

/v1/secret/data/ssh-internal-test-entity-1/data
/v1/secret/data/ssh-internal-test-entity-2/data

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@ndr-brt @lgblaumeiser @FaheemBhatti