Improve dataplane logs to allow better transfer process traceability

[bmg13]

Feature Request

Allow consistent request tracking across control and data plane within one Connector of one company.

Currently, there's a disconnect between logs in control and data plane which makes it hard to debug and trace requests when there's a lot of traffic in the EDC. This feature looks into improving request tracking by e.g. adding TransferProcess Identifiers to data plane logs.

Which Areas Would Be Affected?

Dataplane logs, specially related with transfer. Should not add any breaking change.

Why Is the Feature Desired?

Increased usability due to better log readability and transfer process observability.

Solution Proposal

In PipelineServiceImpl.java considering this log regarding transfer from a source to a destination can be changed to include the DataFlow Id, being the following an example of the new outcome.

DataFlow 7926e16e-7357-4032-8078-db08ecd9d67f Transferring from AmazonS3 to AmazonS3.

Additionally the error logs for unknown data source and unknown data sink can be updated to also improve traceability. The following are examples of those.

DataFlow 7926e16e-7357-4032-8078-db08ecd9d67f Unknown data source type AmazonS3.

DataFlow 7926e16e-7357-4032-8078-db08ecd9d67f Unknown data sink type AmazonS3.

[bmg13]

If the solution is okay and this issue is triaged, please feel free to assigned it to me.

[paullatzelsperger]

sounds reasonable, as long as the log is written at debug level, and the message is provided as a Supplier (not a String).

[bmg13]

sounds reasonable, as long as the log is written at debug level, and the message is provided as a Supplier (not a String).

The current level of the "Transfering" log will be kept as debug in this proposal and used as supplier as well. The concrete suggestion is from

monitor.debug(() -> format("Transferring from %s to %s.", request.getSourceDataAddress().getType(), request.getDestinationDataAddress().getType()));

to

monitor.debug(() -> format("DataFlow %s Transferring from %s to %s.",
request.getProcessId(), request.getSourceDataAddress().getType(), request.getDestinationDataAddress().getType()));

[bmg13]

@paullatzelsperger should I develop this and create the PR or just keep the issue and it will be picked up after the triage label is removed?

[ndr-brt]

looks good to me, feel free to open the PR