Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Why can't OTEL define a standard attribute for the span links relationship? #596

Open
rackerWard opened this issue Dec 8, 2023 · 4 comments
Assignees

Comments

@rackerWard
Copy link

In the https://opentelemetry.io/docs/specs/semconv/faas/aws-lambda/#sqs the SQS Event (receive) and SQS message (process) spans have span links added. The span links do not describe the relationship between the span and the remote context. How will the back-end system know what the correlation means?

What does OTEL expect the receive (SQS Event) span links to represent? What does OTEL expect the process (SQS Message) span link to represent?

If the graph follows both span links in the receive and process spans, how should it do this? Follow only the process (SQS Message) span link unless there isn't a process span for a specific SQS message, then follow the span link in the receive (SQS Event).

Would the receive (SQS Event) span links indicate a receipt of the message and is yet to be processed? Would the process (SQS Message) span link indicate processing the message?

Is this being done because there are different ways back-end systems are supporting message graphs?

What if a backend does not support span links? The span link data is useless. I have used a mechanism called a projection span to resolve this problem. The projection span is created as a child span on the SQS Message's context. The projection span is given span links to the process (SQS Message) span, however, the backend ignores this data. The two spans can be correlated by using an attribute containing a projection span ID. It is possible to search for either span and jump among the traces. The logical flow is client->projection span->process. The process span has the receive span as its real parent span. The projection span is not necessary if the backend supports span links. The projection span helps the backend build a service map. Without it, the backend fails to associate the two processes.

@pyohannes
Copy link
Contributor

There have been some recent changes to the messaging semantic conventions, and it seems the AWS semantic conventions aren't yet in line with those changes.

Instead of requiring two spans for "receive" and "process", there is only supposed to be either one "receive" (for pull-based operations) or "deliver" (for push-based operations) span. There are some examples here.

The span links do not describe the relationship between the span and the remote context. How will the back-end system know what the correlation means?

Usually you would look at the span kinds. If you have a link from a producer to a consumer span, then this link models a producer/consumer relationship.

Would the receive (SQS Event) span links indicate a receipt of the message and is yet to be processed? Would the process (SQS Message) span link indicate processing the message?

We tried to overcome this by clearly defining "receive" and "deliver" spans - both model the operation by which a message is passed to application code (from a messaging SDK), by pull or push-based means.

What if a backend does not support span links? The span link data is useless.

True, but then this backend also isn't OTel compliant.

@rackerWard
Copy link
Author

Thank you for pointing out the changes! At first glance, this might eliminate the need to provide a custom instrumentation wrapper when processing the batch in the client.

Is there a standard way to describe the span link's relationship? Describe if the linked span is a parent or a child or the direction of flow between the spans.

Why doesn't OTEL define attributes that describe the direction for span links? Does OTEL imply the direction based on the details of the span?
What if we wanted to use span links for other kinds of spans? How would a backend know which direction the flow should follow?
I was describing the relationship using a custom attribute but that doesn't work if a backend has other ways it determines the direction.

@pyohannes
Copy link
Contributor

Thank you for pointing out the changes! At first glance, this might eliminate the need to provide a custom instrumentation wrapper when processing the batch in the client.

Yes, that was a goal for these changes.

Is there a standard way to describe the span link's relationship? Describe if the linked span is a parent or a child or the direction of flow between the spans.

No, there isn't (yet). Currently, for messaging scenarios, you infer the flow from the span kinds: a link between a consumer and a producer span has a direction from the producer to the consumer.

Why doesn't OTEL define attributes that describe the direction for span links?

There have been discussions, but nobody starting to work on relate semantic conventions yet:
#1057

@github-actions github-actions bot added the Stale label Feb 7, 2024
@joaopgrassi
Copy link
Member

joaopgrassi commented Feb 14, 2024

@rackerWard was your initial question answered or you still need more clarification on this? Given there's already discussion about the link's relationship attributes happening in the spec (#1057), I'm leaning towards closing this issue. Let me know :)

@github-actions github-actions bot removed the Stale label Feb 21, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants