[Feature] Agent tracing Implementation

[yourbuddyconner]

We need a overarching implementation strategy to correlate agent spans together. In a web programming/microservices context this is often the Request_ID or Correlation_ID.

Summarized:

• If a request has no Correlation_ID header (i.e. this request comes from the user) generate one and add it to downstream requests.
• If a request does have a Correlation_ID header (i.e. this request comes from another microservice), use this value in downstream requests.

In this way, all spans can be correlated/loaded together in a tracing UI to get the full picture of request propagation and latency.

We have a slightly harder time of it because the system we're building has an entirely different architecture but we can still borrow this Correlation_ID concept and use it to our advantage.

In an ideal world, we would be able to implement some Correlation_ID that connects all spans relating to a particular message together. However, due to "The Way It Is" this proves difficult to do in practice, in particular because messages are batched together when relayed and potentially processed out of order.

Ideas for ways to correlate spans together:

• Use a random value for each polling interval for a given agent (this doesn't give us end-to-end correlation, but it does work internally to a given agent instance)
• Use values from a particular struct like CommittedMessage (Link)

[yourbuddyconner]

Implementation Details and Crate Deps Notes:

@prestwich Found this killer twitter thread that helps shed some light on the complex Rust Tracing ecosystem.

[prestwich]

There are 2 separate processes we want to track for each agent

1. Are updates being created and propagated successfully
2. Are messages reaching their destination

I'd suggest that we correlate the first on the Update struct and the second on the CommittedMessage struct?

[prestwich]

oh, now that the Dispatch event includes the current root, we can add that to the CommittedMessage struct and use that to correlate messages to updates

[prestwich]

opening an issue and PR for that

[prestwich]

#654