Adding guide for product teams implementing tracing

.github/tracing/README.md

@@ -48,4 +48,49 @@ This folder contains the following config files:
 
 These config files are for an OTEL collector, grafana Tempo, and a grafana UI instance to run as containers on the same network.
 `otel-collector-dev.yaml` is the configuration for dev (i.e. your local machine) environments, and forwards traces from the otel collector to the grafana tempo instance on the same network. 
-`otel-collector-ci.yaml` is the configuration for the CI runs, and exports the trace data to the artifact from the github run. 
+`otel-collector-ci.yaml` is the configuration for the CI runs, and exports the trace data to the artifact from the github run.
+
+## Instrumenting Custom Traces
+
+Adding custom traces requires identifying an observability gap in a related group of code executions. This is intuitive for the developer:
+
+- "What's the flow of component interaction in this distributed system?"
+- "What's a ballpark latency of funcOne?"
+- "What's the behavior of the JobProcessorOne component when jobs with [x, y, z] attributes are processed?"
+
+Once an execution flow is desired to be traced, the developer will decide which parts of the execution flow should be measured separately. Each logically separate measure will be a span. Spans can have one parent span and multiple children span. The relations between parent and child spans will make a directed acyclic graph.
+
+The most trivial application of a span is measuring latency performance at high levels of granularity (individual func executions). There is much more you can do, including creating human readable and timestamped events within a span (useful for monitoring concurrent access to resources), recording errors, linking parent and children spans through large parts of an application, and even extending a span beyond a single process.
+
+Spans are created by tracers and passed through go applications by Contexts. A tracer must be initialized first. Both core and plugin developers will initialize a tracer from the globally registered trace provider:
+
+```
+tracer := otel.GetTracerProvider().Tracer("example.com/foo")
+```
+
+The globally registered tracer provider is available for plugins after they are initialized, and available in core after configuration is processed (`initGlobals`).
+
+Add spans by:
+```
+  func interestingFunc() {
+    // Assuming there is an appropriate parentContext
+	ctx, span := tracer.Start(parentContext, "hello-span")
+	defer span.End()
+
+	// do some work to track with hello-span
+  }
+```
+As implied by the example, span is a child span of its parent span captured by parentContext.
+
+
+Note that in certain situations, there are 3rd party libraries that will setup spans. For instance:
+
+```
+import (
+	"github.com/gin-gonic/gin"
+	"go.opentelemetry.io/contrib/instrumentation/github.com/gin-gonic/gin/otelgin"
+)
+
+router := gin.Default()
+router.Use(otelgin.Middleware("service-name"))
+```