OpenTelemetry Support (#101)

* Initialize OpenTelemetry SDK * Create OpenTelemetry root and middleware spans * Add exit handlers for otel * Set traceparent and tracestate on the active otel context * Add otel imports to prelude * Refactor grpc switching logic * Use isHoneycomb/isOtel everywhere, including prelude * Export beeline by default * Use OpenTelemetry semantic attributes * Move isHoneycomb/isOtel tests to prelude * Replace type guards with unsafe casts, yolo * Configure sampler * Switch from @opentelemetry/node to @opentelemetry/sdk-trace-node * Fix deprecation warning * Use individual node instrumentations * Change default port to 8000 * Update dependencies.ron, clean up unused dependency * Add @opentelemetry/api dependency * set up W3CTraceContextPropagator * Pretest hook to check typescript types * Simplify ts check script * Break otel functionality into unit-test-able functions * TODOs for instrumentation modules * Wrap honeycomb env var normalization in a function * Clean up tap setup in prelude * Remove a bunch of crufty instrumentations and respect postgres + redis feature flags * a bunch of otel unit tests * t.equals -> t.equal (the former is deprecated) * fix: Add v to version for error-produced urls The url structure is `/v${versionNumber}/`, so these errors produce 404s * whack-a-mole deprecated tap APIs * Don't import middleware/test in prelude tests * Resolve promise on error when closing parent span * Remove testing stubs from honeycomb middleware * Fix template bug * Fix syntax error in dependencies.ron * push honeycomb into core module * Move honeycomb core code out of prelude.ts * build succeeds! but beelines are broken in the middleware layer, womp womp * Don't try to open/close spans when honeycomb isn't even initialized * SMOKE TESTS PASSING * Remove otel functionality to create more manageable pull request * Remove otel deps * Test the honeycomb trace span name logic * boltzmann-cli npm package will now run x64 darwin build on arm * Start the trace *after* handling the ping route * Update middleware.ts to use honeycomb abstraction * Revert "Remove otel deps" This reverts commit 11fb7ef. * Revert "Remove otel functionality to create more manageable pull request" This reverts commit c2e98a2. * rename otlp to otel throughout * Remove search querystring from trace span names * Use HTTP_CLIENT_IP instead of NET_PEER_IP * Set span kind to SERVER * Clean up trace propagation code * Fix W3CTraceContextPropagator import * update core/middleware to use honeycomb facade API * Failing test for honeycomb/otel instrumentation * Respect doNotTrace and fill in test asserts * Attempt to plumb up contexts better * spruced up the test trace propagator * Extra log lines for honeycomb tracing (parent/child spans not connecting in beelines) * Refactor honeycomb mocking - tests should all use the same mock object now * Move span/trace up/down logic back into middleware * Traces are now connecting and have the correct service.name * ignore noisy http auto-instrumentation * WE HAVE INWARD PROPAGATION * rename otelAPI to otel (for convenience) * Minor cleanup/refactors * remove top-level trace span in non-test mode * Remove explicit trace context propagator (w3c works automatically) * Remove spans array and use closures instead * nudge span context property def to make js build happy * Fill in otel trace URL params * Use bole for logging in honeycomb middleware * honeycomb-related logging uses bole when available * type for logger * Organization/cleanup of core/honeycomb.ts * Refactor how honeycomb mock is created * Use the traceparent header for the context ID if available * context.addTraceAttributes method * scope otel "trace" attributes to "app." + note on traces vs spans * Fix setting otel "trace" attributes - strike that, reverse it * Upgrade actions from node 12.x to 16.x * Use node action v2 * Only add properties to otel trace span if defined * Fixes to make dispensary-dashboard build happy * Update honeycomb-beeline.d.ts in build * Make dispensary-dashboard happy with honeycomb-beeline.d.ts formatting * Naively upgrade typescript-eslint libraries to fix huge warning in consumer-app-gateway * Improved comments * Add some extra trace attributes * Refactor "trace attributes" feature * use otel.context.with instead of carrying spans on boltzmann context * Wrap honeycomb logging setup in if-block + add deprecation warning * Minor otel middleware bugfixes * Remove addTraceAttributes helper * Improve logging * Automatically append service_name and trace_type to every span * Silly templating bug * Test that traceExporter respects url * Improved otel logging * imports/exports match between honeycomb + prelude * Mock the span in the test instead of as a fallback * Fix honeycomb export * Honeycomb logging only generated in debug/dev build * Honeycomb logging fixes * Update documentation for OpenTelemetry support * Update examples to use current release build * Bump to 0.6.0 and fill out changelog fields * Expand middleware test to cover handlers * Regenerate all examples * Respect the same defaults as vanilla beelines * Don't write debug feature to package.json * Manage node version with volta (and also template it in the github action!) * Switch from GRPC exporter to HTTP exporter - need to update env vars * Switch on OTEL_* env vars instead of HONEYCOMB_API_HOST * Use otel diagnostic logger * Clean up SDK setup, get rid of errors * Button up otel logging * Remove debug flag from boltzmann generator * Support ALL THREE OTLP trace exporters * Improved comments * More comments * Only do otel logging if LOG_LEVEL === "debug" * Improved comments for OTEL_EXPORTER_OTLP_ENDPOINT * one OVER sampling rate * bugfix - use proto exporter when protocol is http/protobuf * Let otel use the automatically-configured sampler * Make HONEYCOMB_WRITEKEY optional for otel * Fix volta feature bug * Respect OTEL_SERVICE_NAME * More tests for parseEnv * Move honeycomb singleton out of prelude * Don't run honeycomb init inside test block, oops * Rename the default otel http span * Remove comment * Add span attributes for CORS origin * Make honeycomb settings logging more useful for beelines * Set the initialized flag when beeline * Minor logging tweaks * Some gaps on beeline trace_type/service.name attributes * Rename boltzman.honeycomb.trace_type to honeycomb.trace_type * Use {{ version }} instead of hard-coding 1.0 * Remove stale comment * Update stale test description * Remove stale comment * Update honeycomb middleware configuration docs * Tweaks to honeycomb middleware docs * Wrap honeycomb tracing in cors with template if block * Fix tests * fix typed example build * Remove ctor def in span processor + comment onEnd * Comment update Co-authored-by: Chris Dickinson <[email protected]> * Improved documentation and comments * Use INTERNAL SpanKind instead of SERVER * Document factory overrides as internal * Remove graceful shutdown code * Expose boltzmann version in /monitor/status * Remove compatibility + trace_type attributes * Install volta during tests * Try a different strategy to add volta to the PATH * why isnt volta working?? * Try running "volta install node" * Try installing volta during the build step * Remove volta from --all * Clean up release build task * updated package.json * Handle peer dependencies shenanigans as best we can Co-authored-by: Jeffrey Lembeck <[email protected]> Co-authored-by: Chris Dickinson <[email protected]>
entropic-dev · Mar 29, 2022 · 9ae7c0c · 9ae7c0c
1 parent 75cd8c5
commit 9ae7c0c
Show file tree

Hide file tree

Showing 51 changed files with 14,382 additions and 7,188 deletions.
diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml
@@ -83,7 +83,7 @@ jobs:
       - uses: actions/checkout@v2
       - uses: actions/[email protected]
         with:
-          node-version: '12'
+          node-version: '16'
       - name: download release contents
         run: |
           tag=${{ github.ref }}
@@ -128,7 +128,7 @@ jobs:
           const fs = require('fs')
           const spawnSync = require('child_process').spawnSync
 
-          const executable = path.join(__dirname, 'boltzmann_' + os.arch() + '_' + os.platform()) + (os.platform() === 'win32' ? '.exe' : '')
+          const executable = path.join(__dirname, 'boltzmann_' + (os.platform() === 'darwin' ? 'x64' : os.arch()) + '_' + os.platform()) + (os.platform() === 'win32' ? '.exe' : '')
 
           if (!fs.existsSync(executable)) {
             console.error("Sorry, boltzmann is not yet supported on %s", os.platform())

diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml
@@ -19,14 +19,15 @@ jobs:
     steps:
       - uses: actions/checkout@v2
 
-      - name: Use Node.js 12.x
+      - name: Use Node.js ${{ matrix.node_version }}
         uses: actions/[email protected]
         with:
           node-version: ${{ matrix.node_version }}
 
       - uses: actions-rs/toolchain@v1
         with:
           toolchain: stable
+
       - uses: actions/cache@v2
         if: runner.os != 'macOS'
         with:

diff --git a/Cargo.lock b/Cargo.lock
diff --git a/Cargo.toml b/Cargo.toml
@@ -2,7 +2,7 @@
 authors = ["C J Silverio <[email protected]>", "Chris Dickinson <[email protected]>"]
 edition = "2021"
 name = "boltzmann"
-version = "0.5.3"
+version = "0.6.0"
 documentation = "https://www.boltzmann.dev/en/docs/latest/"
 description = "A scaffolder for Boltzmann.js http service projects."
 license = "Apache-2.0"

diff --git a/HACKING.md b/HACKING.md
@@ -36,6 +36,10 @@ The files in `templates/boltzmann/` include inline `tap` tests. You can run
 `npm ci; npm t` to run those tests. This tests the files in-situ. Integration
 tests for the compiled scaffold can be run using the `bin/test` tool.
 
+You can also check whether the TypeScript will compile by running
+`bin/checkts.sh`, which unlike `bin/buildjs.sh` will merely check that the
+code is valid TypeScript.
+
 When writing a module in `templates/boltzmann`, keep the following in mind:
 
 - All modules are concatenated inline using `templates/boltzmann/index.tera`.
@@ -85,7 +89,7 @@ Dependencies are controlled by `src/runscripts.ron`.
 You should have:
 
 - Rust installed (use [rustup])
-- node 14+
+- node 16+
 
 Check the project out and run `npm ci`.
 

diff --git a/README.md b/README.md
@@ -33,7 +33,7 @@ For example, to scaffold with the defaults:
 projects|⇒ npx boltzmann-cli hello
 ```
 
-A complete project is provided for you, with useful package run scripts and linting. To run: `./boltzmann.js`. And to view the response: `curl http://localhost:5000/hello/world`. Want to know more? [Check the docs!](https://www.boltzmann.dev/en/docs/latest/)
+A complete project is provided for you, with useful package run scripts and linting. To run: `./boltzmann.js`. And to view the response: `curl http://localhost:8000/hello/world`. Want to know more? [Check the docs!](https://www.boltzmann.dev/en/docs/latest/)
 
 ## Team
 

diff --git a/bin/checkts.sh b/bin/checkts.sh
@@ -0,0 +1,11 @@
+#!/bin/bash
+
+set -e
+
+if [ ! -e node_modules ]; then
+  npm ci
+else
+  npm i
+fi
+
+node_modules/.bin/tsc --noEmit --project ./tsconfig-build-js.json
diff --git a/docs/content/reference/01-cli.md b/docs/content/reference/01-cli.md
@@ -397,12 +397,26 @@ boltzmann.d.ts  handlers.ts    node_modules/  package.json  tests/
 boltzmann.js*   middleware.ts  nodemon.json   target/       tsconfig.json
 ```
 
+#### `--volta`
+
+{{ changelog(version="0.6.0" }}
+
+Runs [volta](https://volta.sh/) in the project, pinning the Node.js version to
+v16.
+
+**Example use:**
+
+```shell
+$ npx boltzmann-cli . --volta
+```
+
 ## Full usage
 
 The current output of `npx boltzmann-cli help`:
 
 ```shell
-boltzmann 0.3.0
+boltzmann 0.6.0
+C J Silverio <[email protected]>, Chris Dickinson <[email protected]>
 Generate or update scaffolding for a Boltzmann service.
 To enable a feature, mention it or set the option to `on`.
 To remove a feature from an existing project, set it to `off`.
@@ -412,39 +426,92 @@ boltzmann my-project --redis --website
 boltzmann my-project --githubci=off --honeycomb --jwt
 
 USAGE:
-    boltzmann [FLAGS] [OPTIONS] [destination]
-
-FLAGS:
-        --all         Enable everything!
-        --docs        Open the Boltzmann documentation in a web browser
-        --force       Update a git-repo destination even if there are changes
-    -h, --help        Prints help information
-    -q, --quiet       Suppress all output except errors
-        --selftest    Build for a self-test
-    -s, --silent      Suppress all output except errors
-    -V, --version     Prints version information
-    -v, --verbose     Pass -v or -vv to increase verbosity
-        --website     Enable website feature set (templates, csrf, staticfiles, jwt, livereload, ping, status)
+    boltzmann [OPTIONS] [DESTINATION]
+
+ARGS:
+    <DESTINATION>
+            The path to the Boltzmann service
+
+            [default: ]
 
 OPTIONS:
-        --csrf <csrf>                  Enable csrf protection middleware
-        --esbuild <esbuild>            Enable asset bundling via ESBuild
-        --githubci <githubci>          Enable GitHub actions CI
-        --honeycomb <honeycomb>        Enable tracing via Honeycomb
-        --jwt <jwt>                    Enable jwt middleware
-        --livereload <livereload>      Enable live reload in development
-        --oauth <oauth>                Enable OAuth
-        --ping <ping>                  Enable /monitor/ping liveness endpoint; on by default
-        --postgres <postgres>          Enable postgres
-        --redis <redis>                Enable redis
-        --staticfiles <staticfiles>    Enable static file serving in development
-        --status <status>              Enable /monitor/status healthcheck endpoint; on by default
-        --templates <templates>        Enable Nunjucks templates
-        --typescript <typescript>      Scaffold a project implemented in TypeScript
+        --all
+            Enable everything (mostly for testing)
 
-ARGS:
-    <destination>    The path to the Boltzmann service [default: ]
+        --csrf [<CSRF>]
+            Enable csrf protection middleware
+
+        --docs
+            Open the Boltzmann documentation in a web browser
+
+        --esbuild [<ESBUILD>]
+            Enable asset bundling via ESBuild
+
+        --force
+            Update a git-repo destination even if there are changes
+
+        --githubci [<GITHUBCI>]
+            Enable GitHub actions CI
+
+    -h, --help
+            Print help information
+
+        --honeycomb [<HONEYCOMB>]
+            Enable tracing via Honeycomb
+
+        --jwt [<JWT>]
+            Enable jwt middleware
+
+        --livereload [<LIVERELOAD>]
+            Enable live reload in development
+
+        --oauth [<OAUTH>]
+            Enable OAuth
+
+        --ping [<PING>]
+            Enable /monitor/ping liveness endpoint; on by default
+
+        --postgres [<POSTGRES>]
+            Enable postgres middleware
+
+    -q, --quiet
+            Suppress all output except errors; an alias for silent
+
+        --redis [<REDIS>]
+            Enable redis middleware
+
+    -s, --silent
+            Suppress all output except errors
+
+        --selftest
+            Template a project with the self-test code enabled
+
+        --staticfiles [<STATICFILES>]
+            Enable static file serving in development
+
+        --status [<STATUS>]
+            Enable /monitor/status healthcheck endpoint; on by default
+
+        --templates [<TEMPLATES>]
+            Enable Nunjucks templates
+
+        --typescript [<TYPESCRIPT>]
+            Scaffold a project implemented in TypeScript
+
+    -v, --verbose
+            Pass -v or -vv to increase verbosity
+
+    -V, --version
+            Print version information
+
+        --volta [<VOLTA>]
+            Enable node version management via Volta
 
+        --website
+            Enable all features relevant to building websites
+
+            This option group enables the templates, csrf, staticfile, jwt, livereload, ping, and
+            status options.
 ```
 
 [`test`]: #TKTKTK

diff --git a/docs/content/reference/02-handlers.md b/docs/content/reference/02-handlers.md
@@ -394,12 +394,15 @@ async function host(context) {
 ### `id`
 
 {{ changelog(version = "0.0.0") }}
+- **Changed in 0.6.0:** Use `traceparent` as the `id` when available.
+{% end %}
 
 A unique string identifier for the request for tracing purposes. The value is
 drawn from:
 
-1. `x-honeycomb-trace`
+1. `x-honeycomb-trace` (via Honeycomb beeline tracing)
 2. `x-request-id`
+3. `traceparent` (via Honeycomb OpenTelemetry tracing)
 3. A generated [ship name from Iain M Bank's Culture series][culture] (e.g.: `"ROU Frank Exchange Of Views"`)
 
 **Example use:**

diff --git a/docs/content/reference/03-middleware.md b/docs/content/reference/03-middleware.md
@@ -744,9 +744,9 @@ Here is an example of the request logging:
 ```
 
 The `id` fields in logs is the value of the request-id, available on the context object as the `id`
-field. This is set by examining headers for an existing id. Boltzmann consults `x-honeycomb-trace`
-and `x-request-id` before falling back to generating a request id using a short randomly-selected
-string.
+field. This is set by examining headers for an existing id. Boltzmann consults `x-honeycomb-trace`,
+`x-request-id` and `traceparent` before falling back to generating a request id using a short
+randomly-selected string.
 
 To log from your handlers, you might write code like this:
 
@@ -773,19 +773,45 @@ Boltzmann automatically attaches one instance of [`route`](#route).
 
 ### `trace`
 
+{% changelog(version="0.0.0") %}
+- **Changed in 0.6.0:** Tracing now uses OpenTelemetry if any `OTEL_*` environment
+variable is defined
+{% end %}
+
 This middleware is added to your service if you have enabled the `honeycomb` feature.
 This feature sends trace data to the [Honeycomb](https://www.honeycomb.io) service for
 deep observability of the performance of your handlers.
 
-To configure this middleware, set the following environment variables:
-
--   `HONEYCOMB_WRITEKEY`: the honeycomb API key to use; required to enable tracing
--   `HONEYCOMB_DATASET`: the name of the dataset to send trace data to; required to enable tracing
--   `HONEYCOMB_TEAM`: optional; set this to enable links to traces from error reporting
--   `HONEYCOMB_SAMPLE_RATE`: optional; passed to `honeycomb-beeline` to set the sampling rate for events
-
-The sampling rate defaults to 1 if neither sample rate env var is set. Tracing is
-disabled if a write key and dataset are not provided; the middleware is still
-attached but does nothing in this case.
+To configure this middleware for [beeline](https://www.npmjs.com/package/honeycomb-beeline) 
+tracing, set the following environment variables:
+
+-   `HONEYCOMB_API_HOST`: (optional) the honeycomb API endpoint to use - defaults to localhost
+-   `HONEYCOMB_WRITEKEY`: the honeycomb API key to use - required to enable beeline tracing
+-   `HONEYCOMB_DATASET`: the name of the dataset to send trace data to - defaults to `nodejs`
+-   `HONEYCOMB_TEAM`: (optional) set this to enable links to traces from development error reporting
+-   `HONEYCOMB_SAMPLE_RATE`: (optional) passed to `honeycomb-beeline` to set the sampling rate for events - defaults to 1
+
+The honeycomb middleware also supports
+[OpenTelemetry](https://opentelemetry.io/docs/instrumentation/js/getting-started/nodejs/)
+tracing over [the OTLP http/protobuf protocol](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/protocol/otlp.md).
+This is enabled if any `OTEL_*` environment variables are defined.
+
+OpenTelemetry supports many, many environment variables - they document both
+[common SDK environment variables](https://opentelemetry.io/docs/reference/specification/sdk-environment-variables/)
+and [OTLP exporter-specific environment variables](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/protocol/exporter.md),
+and it's worth perusing all of them. However, if you are in a rush:
+
+- `HONEYCOMB_WRITEKEY`: this is still used in an OpenTelemetry configuration
+- `HONEYCOMB_DATASET`: this is also still used in an OpenTelemetry configuration
+- `OTEL_EXPORTER_OTLP_ENDPOINT`: the API endpoint - i.e. `https://api.honeycomb.io`, or your [refinery](https://docs.honeycomb.io/manage-data-volume/refinery/) instance if using one
+
+In you really, really want to use the default OpenTelemetry configuration,
+you can set `OTEL_ENABLE=1` - this isn't meaningful for the OpenTelemetry SDKs
+or OTLP exporters but will trigger the enabling of OpenTelemetry in boltzmann.
+
+Note that OpenTelemetry tracing, while intended for use with Honeycomb, may be
+used with any OTLP tracing backend. You can do this by foregoing the setting of
+`HONEYCOMB_WRITEKEY` and setting `OTEL_EXPORTER_OTLP_HEADERS` to contain the
+alternate headers for your backend.
 
 * * *
-Original file line number
+Diff line change
@@ Expand Up / @@ -33,7 +33,7 @@ For example, to scaffold with the defaults: @@
     projects|⇒ npx boltzmann-cli hello
     ```
-    A complete project is provided for you, with useful package run scripts and linting. To run: `./boltzmann.js`. And to view the response: `curl http://localhost:5000/hello/world`. Want to know more? [Check the docs!](https://www.boltzmann.dev/en/docs/latest/)
+    A complete project is provided for you, with useful package run scripts and linting. To run: `./boltzmann.js`. And to view the response: `curl http://localhost:8000/hello/world`. Want to know more? [Check the docs!](https://www.boltzmann.dev/en/docs/latest/)
     ## Team
@@ Expand Down @@