Export logs as json to Google Cloud Logging

[ashish-b-choudhary-db]

Component(s)

exporter/googlecloud

Is your feature request related to a problem? Please describe.

Currently the Google Cloud exporter sends the logs in an unstructured format under textPayload.

Describe the solution you'd like

Provide option to send the logs in a structured manner as json with the right mappings.

Describe alternatives you've considered

Going to try and look into transforming it ourselves

Additional context

No response

[github-actions]

Pinging code owners:

• exporter/googlecloud: @aabmass @dashpole @jsuereth @punya @psx95

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[dashpole]

The logging exporter will attempt to export json-structured logs if the value type is Map or Bytes, and is able to be converted to json: https://github.com/GoogleCloudPlatform/opentelemetry-operations-go/blob/8da0f42dab085c916987891419461d583a2aa96e/exporter/collector/logs.go#L503

Let me know if you need help formatting logs in that way.

[ashish-b-choudhary-db]

Yes please, would you have any suggestions on how to achieve the same.
Currently we're trying to use the https://opentelemetry.io/docs/zero-code/java/spring-boot-starter/ to send logs to a collector and then to Cloud Logging.
The native logs are structured while these logs are currently unstructured, we're trying to make the format same to have hybrid logs.

But since the exporter does support the same, that gives me more to work with. Spring Boot supports structured logging out of the box now. More to explore.

[dashpole]

You will want to use the transform processor. You should be able to do something like this:

transform:
  log_statements:
    - context: log
      statements:
      - set(attributes["body"], body)
      - set(body, attributes)

That moves the existing body into attributes, and then moves all attributes (which is a map) into the body

[dashpole]

Ah, you probably need to delete attributes as well (since they are now moved to the body. I updated the example above to use keep_keys to remove all keys.

[ashish-b-choudhary-db]

@dashpole that did indeed work, however we didn't find the final result satisfactory.

So alternative we're exploring is to use log4j2 to write logs to a file in JSON having tracing information and hope to use that as the jsonPayLoad value.
However trying it out, the log value still went to textPayload.

Here's one line of app.log file content written using log4j2 and GcpLayout json template -
{"timestamp":"2025-01-03T04:22:56.262Z","severity":"INFO","message":"HV000001: Hibernate Validator 8.0.1.Final","logging.googleapis.com/sourceLocation":{"function":"org.hibernate.validator.internal.util.Version.<clinit>"},"logging.googleapis.com/insertId":"0","_exception":{"stackTrace":""},"_thread":"background-preinit","_logger":"org.hibernate.validator.internal.util.Version","logging.googleapis.com/trace_sampled":true}

The value was read using filelog receiver and exported without transformation using googlecloud exporter:

receivers:
  filelog:
    include: [ /tmp/app.log ]

The final result in Cloud Logging looked like:

{
  "textPayload": "{\"timestamp\":\"2025-01-03T05:53:27.342Z\",\"severity\":\"INFO\",\"message\":\"HV000001: Hibernate Validator 8.0.1.Final\",\"logging.googleapis.com/sourceLocation\":{\"function\":\"org.hibernate.validator.internal.util.Version.<clinit>\"},\"logging.googleapis.com/insertId\":\"0\",\"_exception\":{\"stackTrace\":\"\"},\"_thread\":\"background-preinit\",\"_logger\":\"org.hibernate.validator.internal.util.Version\",\"logging.googleapis.com/trace_sampled\":true}",......
}

We also tried with adding json_parser operator in filelog. That essentially moved all the json fields additionally to label.

When usually the log is written to a container stdout and read by native GKE log agent, we get proper jsonPayload enriched with tracing information. We're looking to see if we can achieve this using Otel collector too.
Any suggestions?

Edit: Making some progress. Adding a json parser which parses from: body to: body helped.

[dashpole]

https://github.com/GoogleCloudPlatform/opentelemetry-operations-java/blob/a08a050905f52dd566ccb0a3ddaeba632fd83ffc/examples/instrumentation-quickstart/otel-collector-config.yaml#L30 might be helpful here. We tried to mimic the behavior of the GKE agents in our quickstart samples using the collector.