How can I convert a log line into json format?

[cgHome]

I came across the vector app by chance and am actually delighted with it. I run my docker container logs via the app and am completely satisfied with a few exceptions. However, there are docker container apps (e.g. Portainer) that do not send the logs in json format. Is there a way to convert them?

Code:

source: |
      structured, err = parse_json(.message)
      if err != null { 
        log(err, level: "error")
        log(string!(.message), level: "error")
      }
      . = merge!(., structured)
      
      .metadata.job = .source_type
      .metadata.stream = .stream
      .metadata.level = .level || "other"      
      .metadata.compose_project = .label."com.docker.compose.project"
      .metadata.container_name = .container_name

portainer-log

2024/03/13 08:28AM INF deployments/deployer_remote.go:207 > Stack deployment output | output="{\"level\":\"info\",\"repository\":\"https://github.com/cgHome/homelab.stacks\",\"composePath\":[\"monitoring/compose.yml\"],\"destination\":\"/share/Container/.portainer/portainer-compose-unpacker\",\"env\":[],\"skipTLSVerify\":false,\"time\":1710318512,\"caller\":\"/home/runner/work/compose-unpacker/compose-unpacker/deploy.go:31\",\"message\":\"Deploying Compose stack from Git repository\"}\n{\"level\":\"info\",\"user\":\"cgHome\",\"time\":1710318512,\"caller\":\"/home/runner/work/compose-unpacker/compose-unpacker/deploy.go:42\",\"message\":\"Using Git authentication\"}\n{\"level\":\"info\",\"directory\":\"/share/Container/.portainer/portainer-compose-unpacker\",\"time\":1710318512,\"caller\":\"/home/runner/work/compose-unpacker/compose-unpacker/deploy.go:57\",\"message\":\"Checking the file system...\"}\n{\"level\":\"info\",\"directory\":\"/share/Container/.portainer/portainer-compose-unpacker/stacks/homelab-monitoring\",\"time\":1710318512,\"caller\":\"/home/runner/work/compose-unpacker/compose-unpacker/deploy.go:83\",\"message\":\"Creating target destination directory on disk\"}\n{\"level\":\"info\",\"repository\":\"https://github.com/cgHome/homelab.stacks\",\"path\":\"/share/Container/.portainer/portainer-compose-unpacker/stacks/homelab-monitoring/homelab.stacks\",\"url\":\"https://github.com/cgHome/homelab.stacks\",\"depth\":1,\"time\":1710318512,\"caller\":\"/home/runner/work/compose-unpacker/compose-unpacker/deploy.go:98\",\"message\":\"Cloning git repository\"}\n{\"level\":\"info\",\"composeFilePaths\":[\"/share/Container/.portainer/portainer-compose-unpacker/stacks/homelab-monitoring/homelab.stacks/monitoring/compose.yml\"],\"workingDirectory\":\"/share/Container/.portainer/portainer-compose-unpacker/stacks/homelab-monitoring/homelab.stacks\",\"projectName\":\"homelab-monitoring\",\"time\":1710318512,\"caller\":\"/home/runner/work/compose-unpacker/compose-unpacker/deploy.go:126\",\"message\":\"Deploying Compose stack\"}\n{\"level\":\"info\",\"time\":1710318516,\"caller\":\"/home/runner/work/compose-unpacker/compose-unpacker/deploy.go:144\",\"message\":\"Compose stack deployment complete\"}\n" 

result

2024-03-13T08:28:37.328415Z ERROR transform{component_kind="transform" component_id=remap_docker component_type=remap}: vrl::stdlib::log: function call error for "parse_json" at (88:108): unable to parse json: trailing characters at line 1 column 5 internal_log_rate_secs=1 vrl_position=129
2024-03-13T08:28:37.328660Z ER 2024/03/13 08:28AM INF deployments/deployer_remote.go:207 > Stack deployment output | output="{\"level\":\"info\",\"repository\":\"https://github.com/cgHome/homelab.stacks\",\"composePath\":[\"monitoring/compose.yml\"],\"destination\":\"/share/Container/.portainer/portainer-compose-unpacker\",\"env\":[],\"skipTLSVerify\":false,\"time\":1710318512,\"caller\":\"/home/runner/work/compose-unpacker/compose-unpacker/deploy.go:31\",\"message\":\"Deploying Compose stack from Git repository\"}\n{\"level\":\"info\",\"user\":\"cgHome\",\"time\":1710318512,\"caller\":\"/home/runner/work/compose-unpacker/compose-unpacker/deploy.go:42\",\"message\":\"Using Git authentication\"}\n{\"level\":\"info\",\"directory\":\"/share/Container/.portainer/portainer-compose-unpacker\",\"time\":1710318512,\"caller\":\"/home/runner/work/compose-unpacker/compose-unpacker/deploy.go:57\",\"message\":\"Checking the file system...\"}\n{\"level\":\"info\",\"directory\":\"/share/Container/.portainer/portainer-compose-unpacker/stacks/homelab-monitoring\",\"time\":1710318512,\"caller\":\"/home/runner/work/compose-unpacker/compose-unpacker/deploy.go:83\",\"message\":\"Creating target destination directory on disk\"}\n{\"level\":\"info\",\"repository\":\"https://github.com/cgHome/homelab.stacks\",\"path\":\"/share/Container/.portainer/portainer-compose-unpacker/stacks/homelab-monitoring/homelab.stacks\",\"url\":\"https://github.com/cgHome/homelab.stacks\",\"depth\":1,\"time\":1710318512,\"caller\":\"/home/runner/work/compose-unpacker/compose-unpacker/deploy.go:98\",\"message\":\"Cloning git repository\"}\n{\"level\":\"info\",\"composeFilePaths\":[\"/share/Container/.portainer/portainer-compose-unpacker/stacks/homelab-monitoring/homelab.stacks/monitoring/compose.yml\"],\"workingDirectory\":\"/share/Container/.portainer/portainer-compose-unpacker/stacks/homelab-monitoring/homelab.stacks\",\"projectName\":\"homelab-monitoring\",\"time\":1710318512,\"caller\":\"/home/runner/work/compose-unpacker/compose-unpacker/deploy.go:126\",\"message\":\"Deploying Compose stack\"}\n{\"level\":\"info\",\"time\":1710318516,\"caller\":\"/home/runner/work/compose-unpacker/compose-unpacker/deploy.go:144\",\"message\":\"Compose stack deployment complete\"}\n" internal_log_rate_secs=1 vrl_position=156 

THX for help
Chris

[jszwedko]

As you probable noticed, the line you have there is not JSON:

2024/03/13 08:28AM INF deployments/deployer_remote.go:207 > Stack deployment output | output="{\"level\":\"info\",\"repository\":\"https://github.com/cgHome/homelab.stacks\",\"composePath\":[\"monitoring/compose.yml\"],\"destination\":\"/share/Container/.portainer/portainer-compose-unpacker\",\"env\":[],\"skipTLSVerify\":false,\"time\":1710318512,\"caller\":\"/home/runner/work/compose-unpacker/compose-unpacker/deploy.go:31\",\"message\":\"Deploying Compose stack from Git repository\"}\n{\"level\":\"info\",\"user\":\"cgHome\",\"time\":1710318512,\"caller\":\"/home/runner/work/compose-unpacker/compose-unpacker/deploy.go:42\",\"message\":\"Using Git authentication\"}\n{\"level\":\"info\",\"directory\":\"/share/Container/.portainer/portainer-compose-unpacker\",\"time\":1710318512,\"caller\":\"/home/runner/work/compose-unpacker/compose-unpacker/deploy.go:57\",\"message\":\"Checking the file system...\"}\n{\"level\":\"info\",\"directory\":\"/share/Container/.portainer/portainer-compose-unpacker/stacks/homelab-monitoring\",\"time\":1710318512,\"caller\":\"/home/runner/work/compose-unpacker/compose-unpacker/deploy.go:83\",\"message\":\"Creating target destination directory on disk\"}\n{\"level\":\"info\",\"repository\":\"https://github.com/cgHome/homelab.stacks\",\"path\":\"/share/Container/.portainer/portainer-compose-unpacker/stacks/homelab-monitoring/homelab.stacks\",\"url\":\"https://github.com/cgHome/homelab.stacks\",\"depth\":1,\"time\":1710318512,\"caller\":\"/home/runner/work/compose-unpacker/compose-unpacker/deploy.go:98\",\"message\":\"Cloning git repository\"}\n{\"level\":\"info\",\"composeFilePaths\":[\"/share/Container/.portainer/portainer-compose-unpacker/stacks/homelab-monitoring/homelab.stacks/monitoring/compose.yml\"],\"workingDirectory\":\"/share/Container/.portainer/portainer-compose-unpacker/stacks/homelab-monitoring/homelab.stacks\",\"projectName\":\"homelab-monitoring\",\"time\":1710318512,\"caller\":\"/home/runner/work/compose-unpacker/compose-unpacker/deploy.go:126\",\"message\":\"Deploying Compose stack\"}\n{\"level\":\"info\",\"time\":1710318516,\"caller\":\"/home/runner/work/compose-unpacker/compose-unpacker/deploy.go:144\",\"message\":\"Compose stack deployment complete\"}\n" 

I see there is some escaped JSON following the output= so you could try to use parse_regex to extract just that part, however, you'd then need to unescape the string before passing it to parse_json and there isn't yet a function to do that in VRL at the moment (I'd welcome a feature request for it here: . Is there any chance you can modify the process emitting logs in that format? Parsing it will be kinda tricky.

[VVvKamper]

If I understand the problem correctly, one could use parse_json!(s'"' + escaped_string + s'"') as a workaround to unescape a JSON string.