Display additional information or test origin for failed tests

[pahaeanx]

I'd be great if there were a way to either add information to tests (such as a name hint) or showing their origin file in case they are pulled in via gossfile.

For example, I split up most tests into what I call group files and test files. Group files simply pull in all tests for a particular group (ansible in this case). Like so:

# file: group_prometheus.yaml
gossfile:
    verify_node_exporter.yaml: {}
    verify_iptables.yaml: {}
    verify_acme_certs.yaml: {}
    verify_nginx.yaml: {}
    verify_prometheus.yaml: {}
    ...

Where each of these files contain tests for the component they are named after ( verify_node_exporter.yaml for the Prometheus node-exporter, etc). This allows test files to be nice and contained but still allows us to quickly and easily add a new groups or tests.

These test files then get templated to a host and validation is done using

# file: goss.yaml
 gossfile:
    /root/goss-tests/tests/*.yaml: {}

Anyway, the problem arises when one or more tests fail. A user has no easy way of telling where the failing test originates from.

Ultimately this is a problem of information: I'd like some way to know more about the failed test. Not everyone knows the system by heart.
Say the port/listening test for port 9100 failed -- there's no immediate way of knowing what is supposed to listen on there.

It is fair to question whether this should be part of a test suite, but I think that even a little extra information would be enough to allow immediate targeted troubleshooting.

Describe the feature:

I'd be great if we can see the file a failed test originates from. Something like this (extends the documentation format).

Failures/Skipped:

Port: tcp:9100: listening:
Expected
    <bool>: false
to equal
    <bool>: true
Origin: verify_node_exporter.yaml

or even some kind of hint we can set, possibly on a per-test basis:

Failures/Skipped:

Port: tcp:9100: listening:
Expected
    <bool>: false
to equal
    <bool>: true
Hint: node-exporter listening port

Both of these solutions immediately give enough additional information to for targeted troubleshooting.

Describe the solution you'd like

This boils down to what is feasible. I guess the most simple solution would just to display the test origin in the final result. This would already be a huge improvement and sufficient for most situations.

Ideally there would be some kind of optional hint resource which can be used to add additional information to all tests contained in a file. This could also be set on a per-test basis:

# The default value
hint: node-exporter tests
command:
  exec: /bin/true
  exit-status: 0
port:
  tcp:9100:
    listening: true
    ip:
      - 111.111.111.111
    # override the default
    hint: node-exporter listening port

If these fail, one would display Hint: node-exporter tests (the default set via hint resource), the other would display Hint: node-exporter listening port.

Describe alternatives you've considered
I don't think there is any other solution or workaround for this problem at the moment.

[sshipway]

This is similar to (but better than!) an idea I had in the past, which was to add a 'description' attribute to tests that could be printed on test failure to give a bit more information to the user on which test failed. Being able to define it at test or file level, defaulting to the file name, would be handy when your tests are split over multiple gossfiles, possibly managed by multiple other sources.

[aelsabbahy]

correct me if I'm wrong but do title and meta cover the "hint"/"description" use case?

example file defining title and meta:

$ cat goss.yaml 
command:
  some_test:
    title: This is some arbitrary title
    meta:
      any-key: any-value
      version: 17
      severity: very severe
      contact: call bob
      hint: node-exporter listening port
    exec: |
      echo "hi"
    stdout:
    - bye

example output:

$ goss v
FF

Failures/Skipped:

Title: This is some arbitrary title
Meta:
    any-key: any-value
    contact: call bob
    hint: node-exporter listening port
    severity: very severe
    version: 17
some_test: exit-status: Error: Missing Required Attribute
Command: some_test: stdout: patterns not found: [bye]

Total Duration: 0.004s
Count: 2, Failed: 2, Skipped: 0

Just realized they're documented in the readme, but not the manual https://github.com/goss-org/goss#manually-editing-goss-files

Basically, title can be set to any string and meta can be set to any key/value (both are stored strings).

[pahaeanx]

Oh wow, I somehow completely missed that.

After playing around with these a bit, the only criticism I have is that it seems that neither can be set at the file level?

# goss.yaml
title: testing file level titles
command:
 fail_w/o_title:
   exec: /bin/false
   exit-status: 0
 fail_w_title:
   title: This test has a title set
   exec: /bin/false
   exit-status: 0

Output:

$ goss v
FF

Failures/Skipped:

Command: fail_w/o_title: exit-status:
Expected
    <int>: 1
to equal
    <int>: 0

Title: This test has a title set
Command: fail_w_title: exit-status:
Expected
    <int>: 1
to equal
    <int>: 0

Total Duration: 0.002s
Count: 2, Failed: 2, Skipped: 0

I guess it'd be nice to be able to set title and meta attributes on a file level that act as default values for tests that don't have these set.

The current limitation can be worked around by setting the same title for each test (I'd simply set the file name) but that seems like a lot of unnecessary work and code.

Apart from that, that's exactly what I was looking for. Thanks!

[aelsabbahy]

@sshipway just want to confirm, does the current functionality of title/meta meet your suggestion above?

@pahaeanx I like your suggestion, but it might be a little (long) while before I get to it to be honest.

Mostly because, I'm thinking as a future enhancement Goss should have the ability to group tests into a test suite.. that suite can then have title/meta.. but it can also have other things like setup/teardown.

Perhaps the file will be the group.. or maybe there would be some other top level suite concept introduced that allows multiple suites to be set for a given file.

Once I get through the final Goss v4 release, I'll open up some issues mapping out the future roadmap. Hopefully, I can stimulate some OSS contributions to the projects for some of them.