logging does not work when running jsnapy with ansible

[avocadonetwork1]

Description of Issue/Question

Hi,
I have integrated jsnapy with ansible so that I can use ansible playbook to run jsnapy.
However the info and err from my jsnapy tests are not outputing when I run the ansible playbook.
As the documentation suggests, I have tried both of the configs below on ansible.cfg but still no output
callback_enabled = juniper.device.jsnapy
callback_whitelist = juniper.device.jsnapy

Setup

The following files are created

1. snapcheck_playbook.yml (ansible playbook) ,
2. jsnapy_tests.yml (which tests I am running) and
3. test_snmp.yml (test itself)

I am adding the files below

`
snapcheck_playbook.yml

• name: "Ansible Jsnapy Snapcheck"
  hosts: all
  strategy: free
  connection: local
  gather_facts: false

  tasks:

  • name: Run Jsnapy check
    juniper.device.jsnapy:
    action: snapcheck
    config_file: jsnapy_tests.yml
    logfile: jsnapy_ansible.log
    register: result

  • name: Run Jsnapy check
    assert:
    that:

    • "result.passPercentage == 100"

jsnapy_tests.yml

tests:

• test_snmp.yml

test_snmp.yml

Test SNMP config:

• rpc: get-configuration
• item:
  xpath: /configuration/snmp
  tests:
  - exists: community[name='mycommunity']
  info: SNMP community ''mycommunity' is configured.
  err: SNMP community ''mycommunity' is not configured!
  `

Steps to Reproduce Issue

(Include debug logs if possible and relevant. Error trace would be helpful too)
To run the playbook:
$ ansible-playbook snapcheck_playbook.yml -l mydevice

PLAY [Ansible Jsnapy Snapcheck] *******************************************************************************************************

TASK [Run Jsnapy check] ***************************************************************************************************************
ok: [mydevice]

TASK [Run Jsnapy check] ***************************************************************************************************************
fatal: [mydevice]: FAILED! => {
"assertion": "result.passPercentage == 100",
"changed": false,
"evaluated_to": false,
"msg": "Assertion failed"
}

PLAY RECAP ****************************************************************************************************************************
mydevice : ok=1 changed=0 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0

As you can see from the above, the tests failed as expected (because the snmp community is not set at my router yet. However the err message is not displayed on the output.
I can see the errors on the jsnapy.log but not on the output from the command

Versions Report

(Provided by running jsnapy --version. Please also mention python version.)
$ jsnapy --version
JSNAPy version: 1.3.6

$ ansible --version
ansible [core 2.11.1]

$ python --version
Python 3.9.5

[chidanandpujar]

issue is reproducible with the following information

cat snapcheck_playbook.yml 
---
- name: Test juniper.device.jsnapy module
  hosts: all
  gather_facts: false
  tasks:
##################################################
####    TEST 2                                  ##
##################################################
    - name: "TEST 2 - Execute SNAPCHECK "
      juniper.device.jsnapy:
        config_file: jsnapy_tests.yml
        action: snapcheck
      register: result 

    - name: Verify JSNAPy tests passed
      assert:
        that:
          - "result.passPercentage == 100"

cat jsnapy_tests.yml 
tests:
  test_snmp.yml

 cat test_snmp.yml 
tests_include:
  - test_snmp_config 

test_snmp_config:
  - rpc: get-configuration
  - iterate:
      xpath: /configuration/snmp
      tests:
        - exists: community[name='mycommunity'] 
          info: SNMP community ''mycommunity' is configured. 
          err: SNMP community ''mycommunity' is not configured!

jsnapy --snapcheck -f jsnapy_tests_with_host.yml
Connecting to device x.x.x.x ................
Taking snapshot of RPC: get-configuration
**************************** Device: x.x.x.x ****************************
Tests Included: test_snmp_config 
*************************RPC is get-configuration*************************
ERROR!! Nodes are not present in given <Xpath> : </configuration/snmp>
FAIL | All "community[name='mycommunity']" do not exists at xpath "/configuration/snmp" [ 0 value matched / 1 value failed ]
------------------------------- Final Result!! -------------------------------
test_snmp_config : Failed
Total No of tests passed: 0
Total No of tests failed: 1 
Overall Tests failed!!! 




ansible-playbook snapcheck_playbook.yml 

PLAY [Test juniper.device.jsnapy module] **********************************************************************************************************************************

TASK [TEST 2 - Execute SNAPCHECK] *****************************************************************************************************************************************
ok: [local_connection_testcases_mode_ssh]

TASK [Verify JSNAPy tests passed] *****************************************************************************************************************************************
fatal: [local_connection_testcases_mode_ssh]: FAILED! => changed=false 
  assertion: result.passPercentage == 100
  evaluated_to: false
  msg: Assertion failed

PLAY RECAP ****************************************************************************************************************************************************************
local_connection_testcases_mode_ssh : ok=1    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0   

Thanks
Chidanand

[chidanandpujar]

Pass logs:

##################################################
####    TEST JSNAPY PRE                         ##
##################################################

    - name: "Check after PRE - POST check"
      juniper.device.jsnapy:
        test_files: test_snmp.yml
        action: "snapcheck"
        dest_dir: "{{ backup_dir }}"  
      register: test_check
      tags: 
        - jsnapy

    - debug: var=test_check



TASK [Check after PRE - POST check] ***************************************************************************************************************************************
ok: [x.x.x.x]

TASK [debug] **************************************************************************************************************************************************************
ok: [x.x.x.x] => {
    "test_check": {
        "changed": false,
        "failed": false,
        "get-configuration": [
            {
                "count": {
                    "fail": 1,
                    "pass": 0
                },
                "err": "SNMP community ''mycommunity' is not configured!",
                "failed": [
                    {
                        "actual_node_value": null,
                        "id": {},
                        "post": {},
                        "pre": {},
                        "xpath_error": true
                    }
                ],
                "node_name": "community[name='mycommunity']",
                "passed": [],
                "result": false,
                "test_name": "test_snmp_config",
                "testoperation": "exists",
                "xpath": "/configuration/snmp"
            }
        ]
    }
}

PLAY RECAP ****************************************************************************************************************************************************************
x.x.x.x              : ok=8    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   


cat backup_dir/x.x.x.x_test_snmp_config_False.text 
[{'xpath': '/configuration/snmp', 'testoperation': 'exists', 'passed': [], 'failed': [{'id': {}, 'pre': {}, 'post': {}, 'actual_node_value': None, 'xpath_error': True}], 'test_name': 'test_snmp_config', 'node_name': "community[name='mycommunity']", 'err': "SNMP community ''mycommunity' is not configured!", 'result': False, 'count': {'pass': 0, 'fail': 1}}]

Thanks
Chidanand

[chidanandpujar]

operator exists now reports info/err messages as part of fix #415 and also failed tests can be copied to dest_dir and it is supported via Juniper/ansible-junos-stdlib#678
callback issue will be handled in the ansible-junos-stdlib via issue Juniper/ansible-junos-stdlib#599