juniper_junos_command missing data, json duplicate keys not converted to list

[cwiech]

Issue Type

• Bug Report

Module Name

juniper_junos_command

juniper.device collection and Python libraries version

ansible --version
ansible 2.10.8
  config file = /Users/xxx/git/ansible-scripts/ansible.cfg
  configured module search path = ['/Users/xxx/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /Users/xxx/Library/Python/3.8/lib/python/site-packages/ansible
  executable location = /Users/xxx/Library/Python/3.8/bin/ansible
  python version = 3.8.2 (default, Jun  8 2021, 11:59:35) [Clang 12.0.5 (clang-1205.0.22.11)]


pip freeze
ansible==2.10.7
ansible-base==2.10.8
ansible-lint==4.3.7
bcrypt==3.2.0
Cerberus==1.3.2
certifi==2020.6.20
cffi==1.14.3
chardet==3.0.4
click==7.1.2
colorama==0.4.4
commonmark==0.9.1
cryptography==3.1.1
Flask==1.1.2
flask-swagger-ui==3.36.0
future==0.18.2
gunicorn==20.1.0
idna==2.10
ipaddr==2.2.0
itsdangerous==1.1.0
Jinja2==2.11.2
junos-eznc==2.6.2
jxmlease==1.0.3
lxml==4.5.2
MarkupSafe==1.1.1
mysql-connector-python==8.0.22
ncclient==0.6.9
netaddr==0.8.0
ntc-templates==1.5.0
numpy==1.20.3
packaging==20.9
pandas==1.2.4
paramiko==2.7.2
pathspec==0.8.1
prometheus-client==0.9.0
protobuf==3.14.0
pyang==2.3.2
pycparser==2.20
Pygments==2.7.3
PyNaCl==1.4.0
pyparsing==2.4.7
pyserial==3.4
python-dateutil==2.8.1
pytricia==1.0.2
pytz==2021.1
PyYAML==5.3.1
requests==2.24.0
rich==9.6.1
ruamel.yaml==0.16.12
ruamel.yaml.clib==0.2.2
scp==0.13.2
six==1.15.0
termcolor==1.1.0
textfsm==1.1.0
transitions==0.8.3
typing-extensions==3.7.4.3
urllib3==1.25.10
Werkzeug==1.0.1
xmltodict==0.12.0
yamllint==1.25.0
yamlordereddictloader==0.4.0

OS / Environment

Model: ptx10008
Junos: 17.3R3-S12.3

Summary

When running the command "show bgp neighbor" and asking for json output the output returns duplicate keys (addpath-send and addpath-recieve are used multiple times). The json retrieved by ansible omits the duplicate keys and we loose data... the json returned is incomplete. Python does not permitted duplicate keys in a dictionary, so only the last entry will survive.

If I take a "show bgp neighbor | display json" from the CLI directly, its possible to write my own json.loads function to turn the duplicate keys into an array (as discussed in this link and other places), but the ansible parsing module would have to updated in a similar way to avoid loosing output.

https://stackoverflow.com/questions/24416960/convert-json-object-with-duplicate-keys-to-json-array

Steps to reproduce

You can see the issue on the router CLI... the json keys are duplicated. I guess this is technically not an RFC violation, but it would seem that most conversion utilities (python) will not allow this.

 show bgp neighbor | display json
 
 <snip>
               "addpath-send" : [
                {
                    "nlri-type" : [
                    {
                        "data" : "inet-unicast"
                    }
                    ],
                    "addpath-send-type" : [
                    {
                        "data" : "Specified number of paths"
                    }
                    ],
                    "addpath-send-count" : [
                    {
                        "data" : "2"
                    }
                    ]
                }
                ],
                "addpath-receive" : [
                {
                    "nlri-type" : [
                    {
                        "data" : "inet-unicast"
                    }
                    ],
                    "addpath-receive-type" : [
                    {
                        "data" : "enabled"
                    }
                    ]
                }
                ],
                "addpath-send" : [
                {
                    "nlri-type" : [
                    {
                        "data" : "inet6-labeled-unicast"
                    }
<snip>

When the playbook is run... the resulting output will only show the last entry in this case inet6-labeled-unicast, the inet-unicast data is lost.

TASK [debug return data] *******************************************************
ok: [edge10.nso5] => {
    "msg": {
        "changed": false,
        "command": "show bgp neighbor",
        "failed": false,
        "format": "json",
        "msg": "The command executed successfully.",
        "parsed_output": {
            "bgp-information": [
                {

<snip>

                            "bgp-option-information": [
                                {
                                    "addpath-receive": [
                                        {
                                            "addpath-receive-type": [
                                                {
                                                    "data": "enabled"
                                                }
                                            ],
                                            "nlri-type": [
                                                {
                                                    "data": "inet6-labeled-unicast"
                                                }
                                            ]
                                        }
                                    ],
                                    "addpath-send": [
                                        {
                                            "addpath-send-count": [
                                                {
                                                    "data": "2"
                                                }
                                            ],
                                            "addpath-send-type": [
                                                {
                                                    "data": "Specified number of paths"
                                                }
                                            ],
                                            "nlri-type": [
                                                {
                                                    "data": "inet6-labeled-unicast"
                                                }
                                            ]
                                        }
                                    ],

- hosts: "{{device}}"
  gather_facts: no
  connection: local
  roles:
    - juniper.junos

  - name: get junos bgp neighbor data
    juniper_junos_command:
        commands:
           - show bgp neighbor
        display: json
    register: show_bgp_neighbor
    when: os == "junos"

  - name: debug return data
    debug:
       msg: "{{ show_bgp_neighbor }}"

Expected results

The duplicate keys from the juniper must converted to ensure data is not lost. The juniper module needs to take the duplicate keys from the raw router json and convert them into a list (or the duplicates would have to be eliminated in junos). I assume the juniper ansible module is using json.loads or something similar, by default, json.loads will overwrite the duplicate keys so only the last entry will survive.

The ansible moudles must return something like this in the ansible register to avoid loosing parts of the output and ensure the playbook has the full contents of the command from the router.

      "addpath-send": [
         {
            "nlri-type": "inet-unicast",
            "addpath-send-type": "Specified number of paths",
            "addpath-send-count": "2"
         },
         {
            "nlri-type": "inet6-labeled-unicast",
            "addpath-send-type": "Specified number of paths",
            "addpath-send-count": "2"
         }
      ],
      "addpath-receive": [
         {
            "nlri-type": "inet-unicast",
            "addpath-receive-type": "enabled"
         },
         {
            "nlri-type": "inet6-labeled-unicast",
            "addpath-receive-type": "enabled"
         }
      ],

Actual results

juniper_junos_command is returning incomplete data from the router when the display type is json.

                        "bgp-option-information": [
                            {
                                "addpath-receive": [
                                    {
                                        "addpath-receive-type": [
                                            {
                                                "data": "enabled"
                                            }
                                        ],
                                        "nlri-type": [
                                            {
                                                "data": "inet6-labeled-unicast"
                                            }
                                        ]
                                    }
                                ],
                                "addpath-send": [
                                    {
                                        "addpath-send-count": [
                                            {
                                                "data": "2"
                                            }
                                        ],
                                        "addpath-send-type": [
                                            {
                                                "data": "Specified number of paths"
                                            }
                                        ],
                                        "nlri-type": [
                                            {
                                                "data": "inet6-labeled-unicast"
                                            }
                                        ]
                                    }
                                ],