Skip to content

2. Writing test and config file

Jump to bottom
sidhujasminder edited this page Mar 14, 2018 · 17 revisions

Writing config file

Config file supports following parameters:

  1. hosts: here you have to specify login credential of device(s) to be connected.
hosts:
  - device: 10.20.1.24
    username : foo
    passwd: bar

With timeout as device parameter:

  hosts:
    - device: 10.20.1.24
      username : foo
      passwd: bar
For mutiple hosts either we can:

    a. write all login credentials in one yaml file and import using "include" option

    b. provide list of hosts in the config file itself.  

hosts:
  - include: devices.yml
    group: EX
hosts:
  - device: 10.20.1.24
    username : foo
    passwd: bar
  - device: 172.12.1.38
    username : xxxx
    passwd: bar

Note: If ssh is configured no need to pass user credential in this plain yaml file. JSNAPy will take the credentials from ssh config itself

  1. tests: specify test files that you want to run with tests key and value being list of test files. These test files should be located at /etc/jsnapy/testfiles.

    User can chose different location by setting test_file_path in /etc/jsnapy/jsnapy.cfg

    They can also pass --folder option at runtime to commandline to chose testfiles from given folder.

    Refer: Conditional Test Operators

hosts:
  - device: 10.20.1.24
    username : foo
    passwd: bar
tests:
 - test_no_diff.yml
 - test_bgp_neighbor.yml

  1. sqlite(optional): Use this key if you want to compare/store snapshots in database.

  2. mail(optional): Use this key if you want to get notification of test results via mail.

  3. local(optional): Use this key if you want to run snapcheck on stored snapshots. Will work with --snapcheck command only. Here keys are name for which snaps where taken. For ex in below example STORED is being used. So we might have already done

    jsnapy --snap STORED -f config.yaml

Examples

  • Sample config file for single device
hosts:
  - device: 10.20.1.24
    username : foo
    passwd: bar
tests:
 - test_no_diff.yml
 - test_bgp_neighbor.yml
# (optional) use only when you want to store and compare snapshots from database  
sqlite:
  - store_in_sqlite: True
    check_from_sqlite: True 
    database_name: jbb.db
    compare: 1,0
# (optional) use when you want to send mail about test results 
mail: send_mail.yml
# (optional) use when you want to run snapcheck on stored snapshots. 
# Will work with --snapcheck command only.
# Specify the list of stored snapshot names on which you want to run snapcheck
local:
  - STORED
  • Sample config file for connecting to multiple devices
# for multiple devices with databse
hosts:
  - include: devices.yml
    group: EX
tests:
  - test_is_equal.yml
  - test_is_in.yml
# (optional) use only when you want to store and compare snapshots from database
sqlite:
  - store_in_sqlite: yes
    check_from_sqlite: yes
    database_name: jbb.db
# (optional) use when you want to send mail about test results
mail: send_mail.yml
  • devices.yml
MX:                      
  - 10.20.1.20:
      username: root
      passwd: root123
  - 10.21.13.14:
      username: root
      passwd: root123
  - 10.20.6.26:
      username: jsnapy
      passwd: jsnapy123
EX: 
  - 10.2.15.210:                    
      username: root
      passwd: root123
  - 10.9.16.22:
      username: abc
      passwd: pqr
QFX:                     
  - 10.29.1.24:
      username: abc
      passwd: pqr
  - 10.29.6.1:
      username: abc
      passwd: pqr123

Writing test file

Purpose of writing test file is to specify command/rpc whose snapshot is to be taken and what all nodes user wants to test and how to test them.

  1. tests_include(optional): use this tag if you want to include only some test cases. If you do not include this tag, then jsnapy will run all test cases by default.

  2. command/rpc : can give either command or rpc to perform testing

    • format: can specify output format [text,xml] for comparing text output, only --diff option is supported

  3. kwargs(optional): (used only with rpc)

    • filter_xml: (can provide filtered output for get-config rpc)
    • other arguments

    Example:

    1

    - rpc: get-config
- kwargs:
    filter_xml: configuration/system/login

    2

    - rpc: get-interface-information
  format: text
- kwargs:
    interface-name: em0
    media: True
    detail: True

    Note: To mention that some RPC can take longer time to respond, for that case the testcase might respond with some error. Can try with giving dev_timeout: n where n is seconds, for which you want to wait to get response. By default it is 30 seconds. Please see the example below:

    3

    - rpc: get-interface-information
  format: text
- kwargs:
    interface-name: em0
    media: True
    detail: True
    dev_timeout: 45

  4. item/iterate: (can have multiple iterate/item under one command/rpc) item: if you want to use only first node in xpath iterate: if want to keep iterating for all nodes in xpath

  5. xpath: path from where you want to test output

  6. tests: (this section specify test-cases, can have multiple test cases inside one iterate/item)

    test-operator: <condition>

    info: <mssg>

    err: <mssg>

    Example:

    tests:
  - is-equal: //minimum-time, 60
    info: "Test Succeeded!!, minimum-time now is equal to <{{post['//minimum-time']}}>"
    err: "Test Failed!!!, minimum-time is not equal to 60, it is <{{post['//minimum-time']}}>"

    For set of test operators that can be used, please refer: supported-test-operators

    info/err supports runtime variable usage in the message. For example:

    test_interfaces_terse:
  - rpc: get-interface-information
  - kwargs:
      terse: True
  - iterate:
      xpath: //physical-interface
      tests:
        - is-equal: admin-status, up
          info: "Test Succeeded !! admin-status is equal to {{pre['admin-status']}} with oper-status {{pre['oper-status']}}"
          err: "Test Failed !! admin-status is not equal to up, it is {{pre['admin-status']}} with oper-status {{pre['oper-status']}}"

    Say, for command show interfaces terse (rpc: get-interface-information & terse True) we have below xml snippet.

    xpath defined in testfile is physical-interface, so for info/err we can use any of the child tag of physical-interface to get the value.

    When we do {{pre['oper-status']}} it will include value for /physical-interface/oper-status. Similarly we can use {{post['oper-status']}} when running check command

  <physical-interface>
  <name>
  xe-1/1/0
  </name>
  <admin-status format="Enabled">
  up
  </admin-status>
  <oper-status>
  down
  </oper-status>
  <local-index>
  156
  </local-index>
  <snmp-index>
  532
  </snmp-index>
  <link-level-type>
  Ethernet
  </link-level-type>
  ---
  </physical-interface>

  1. id (optional): refer example 1 where id is used. id is used in case of check functionality. We need to compare pre and post snapshot based on certain key(s) when there are n number of items for given xpath. pre & post snapshots can have elements in different orders/numbers. id helps in comparing values from xml elements of matched keys using id.

  2. ignore-null(optional): Added in v1.1 Boolean. Set to True if you want to skip tests where nodes are not present in any of the specified xpaths in test. xpath is used here in the general sense and doesn't refer to the above mentioned attribute. Can be used at two levels -- the topmost test level as well as the ETC level. Refer Example 5 and Example 6 for more info. The ignore-null specified at the ETC level takes precedence over the test level.

Examples

Example 1: Sample test file1 (test_is_equal.yml)

test_interfaces_terse:
  - command: show interfaces terse lo* 
  - item:
      id: ./name
      xpath: //physical-interface[normalize-space(name) = "lo0"]
      tests:
        - is-equal: admin-status, down
          info: "Test Succeeded !! admin-status is equal to post: <{{post['admin-status']}}>  pre: <{{pre['admin-status']}}> with oper-status pre:<{{pre['oper-status']}}> post:<{{post['oper-status']}}>"
          err: "Test Failed !! admin-status is not equal to down, it is post: <{{post['admin-status']}}>  pre:<{{pre['admin-status']}}> with oper-status <{{pre['oper-status']}}>"

Example 2: Sample test file2 (multiple iterators and tests for single command)

tests_include:
  - test_multiple_tests
  - test_multiple_iter

test_multiple_tests:
  - command: show interfaces terse lo* 
  - item:
      id: ./name
      xpath: //physical-interface[normalize-space(name) = "lo0"]
      tests:
        - is-equal: admin-status, down
          info: "Test Succeeded !! admin-status is equal to <{{post['admin-status']}}>  <{{pre['admin-status']}}> with oper-status <{{pre['oper-status']}}>"
          err: "Test Failed !! admin-status is not equal to down, it is <{{post['admin-status']}}> <{{pre['admin-status']}}> with oper-status <{{pre['oper-status']}}>"
        - is-in: oper-status, downoo, up
          info: "Test Succeeded!! Physical operational status is-in downoo-up, it is: <{{post['oper-status']}}> with admin status <{{post['admin-status']}}>"
          err: "Test Failed!!! Physical operational status is not in downoo-up, it is: <{{post['oper-status']}}> with admin status <{{post['admin-status']}}> "

test_multiple_iter:
  - command: show interfaces terse lo* 
  - item:
      id: ./name
      xpath: //physical-interface[normalize-space(name) = "lo0"]
      tests:
        - is-equal: admin-status, down
          info: "Test Succeeded !! admin-status is equal to <{{post['admin-status']}}>  <{{pre['admin-status']}}> with oper-status <{{pre['oper-status']}}>"
          err: "Test Failed !! admin-status is not equal to down, it is <{{post['admin-status']}}> <{{pre['admin-status']}}> with oper-status <{{pre['oper-status']}}>"

  - iterate:
      xpath: //physical-interface[normalize-space(name) = "lo0"]
      tests:
        - is-in: oper-status, down, up
          info: "Test Succeeded!! Physical operational status is-in downoo-up, it is: <{{post['oper-status']}}> with admin status <{{post['admin-status']}}>"
          err: "Test Failed!!! Physical operational status is not in downoo-up, it is: <{{post['oper-status']}}> with admin status <{{post['admin-status']}}> "

Example 3: Sample test file3 (test_bgp_neighbor.yml)

tests_include:
  - test_command_bgp
  - test_rpc_bgp

test_command_bgp:
  - command: show bgp neighbor
  - iterate:
      xpath: '/bgp-information/bgp-peer'
      tests:
        - is-equal: peer-address, 10.209.19.203       # element in which test is performed
          err: "Test Failed!! peer-address got changed, it is now <{{post['peer-address']}}>"
          info: "Test succeeded!! peer-address is equal to <{{post['peer-address']}}>"

        - in-range: peer-as, 100,900        # element in which test is performed
          err: "Test Failed!! peer-as is not in range of 100-200, it is: <{{post['peer-as']}}>"
          info: "Test succeeded!! peer-as is in range of 100-200, it is now <{{post['peer-as']}}>"

  - iterate:
      xpath: '//bgp-information/bgp-peer/bgp-option-information'
      tests:
        - is-gt: holdtime, 10       # element in which test is performed
          err: "Test Failed!! holdtime is not greater than 10, it is: <{{post['holdtime']}}>"
          info: "Test succeeded!! holdtime is greater than 10, it is: <{{post['holdtime']}}>"

        - is-lt: preference, 200      # element in which test is performed
          err: "Test Failed!! preference is not less than 10, <{{post['preference']}}>"
          info: "Test succeeded!! preference is less than 10, <{{post['preference']}}>"

test_rpc_bgp:
  - rpc: get-bgp-neighbor-information
  - iterate:
      xpath: '//bgp-information/bgp-peer'
      tests:
        - not-equal: last-state,Idle       # element in which test is performed
          err: "Test Failed!! last state is <{{post['last-state']}}>"
          info: "Test succeeded!! last state is not equal to idle, it is: <{{post['last-state']}}>"

        - all-same: flap-count
          err: "Test Failed!!! flap count are not all same!!, it is <{{post['flap-count']}}> "
          info: "Test Succeeded!! flap count are all same, it is now <{{post['flap-count']}}>!!!"

        - is-equal: flap-count, 0
          err: "Test Failed!!! flap count is not equal to 0, it is: <{{post['flap-count']}}> "
          info: "Test Succeeded!! flap count is equal to <{{post['flap-count']}}> !!"

Example 4: Sample test file containing multiple IDs

For defining multiple ids, there are two ways

a. make a list of ids

 id: [name, ../peer-address]

b. mention all ids in one line, as comma separated string

 id: name, ../peer-address

Example: test_bgp_summary.yml

bgp-summary:
- command: show bgp summary
- iterate:
    id: name, ../peer-address 
    tests:
    - delta: active-prefix-count, 20%
      err: ' ERROR: The number of active prefix of the BGP Table have changed more than 20%. name: {{id_0}} and peer-address: {{id_1}} [Before = {{pre["active-prefix-count"]}} / After = {{post["active-prefix-count"]}}]'
      info: 'Checking BGP peer active prefix count (tolerance 20%) name is <{{id_0}}> and peer-address: {{id_1}} pre:  {{pre["active-prefix-count"]}} post:{{post["active-prefix-count"]}}'
    xpath: '/bgp-information/bgp-peer/bgp-rib'

Example 5: ignore-null example:

tests_include:
  - check_chassis_fpc

check_chassis_fpc:
  - command: show chassis fpc
  - ignore-null: True
  - iterate: 
      xpath: //fpc[normalize-space(slot) = "0"]
      tests:
        - is-lt: cpu-total, 8
          info: "Test Succeeded!! cpu total is less than 1 and its value is {{post['cpu-total']}}"
          err: "Test Failed!!! cpu total is greater than 1 and its value is {{post['cpu-total']}}"
        - is-gt: beta-cpu-total, 2
          info: "Test Succeeded!! cpu total is less than 1 and its value is {{post['cpu-total']}}"
          err: "Test Failed!!! cpu total is greater than 1 and its value is {{post['cpu-total']}}"

Example 6: ignore-null example 2:

tests_include:
  - check_chassis_fpc

check_chassis_fpc:
  - command: show chassis fpc
  - iterate: 
      xpath: //fpc[normalize-space(slot) = "0"]
      tests:
        - is-lt: cpu-total, 8
          info: "Test Succeeded!! cpu total is less than 1 and its value is {{post['cpu-total']}}"
          err: "Test Failed!!! cpu total is greater than 1 and its value is {{post['cpu-total']}}" 
        - is-gt: beta-cpu-total, 2
          info: "Test Succeeded!! cpu total is less than 1 and its value is {{post['cpu-total']}}"
          err: "Test Failed!!! cpu total is greater than 1 and its value is {{post['cpu-total']}}"
          ignore-null: True
Clone this wiki locally