Skip to content

Commit

Permalink
Improve model unit tests and device simulation
Browse files Browse the repository at this point in the history 
Explicit tell model developpers that unit tests for models are optional

YAML simulation files are placed under /examples/device-simulation
- A README.md explains how to generate a YAML simulation file
- The script device2yaml.rb does most of the task

A README.md explains how to write model unit tests
- New function result2file to generate the expected output of Oxidized
when runned against the YAML simulation file
- interpolate_yaml uses String#undump

YAML simulation files & model unit tests for aoscx, ios and asa
  • Loading branch information
@robertcheramy
robertcheramy committed Oct 22, 2024
1 parent 754f9b1 commit e794481
Show file tree
Hide file tree
Showing 19 changed files with 2,582 additions and 97 deletions.
98 changes: 16 additions & 82 deletions docs/Creating-Models.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -87,97 +87,31 @@ Intuitively, it is also possible to:
* Testing/validation of an updated model from the [Oxidized GitHub repo models](https://github.com/ytti/oxidized/tree/master/lib/oxidized/model) by placing an updated model in the proper location without disrupting the gem-supplied model files.

## Create unit tests for the model
> :warning model unit tests are still work in progress and need some polishing.
> :warning: model unit tests are still a work in progress and need some polishing.
If you want the model to be integrated into oxidized, you can
[submit a pull request on github](https://github.com/ytti/oxidized/pulls).
This is a greatly appreciated submission, as there are probably other users
using the same network device as you are.

A good practice for submissions is to provide a unit test for your model. This
reduces the risk that further developments don't break it, and facilitates
debugging issues without having access to a physical network device for the
model. Writing a model unit test for SSH is described in the next lines. Most
of the work is writing a YAML file with the commands and their output, the ruby
code itself is copy & paste with a few modifications. If you encounter
problems, open an issue or ask for help within the pull request.

You can have a look at the [Garderos unit test](/spec/model/garderos_spec.rb) for an example. The model unit test
consists of (at least) two files:
- a yaml file under `examples/model/`, containing the data used to simulate the network device.
- Please name your file `<model>_<hardware type>_<software_version>.yaml`, for example in the garderos unit test: [garderos_R7709_003_006_068.yaml](/examples/model/garderos_R7709_003_006_068.yaml).
- You can create multiple files in order to support multiple devices or software versions.
- You may append a comment after the software version to differentiate between two tested features (something like `garderos_R7709_003_006_068_with_ipsec.yaml`).
- a ruby script containing the tests under `spec/model/`.
- It is named `<model>_spec.rb`, for the garderos model: [garderos_spec.rb](/spec/model/garderos_spec.rb).
- The script described below is a minimal example; you can add as many tests as needed.

### YAML description to simulate the network device.
The yaml file has three sections:
- init_prompt: describing the lines send by the device before we can send a command. It may include motd banners, and mus include the first prompt.
- commands: the commands the model sends to the network device and the expected output. Do not forget the command needed to logout from the device.
- oxidized_output: the expected output of oxidized, so that you can compare it to the output generated by the unit test.

The outputs are multiline and use yaml block scalars (`|`), with the trailing \n removed (`-` after `|`). The outputs includes the echo of the given command and the next prompt. Some escape characters are interpreted, currently \n, \r, \x\<octal char number\>, \\\\

Here is a shortened example of a YAML file:
```yaml
---
# Trailing white spaces are coded as \x20 because some editors automatically remove trailing white spaces
init_prompt: |-
\e[4m\rLAB-R1234_Garderos#\e[m\x20
commands:
show system version: |-
show system version
grs-gwuz-armel/003_005_068 (Garderos; 2021-04-30 16:19:35)
\e[4m\rLAB-R1234_Garderos#\e[m\x20
# ...
exit: ""
oxidized_output: |-
# grs-gwuz-armel/003_005_068 (Garderos; 2021-04-30 16:19:35)
#\x20
# ...
```

### Model unit test
When creating the unit test, it is handy to have a specific section for testing different
prompts without testing the whole configuration. This is done by the first test in the following
example. The second tests takes the defined yaml file, runs the model against it and
compares the result against the yaml-section `oxidized_output`.

```ruby
require_relative 'model_helper'

describe 'model/Garderos' do
# For each test, we initialize oxidized to some default values
# and create a node with the model we want to test
# replace 'garderos' with your model
before(:each) do
init_model_helper
@node = Oxidized::Node.new(name: 'example.com',
input: 'ssh',
model: 'garderos')
end
A good (and optional) practice for submissions is to provide a
[unit test for your model](/spec/model). This reduces the risk that further
developments don't break it, and facilitates debugging issues without having
access to a physical network device for the model.

it 'matches different prompts' do
_('LAB-R1234_Garderos# ').must_match Garderos.prompt
end

# Name the test after the tesed HW and SW. Link to your yaml data
it 'runs on R7709 with OS 003_006_068' do
mockmodel = MockSsh.new('examples/model/garderos_R7709_003_006_068.yaml')
Net::SSH.stubs(:start).returns mockmodel

status, result = @node.run
In order to simulate the device in the unit test, you need a
[YAML simulation file](/examples/device-simulation/), have a look at the
link for an explanation on how to create one.

_(status).must_equal :success
_(result.to_cfg).must_equal mockmodel.oxidized_output
end
end
```
Creating the unit test itself is explained in
[README.md in the model unit test directory](/spec/model/README.md).

The unit tests use [minitest/spec](https://github.com/minitest/minitest?tab=readme-ov-file#specs-) and [mocha](https://github.com/freerange/mocha).
If you need more expectations for you tests, have a look at the [minitest documentation for expectations](https://docs.seattlerb.org/minitest/Minitest/Expectations.html)
Remember - producing a YAML simulation file and/or writing a unit test is
optional.
The most value comes from the YAML simulation file. The unit
test can be written by someone else, but you need access to the device for the
YAML simulation file. If you encounter problems, open an issue or ask for help
in your pull request.

## Advanced features

Expand Down
167 changes: 167 additions & 0 deletions examples/device-simulation/README.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,167 @@
# Device simulation
Oxidized supports [150+ devices](/docs/Supported-OS-Types.md).
No developer has access to all of these devices, which makes the task of
maintaining Oxidized difficult:

- issues can't be resolved because the developer has no access to the device.
- further developments can produce regressions.

In order to address this, we can simulate the devices. An example for a
simulation are the [model unit tests](/spec/model) but one could also simulate
a device within a ssh server.

The simulation of devices is currently focused on ssh-based devices. This may
be extended to other inputs like telnet or ftp in the future.

## YAML Simulation Data
The underlying data for the simulation is a [YAML](https://yaml.org/) file in
which we store all relevant information about the device. The most important
information is the responses to the commands used in the oxidized models.

The YAML simulation files are stored under
[/examples/device-simulation/yaml/](/examples/device-simulation/yaml/).

### Creating a YAML file with device2yaml.rb
A device does not only output the ASCII text we can see in the console.
It adds ANSI-escape code for nice colors, bold and underline, \r and so on.
These are key factors in prompt issues so they must be represented in the YAML
file. We use the ruby string format with interpolations like \r \e and so on.
Another important point is trailing spaces at the end of lines. Some text
editors automatically remove trailing spaces, so we code them with \x20.

Although a YAML file could be written by hand, this is quite a tedious task to
catch all the extra codes and code them into YAML. This can be
automated with the ruby script
[device2yaml.rb](/examples/device-simulation/device2yaml.rb).

`device2yaml.rb` needs ruby and the gem
[net-ssh](https://rubygems.org/gems/net-ssh/) to run. On debian, you can install
them with `sudo apt install ruby-net-ssh`

Run `device2yaml.rb` in the directory `/examples/device-simulation/`, the
online help tells you the options.
```
device-simulation$ ./device2yaml.rb
Usage: model-yaml.rb [user@]host [options]
-o, --output file Specify an output file instead of stdout
-c, --cmdset file Mandatory: specify the commands to be run
-t, --timout value Specify the idle timeout beween commands (default: 5 seconds)
-h, --help Print this help
```

- `[user@]host` specifies the user and host to connect to the device. The
password will be prompted interactively by the script. If you do not specify a
user, it will use the user executing the script.
- You must list the commands you want to run on the device in a file. Just
enter one command per line. It is important that you enter exactly the commands
used by the oxidized model, and no abbreviation like `sh run`. Do not forget
to insert the `post_login` commands at the beginning if the model has some and
also the `pre_logout`commands at the end.
Predefined command sets for some models are stored in
`/examples/device-simulation/cmdsets`.
- `device2yaml.rb` waits an idle timeout after the last received data before
sending the next command. The default is 5 seconds. If your device makes a
longer pause than 5 seconds before or within a command, you will see that the
output of the command is shortened or slips into the next command in the yaml
file. You will have to change the idle timeout to a greater value to address
this.
- When run without the output argument, `device2yaml.rb` will only print the ssh
output to the standard output. You must use `-o <model_HW_SW.yaml>` to store the
collected data in a YAML file.

Note that `device2yaml.rb` takes some time to run because of the idle
timeout of (default) 5 seconds between each command. You can press the "Escape"
key if you know there is no more data to come for the current command (when you
see the prompt for the next command), and the script will stop waiting and
directly process the next command.

Here is an example of how to run the script:
```
./device2yaml.rb OX-SW123.sample.domain -c cmdsets/aoscx -o yaml/aoscx_R8N85A-C6000-48G-CL4_PL.10.08.1010.yaml
```

### Publishing the YAML simulation file to oxidized
Publishing the YAML simulation file of your device helps maintain oxidized.
This task may take some time, and we are very grateful that you take this time
for the community!

You should pay attention to removing or replacing anything you don't want to
share with the rest of the world, for example:

- Passwords
- IP Adresses
- Serial numbers

You can also shorten the configuration if you want - we don't need 48 times the
same config for each interface, but it doesn't hurt either.

Take your time, this is an important task: after you have
uploaded your file on github, it may be impossible to remove it. You can use
search/replace to make consistent and faster changes (change the hostname).

You can leave the section `oxidized_output` unchanged, it is only used for
[model unit tests](/spec/model). You will find an explanation of how to produce
the `oxidized_output`-section in the README.md there.

The YAML simulation file should be stored under
[/examples/device-simulation/yaml/](/examples/device-simulation/yaml/. It
should be named so that it can be easily recognized: model, hardware type,
software version and optionally a description if you need to differentiate two
YAML files:

- #model_#hardware_#software.yaml
- #model_#hardware_#software_#description.yaml

Examples:

- garderos_R7709_003_006_068.yaml
- iosxe_C9200L-24P-4G_17.09.04a.yaml
- asa_5512_9.12-4-67_single-context.yaml

### Interactive mode
The `device2yaml.rb` script is a little dumb and needs some help, especially
when having a device sending its output page by page and requiring you to press
space for the next page. `device2yaml.rb` does not know how to handle this.

While `device2yaml.rb` is running, you can type anything to the keyboard, it
will be send to the remote device. So you can press space or 'n' to get the
next page.

You can also use this to enter an enable password.

If you press the "Esc" key, `device2yaml.rb` will not wait for the idle timeout
and will process the next command right away.

### YAML Format
The yaml file has three sections:
- init_prompt: describing the lines send by the device before we can send a
command. It usually includes MOTD banners, and must include the first prompt.
- commands: the commands the oxidized model sends to the network device and the
expected output.
- oxidized_output: the expected output of oxidized, so that you can compare it
to the output generated by the unit test. This is optional and only used for
unit tests.

The outputs are multiline and use YAML block scalars (`|`), with the trailing \n
removed (`-` after `|`). The outputs include the echo of the given command and
the next prompt. Escape characters are coded in Ruby style (\n, \r...).

Here is a shortened example of a YAML file:
```yaml
---
init_prompt: |-
\e[4m\rLAB-R1234_Garderos#\e[m\x20
commands:
show system version: |-
show system version
grs-gwuz-armel/003_005_068 (Garderos; 2021-04-30 16:19:35)
\e[4m\rLAB-R1234_Garderos#\e[m\x20
# ...
exit: ""
oxidized_output: |
# grs-gwuz-armel/003_005_068 (Garderos; 2021-04-30 16:19:35)
#\x20
# ...
```


10 changes: 10 additions & 0 deletions examples/device-simulation/cmdsets/aoscx
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
no page
show version
show environment
show module
show interface transceiver
show system | exclude "Up Time" | exclude "CPU" | exclude "Memory" | exclude "Pkts .x" | exclude "Lowest" | exclude "Missed"
show running-config
# commands beyond the oxidized model
show system
exit
7 changes: 7 additions & 0 deletions examples/device-simulation/cmdsets/asa
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
enable
terminal pager 0
show mode
show version
show inventory
more system:running-config
exit
7 changes: 7 additions & 0 deletions examples/device-simulation/cmdsets/ios
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
terminal length 0
terminal width 0
show version
show vtp status
show inventory
show running-config
exit
Loading

0 comments on commit e794481

Please sign in to comment.