Skip to content

Commit

Permalink
Merge dev into main (Release v1.2) (#326)
Browse files Browse the repository at this point in the history
* Implement test orchestrator (#4)

* Initial work on test-orchestrator

* Ignore runtime folder

* Update runtime directory for test modules

* Fix logging
Add initial framework for running tests

* logging and misc cleanup

* logging changes

* Add a stop hook after all tests complete

* Refactor test_orc code

* Add arg passing
Add option to use locally cloned via install or remote via main project network orchestrator

* Fix baseline module
Fix orchestrator exiting only after timeout

* Add result file to baseline test module
Change result format to match closer to design doc

* Refactor pylint

* Skip test module if it failed to start

* Refactor

* Check for valid log level

---------

Co-authored-by: Jacob Boddey <[email protected]>

* Add issue report templates (#7)

* Add issue templates

* Update README.md

* Discover devices on the network (#5)

* Test run sync (#8)

* Initial work on test-orchestrator

* Ignore runtime folder

* Update runtime directory for test modules

* Fix logging
Add initial framework for running tests

* logging and misc cleanup

* logging changes

* Add a stop hook after all tests complete

* Refactor test_orc code

* Add arg passing
Add option to use locally cloned via install or remote via main project network orchestrator

* Fix baseline module
Fix orchestrator exiting only after timeout

* Add result file to baseline test module
Change result format to match closer to design doc

* Refactor pylint

* Skip test module if it failed to start

* Refactor

* Check for valid log level

* Add config file arg
Misc changes to network start procedure

* fix merge issues

* Update runner and test orch procedure
Add useful runtiem args

* Restructure test run startup process
Misc updates to work with net orch updates

* Refactor

---------

* Quick refactor (#9)

* Fix duplicate sleep calls

* Add net orc (#11)

* Add network orchestrator repository

* cleanup duplicate start and install scripts

* Temporary fix for python dependencies

* Remove duplicate python requirements

* remove duplicate conf files

* remove remote-net option

* cleanp unecessary files

* Add the DNS test module (#12)

* Add network orchestrator repository

* cleanup duplicate start and install scripts

* Temporary fix for python dependencies

* Remove duplicate python requirements

* remove duplicate conf files

* remove remote-net option

* cleanp unecessary files

* Add dns test module
Fix test module build process

* Add mac address of device under test to test container
Update dns test to use mac address filter

* Update dns module tests

* Change result output

* logging update

* Update test module for better reusability

* Load in module config to test module

* logging cleanup

* Update baseline module to new template
Misc cleanup

* Add ability to disable individual tests

* remove duplicate readme

* Update device directories

* Remove local folder

* Update device template
Update test module to work with new device config file format

* Change test module network config options
Do not start network services for modules not configured for network

* Refactor

---------

* Add baseline and pylint tests (#25)

* Discover devices on the network (#22)

* Discover devices on the network

* Add defaults when missing from config
Implement monitor wait period from config

* Add steady state monitor
Remove duplicate callback registrations

* Load devices into network orchestrator during testrun start

---------

Co-authored-by: jhughesbiot <[email protected]>

* Build dependencies first (#21)

* Build dependencies first

* Remove debug message

* Add depend on option to test modules

* Re-add single interface option

* Import subprocess

---------

Co-authored-by: jhughesbiot <[email protected]>

* Port scan test module (#23)

* Add network orchestrator repository

* cleanup duplicate start and install scripts

* Temporary fix for python dependencies

* Remove duplicate python requirements

* remove duplicate conf files

* remove remote-net option

* cleanp unecessary files

* Add dns test module
Fix test module build process

* Add mac address of device under test to test container
Update dns test to use mac address filter

* Update dns module tests

* Change result output

* logging update

* Update test module for better reusability

* Load in module config to test module

* logging cleanup

* Update baseline module to new template
Misc cleanup

* Add ability to disable individual tests

* remove duplicate readme

* Update device directories

* Remove local folder

* Update device template
Update test module to work with new device config file format

* Change test module network config options
Do not start network services for modules not configured for network

* Initial nmap test module add
Add device ip resolving to base module
Add network mounting for test modules

* Update ipv4 device resolving in test modules

* Map in ip subnets and remove hard coded references

* Add ftp port test

* Add ability to pass config for individual tests within a module
Update nmap module scan to run tests based on config

* Add full module check for compliance

* Add all tcp port scans to config

* Update nmap commands to match existing DAQ tests
Add udp scanning and tests

* logging cleanup

* Update TCP port scanning range
Update logging

* Merge device config into module config
Update device template

* fix merge issues

* Update timeouts
Add multi-threading for multiple scanns to run simultaneously
Add option to use scan scripts for services

* Fix merge issues

* Fix device configs

* Remove unecessary files

* Cleanup duplicate properties

* Cleanup install script

* Formatting (#26)

* Fix pylint issues in net orc

* more pylint fixes

* fix listener lint issues

* fix logger lint issues

* fix validator lint issues

* fix util lint issues

* Update base network module linting issues

* Cleanup linter issues for dhcp modules
Remove old code testing code

* change to single quote delimeter

* Cleanup linter issues for ntp module

* Cleanup linter issues for radius module

* Cleanup linter issues for template module

* fix linter issues with faux-dev

* Test results (#27)

* Collect all module test results

* Fix test modules without config options

* Add timestamp to test results

* Test results (#28)

* Collect all module test results

* Fix test modules without config options

* Add timestamp to test results

* Add attempt timing and device info to test results

* Ignore disabled test containers when generating results

* Fully skip modules that are disabled

* Fix pylint test and skip internet tests so CI passes (#29)

* disable internet checks for pass

* fix pylint test

* Increase pylint score (#31)

* More formatting fixes

* More formatting fixes

* More formatting fixes

* More formatting fixes

* Misc pylint fixes
Fix test module logger

---------

Co-authored-by: jhughesbiot <[email protected]>

* Pylint (#32)

* More formatting fixes

* More formatting fixes

* More formatting fixes

* More formatting fixes

* Misc pylint fixes
Fix test module logger

* remove unused files

* more formatting

* revert breaking pylint changes

* more formatting

* fix results file

* More formatting

* ovs module formatting

---------

Co-authored-by: Jacob Boddey <[email protected]>

* Add license header (#36)

* More formatting fixes

* More formatting fixes

* More formatting fixes

* More formatting fixes

* Misc pylint fixes
Fix test module logger

* remove unused files

* more formatting

* revert breaking pylint changes

* more formatting

* fix results file

* More formatting

* ovs module formatting

* Add ovs control into network orchestrator

* Add verification methods for the base network

* Add network validation and misc logging updates

* remove ovs module

* add license header to all python files

---------

Co-authored-by: Jacob Boddey <[email protected]>
Co-authored-by: SuperJonotron <[email protected]>

* Ovs (#35)

* More formatting fixes

* More formatting fixes

* More formatting fixes

* More formatting fixes

* Misc pylint fixes
Fix test module logger

* remove unused files

* more formatting

* revert breaking pylint changes

* more formatting

* fix results file

* More formatting

* ovs module formatting

* Add ovs control into network orchestrator

* Add verification methods for the base network

* Add network validation and misc logging updates

* remove ovs module

---------

Co-authored-by: Jacob Boddey <[email protected]>
Co-authored-by: SuperJonotron <[email protected]>

* remove ovs files added back in during merge

* Nmap (#38)

* More formatting fixes

* More formatting fixes

* More formatting fixes

* More formatting fixes

* Misc pylint fixes
Fix test module logger

* remove unused files

* more formatting

* revert breaking pylint changes

* more formatting

* fix results file

* More formatting

* ovs module formatting

* Add ovs control into network orchestrator

* Add verification methods for the base network

* Add network validation and misc logging updates

* remove ovs module

* add license header to all python files

* Update tcp scans to speed up full port range scan
Add version checking
Implement ssh version checking

* Add unknown port checks
Match unknown ports to existing services
Add unknown ports without existing services to results file

---------

Co-authored-by: Jacob Boddey <[email protected]>
Co-authored-by: SuperJonotron <[email protected]>

* Create startup capture (#37)

* Connection (#40)

* Initial add of connection test module with ping test

* Update host user resolving

* Update host user resolving for validator

* add get user method to validator

* Conn mac oui (#42)

* Initial add of connection test module with ping test

* Update host user resolving

* Update host user resolving for validator

* add get user method to validator

* Add mac_oui test
Add option to return test result and details of test for reporting

* Con mac address (#43)

* Initial add of connection test module with ping test

* Update host user resolving

* Update host user resolving for validator

* add get user method to validator

* Add mac_oui test
Add option to return test result and details of test for reporting

* Add connection.mac_address test

* Dns (#44)

* Add MDNS test

* Update existing mdns logging to be more consistent with other tests

* Add startup and monitor captures

* File permissions (#45)

* Fix validator file permissions

* Fix test module permissions

* Fix device capture file permissions

* Fix device results permissions

* Add connection single ip test (#47)

* Nmap results (#49)

* Update processing of nmap results to use xml output and json conversions for stability

* Update matching with regex to prevent wrong service matches and duplicate processing for partial matches

* Update max port scan range

* Framework restructure (#50)

* Restructure framework and modules

* Fix CI paths

* Fix base module

* Add build script

* Remove build logs

* Update base and template docker files to fit the new format
Implement a template option on network modules
Fix skipping of base image build

* remove base image build in ci

* Remove group from chown

---------

Co-authored-by: jhughesbiot <[email protected]>

* Ip control (#51)

* Add initial work for ip control module

* Implement ip control module with additional cleanup methods

* Update link check to not use error stream

* Add error checking around container network configurations

* Add network cleanup for namespaces and links

* formatting

* Move config to /local (#52)

* Move config to /local

* Fix testing config

* Fix ovs_control config location

* Fix faux dev config location

* Add documentation (#53)

* Sync dev to main (#56)

* Merge dev into main (Sprint 7 and 8) (#33)

* Implement test orchestrator (#4)

* Initial work on test-orchestrator

* Ignore runtime folder

* Update runtime directory for test modules

* Fix logging
Add initial framework for running tests

* logging and misc cleanup

* logging changes

* Add a stop hook after all tests complete

* Refactor test_orc code

* Add arg passing
Add option to use locally cloned via install or remote via main project network orchestrator

* Fix baseline module
Fix orchestrator exiting only after timeout

* Add result file to baseline test module
Change result format to match closer to design doc

* Refactor pylint

* Skip test module if it failed to start

* Refactor

* Check for valid log level

---------

Co-authored-by: Jacob Boddey <[email protected]>

* Add issue report templates (#7)

* Add issue templates

* Update README.md

* Discover devices on the network (#5)

* Test run sync (#8)

* Initial work on test-orchestrator

* Ignore runtime folder

* Update runtime directory for test modules

* Fix logging
Add initial framework for running tests

* logging and misc cleanup

* logging changes

* Add a stop hook after all tests complete

* Refactor test_orc code

* Add arg passing
Add option to use locally cloned via install or remote via main project network orchestrator

* Fix baseline module
Fix orchestrator exiting only after timeout

* Add result file to baseline test module
Change result format to match closer to design doc

* Refactor pylint

* Skip test module if it failed to start

* Refactor

* Check for valid log level

* Add config file arg
Misc changes to network start procedure

* fix merge issues

* Update runner and test orch procedure
Add useful runtiem args

* Restructure test run startup process
Misc updates to work with net orch updates

* Refactor

---------

* Quick refactor (#9)

* Fix duplicate sleep calls

* Add net orc (#11)

* Add network orchestrator repository

* cleanup duplicate start and install scripts

* Temporary fix for python dependencies

* Remove duplicate python requirements

* remove duplicate conf files

* remove remote-net option

* cleanp unecessary files

* Add the DNS test module (#12)

* Add network orchestrator repository

* cleanup duplicate start and install scripts

* Temporary fix for python dependencies

* Remove duplicate python requirements

* remove duplicate conf files

* remove remote-net option

* cleanp unecessary files

* Add dns test module
Fix test module build process

* Add mac address of device under test to test container
Update dns test to use mac address filter

* Update dns module tests

* Change result output

* logging update

* Update test module for better reusability

* Load in module config to test module

* logging cleanup

* Update baseline module to new template
Misc cleanup

* Add ability to disable individual tests

* remove duplicate readme

* Update device directories

* Remove local folder

* Update device template
Update test module to work with new device config file format

* Change test module network config options
Do not start network services for modules not configured for network

* Refactor

---------

* Add baseline and pylint tests (#25)

* Discover devices on the network (#22)

* Discover devices on the network

* Add defaults when missing from config
Implement monitor wait period from config

* Add steady state monitor
Remove duplicate callback registrations

* Load devices into network orchestrator during testrun start

---------

Co-authored-by: jhughesbiot <[email protected]>

* Build dependencies first (#21)

* Build dependencies first

* Remove debug message

* Add depend on option to test modules

* Re-add single interface option

* Import subprocess

---------

Co-authored-by: jhughesbiot <[email protected]>

* Port scan test module (#23)

* Add network orchestrator repository

* cleanup duplicate start and install scripts

* Temporary fix for python dependencies

* Remove duplicate python requirements

* remove duplicate conf files

* remove remote-net option

* cleanp unecessary files

* Add dns test module
Fix test module build process

* Add mac address of device under test to test container
Update dns test to use mac address filter

* Update dns module tests

* Change result output

* logging update

* Update test module for better reusability

* Load in module config to test module

* logging cleanup

* Update baseline module to new template
Misc cleanup

* Add ability to disable individual tests

* remove duplicate readme

* Update device directories

* Remove local folder

* Update device template
Update test module to work with new device config file format

* Change test module network config options
Do not start network services for modules not configured for network

* Initial nmap test module add
Add device ip resolving to base module
Add network mounting for test modules

* Update ipv4 device resolving in test modules

* Map in ip subnets and remove hard coded references

* Add ftp port test

* Add ability to pass config for individual tests within a module
Update nmap module scan to run tests based on config

* Add full module check for compliance

* Add all tcp port scans to config

* Update nmap commands to match existing DAQ tests
Add udp scanning and tests

* logging cleanup

* Update TCP port scanning range
Update logging

* Merge device config into module config
Update device template

* fix merge issues

* Update timeouts
Add multi-threading for multiple scanns to run simultaneously
Add option to use scan scripts for services

* Fix merge issues

* Fix device configs

* Remove unecessary files

* Cleanup duplicate properties

* Cleanup install script

* Formatting (#26)

* Fix pylint issues in net orc

* more pylint fixes

* fix listener lint issues

* fix logger lint issues

* fix validator lint issues

* fix util lint issues

* Update base network module linting issues

* Cleanup linter issues for dhcp modules
Remove old code testing code

* change to single quote delimeter

* Cleanup linter issues for ntp module

* Cleanup linter issues for radius module

* Cleanup linter issues for template module

* fix linter issues with faux-dev

* Test results (#27)

* Collect all module test results

* Fix test modules without config options

* Add timestamp to test results

* Test results (#28)

* Collect all module test results

* Fix test modules without config options

* Add timestamp to test results

* Add attempt timing and device info to test results

* Ignore disabled test containers when generating results

* Fully skip modules that are disabled

* Fix pylint test and skip internet tests so CI passes (#29)

* disable internet checks for pass

* fix pylint test

* Increase pylint score (#31)

* More formatting fixes

* More formatting fixes

* More formatting fixes

* More formatting fixes

* Misc pylint fixes
Fix test module logger

---------

Co-authored-by: jhughesbiot <[email protected]>

* Pylint (#32)

* More formatting fixes

* More formatting fixes

* More formatting fixes

* More formatting fixes

* Misc pylint fixes
Fix test module logger

* remove unused files

* more formatting

* revert breaking pylint changes

* more formatting

* fix results file

* More formatting

* ovs module formatting

---------

Co-authored-by: Jacob Boddey <[email protected]>

* Add license header (#36)

* More formatting fixes

* More formatting fixes

* More formatting fixes

* More formatting fixes

* Misc pylint fixes
Fix test module logger

* remove unused files

* more formatting

* revert breaking pylint changes

* more formatting

* fix results file

* More formatting

* ovs module formatting

* Add ovs control into network orchestrator

* Add verification methods for the base network

* Add network validation and misc logging updates

* remove ovs module

* add license header to all python files

---------

Co-authored-by: Jacob Boddey <[email protected]>
Co-authored-by: SuperJonotron <[email protected]>

* Ovs (#35)

* More formatting fixes

* More formatting fixes

* More formatting fixes

* More formatting fixes

* Misc pylint fixes
Fix test module logger

* remove unused files

* more formatting

* revert breaking pylint changes

* more formatting

* fix results file

* More formatting

* ovs module formatting

* Add ovs control into network orchestrator

* Add verification methods for the base network

* Add network validation and misc logging updates

* remove ovs module

---------

Co-authored-by: Jacob Boddey <[email protected]>
Co-authored-by: SuperJonotron <[email protected]>

* remove ovs files added back in during merge

* Nmap (#38)

* More formatting fixes

* More formatting fixes

* More formatting fixes

* More formatting fixes

* Misc pylint fixes
Fix test module logger

* remove unused files

* more formatting

* revert breaking pylint changes

* more formatting

* fix results file

* More formatting

* ovs module formatting

* Add ovs control into network orchestrator

* Add verification methods for the base network

* Add network validation and misc logging updates

* remove ovs module

* add license header to all python files

* Update tcp scans to speed up full port range scan
Add version checking
Implement ssh version checking

* Add unknown port checks
Match unknown ports to existing services
Add unknown ports without existing services to results file

---------

Co-authored-by: Jacob Boddey <[email protected]>
Co-authored-by: SuperJonotron <[email protected]>

* Create startup capture (#37)

* Connection (#40)

* Initial add of connection test module with ping test

* Update host user resolving

* Update host user resolving for validator

* add get user method to validator

* Conn mac oui (#42)

* Initial add of connection test module with ping test

* Update host user resolving

* Update host user resolving for validator

* add get user method to validator

* Add mac_oui test
Add option to return test result and details of test for reporting

* Con mac address (#43)

* Initial add of connection test module with ping test

* Update host user resolving

* Update host user resolving for validator

* add get user method to validator

* Add mac_oui test
Add option to return test result and details of test for reporting

* Add connection.mac_address test

* Dns (#44)

* Add MDNS test

* Update existing mdns logging to be more consistent with other tests

* Add startup and monitor captures

* File permissions (#45)

* Fix validator file permissions

* Fix test module permissions

* Fix device capture file permissions

* Fix device results permissions

* Add connection single ip test (#47)

* Nmap results (#49)

* Update processing of nmap results to use xml output and json conversions for stability

* Update matching with regex to prevent wrong service matches and duplicate processing for partial matches

* Update max port scan range

* Framework restructure (#50)

* Restructure framework and modules

* Fix CI paths

* Fix base module

* Add build script

* Remove build logs

* Update base and template docker files to fit the new format
Implement a template option on network modules
Fix skipping of base image build

* remove base image build in ci

* Remove group from chown

---------

Co-authored-by: jhughesbiot <[email protected]>

* Ip control (#51)

* Add initial work for ip control module

* Implement ip control module with additional cleanup methods

* Update link check to not use error stream

* Add error checking around container network configurations

* Add network cleanup for namespaces and links

* formatting

* Move config to /local (#52)

* Move config to /local

* Fix testing config

* Fix ovs_control config location

* Fix faux dev config location

* Add documentation (#53)

---------

Co-authored-by: jhughesbiot <[email protected]>
Co-authored-by: jhughesbiot <[email protected]>
Co-authored-by: Noureddine <[email protected]>
Co-authored-by: SuperJonotron <[email protected]>

* Sprint 8 Hotfix (#54)

* Fix connection results.json

* Re add try/catch

* Fix log level

* Debug test module load order

* Add depends on to nmap module

* Remove logging change

---------

Co-authored-by: jhughesbiot <[email protected]>
Co-authored-by: jhughesbiot <[email protected]>
Co-authored-by: Noureddine <[email protected]>
Co-authored-by: SuperJonotron <[email protected]>

* Fix missing results on udp tests when tcp ports are also defined (#59)

* Add licence header (#61)

* Resolve merge conflict

* Add network docs (#63)

* Add network docs

* Rename to readme

* Add link to template module

* Dhcp (#64)

* Add initial work for ip control module

* Implement ip control module with additional cleanup methods

* Update link check to not use error stream

* Add error checking around container network configurations

* Add network cleanup for namespaces and links

* formatting

* initial work on adding grpc functions for dhcp tests

* rework code to allow for better usage and unit testing

* working poc for test containers and grpc client to dhcp-1

* Move grpc client code into base image

* Move grpc proto builds outside of dockerfile into module startup script

* Setup pythonpath var in test module base startup process
misc cleanup

* pylinting and logging updates

* Add python path resolving to network modules
Update grpc path to prevent conflicts
misc pylinting

* Change lease resolving method to fix pylint issue

* cleanup unit tests

* cleanup unit tests

* Add grpc updates to dhcp2 module
Update dhcp_config to deal with missing optional variables

* Add grpc updates to dhcp2 module
Update dhcp_config to deal with missing optional variables

* fix line endings

* misc cleanup

* Dhcp (#67)

* Add initial work for ip control module

* Implement ip control module with additional cleanup methods

* Update link check to not use error stream

* Add error checking around container network configurations

* Add network cleanup for namespaces and links

* formatting

* initial work on adding grpc functions for dhcp tests

* rework code to allow for better usage and unit testing

* working poc for test containers and grpc client to dhcp-1

* Move grpc client code into base image

* Move grpc proto builds outside of dockerfile into module startup script

* Setup pythonpath var in test module base startup process
misc cleanup

* pylinting and logging updates

* Add python path resolving to network modules
Update grpc path to prevent conflicts
misc pylinting

* Change lease resolving method to fix pylint issue

* cleanup unit tests

* cleanup unit tests

* Add grpc updates to dhcp2 module
Update dhcp_config to deal with missing optional variables

* Add grpc updates to dhcp2 module
Update dhcp_config to deal with missing optional variables

* fix line endings

* misc cleanup

* Move isc-dhcp-server and radvd to services
Move DHCP server monitoring and booting to python script

* Add grpc methods to interact with dhcp_server module
Update dhcp_server to control radvd server directly from calls
Fix radvd service status method

* Add updates to dhcp2 module
Update radvd service

* Add license headers

* Add connection.dhcp_address test (#68)

* Add NTP tests (#60)

* Add ntp support test

* Add extra log message

* Modify descriptions

* Pylint

* Pylint (#69)

---------

Co-authored-by: jhughesbiot <[email protected]>

* Add ipv6 tests (#65)

* Add ipv6 tests

* Check for ND_NS

* Connection private address (#71)

* Add ntp support test

* Add extra log message

* Modify descriptions

* Pylint

* formatting

* Change isc-dhcp service setup
Fix dhcpd logging
Add start and stop methods to grpc dhcp client
Add dhcp2 client
Inttial private_addr test

* Add max lease time
Add unit tests

* fix last commit

* finish initial work on test

* pylinting

* Breakup test and allow better failure reporting

* restore network after test

* Wait for device to get a lease from original dhcp range after network restore

* pylinting

* Fix ipv6 tests

---------

Co-authored-by: Jacob Boddey <[email protected]>

* fix windows line ending

* Fix python import

* move isc-dhcp service commands to their own class
update logging
pylinting

* fix dhcp1

* Initial CI testing for tests (#72)

* Fix radvd conf

* Fix individual test disable

* Add NTP Pass CI test (#76)

* add shared address test (#75)

* Fix single ip test (#58)

* Fix single ip test from detecting faux-device during validation as a failure

* remove dhcp server capture file from scan

---------

Co-authored-by: J Boddey <[email protected]>

* Merge API into dev (#70)

* Start API

* Write interfaces

* Get current configuration

* Set versions

* Add more API methods

* Correct no-ui flag

* Do not launch API on baseline test

* Move loading devices back to Test Run core

* Merge dev into api (#74)

* Merge dev into main (Add license header) (#62)

Add license header

* Add network docs (#63)

* Add network docs

* Rename to readme

* Add link to template module

* Dhcp (#64)

* Add initial work for ip control module

* Implement ip control module with additional cleanup methods

* Update link check to not use error stream

* Add error checking around container network configurations

* Add network cleanup for namespaces and links

* formatting

* initial work on adding grpc functions for dhcp tests

* rework code to allow for better usage and unit testing

* working poc for test containers and grpc client to dhcp-1

* Move grpc client code into base image

* Move grpc proto builds outside of dockerfile into module startup script

* Setup pythonpath var in test module base startup process
misc cleanup

* pylinting and logging updates

* Add python path resolving to network modules
Update grpc path to prevent conflicts
misc pylinting

* Change lease resolving method to fix pylint issue

* cleanup unit tests

* cleanup unit tests

* Add grpc updates to dhcp2 module
Update dhcp_config to deal with missing optional variables

* Add grpc updates to dhcp2 module
Update dhcp_config to deal with missing optional variables

* fix line endings

* misc cleanup

* Dhcp (#67)

* Add initial work for ip control module

* Implement ip control module with additional cleanup methods

* Update link check to not use error stream

* Add error checking around container network configurations

* Add network cleanup for namespaces and links

* formatting

* initial work on adding grpc functions for dhcp tests

* rework code to allow for better usage and unit testing

* working poc for test containers and grpc client to dhcp-1

* Move grpc client code into base image

* Move grpc proto builds outside of dockerfile into module startup script

* Setup pythonpath var in test module base startup process
misc cleanup

* pylinting and logging updates

* Add python path resolving to network modules
Update grpc path to prevent conflicts
misc pylinting

* Change lease resolving method to fix pylint issue

* cleanup unit tests

* cleanup unit tests

* Add grpc updates to dhcp2 module
Update dhcp_config to deal with missing optional variables

* Add grpc updates to dhcp2 module
Update dhcp_config to deal with missing optional variables

* fix line endings

* misc cleanup

* Move isc-dhcp-server and radvd to services
Move DHCP server monitoring and booting to python script

* Add grpc methods to interact with dhcp_server module
Update dhcp_server to control radvd server directly from calls
Fix radvd service status method

* Add updates to dhcp2 module
Update radvd service

* Add license headers

* Add connection.dhcp_address test (#68)

* Add NTP tests (#60)

* Add ntp support test

* Add extra log message

* Modify descriptions

* Pylint

* Pylint (#69)

---------

Co-authored-by: jhughesbiot <[email protected]>

* Add ipv6 tests (#65)

* Add ipv6 tests

* Check for ND_NS

* Merge dev into main (Sprint 9) (#66)

* Implement test orchestrator (#4)

* Initial work on test-orchestrator

* Ignore runtime folder

* Update runtime directory for test modules

* Fix logging
Add initial framework for running tests

* logging and misc cleanup

* logging changes

* Add a stop hook after all tests complete

* Refactor test_orc code

* Add arg passing
Add option to use locally cloned via install or remote via main project network orchestrator

* Fix baseline module
Fix orchestrator exiting only after timeout

* Add result file to baseline test module
Change result format to match closer to design doc

* Refactor pylint

* Skip test module if it failed to start

* Refactor

* Check for valid log level

---------

Co-authored-by: Jacob Boddey <[email protected]>

* Add issue report templates (#7)

* Add issue templates

* Update README.md

* Discover devices on the network (#5)

* Test run sync (#8)

* Initial work on test-orchestrator

* Ignore runtime folder

* Update runtime directory for test modules

* Fix logging
Add initial framework for running tests

* logging and misc cleanup

* logging changes

* Add a stop hook after all tests complete

* Refactor test_orc code

* Add arg passing
Add option to use locally cloned via install or remote via main project network orchestrator

* Fix baseline module
Fix orchestrator exiting only after timeout

* Add result file to baseline test module
Change result format to match closer to design doc

* Refactor pylint

* Skip test module if it failed to start

* Refactor

* Check for valid log level

* Add config file arg
Misc changes to network start procedure

* fix merge issues

* Update runner and test orch procedure
Add useful runtiem args

* Restructure test run startup process
Misc updates to work with net orch updates

* Refactor

---------

* Quick refactor (#9)

* Fix duplicate sleep calls

* Add net orc (#11)

* Add network orchestrator repository

* cleanup duplicate start and install scripts

* Temporary fix for python dependencies

* Remove duplicate python requirements

* remove duplicate conf files

* remove remote-net option

* cleanp unecessary files

* Add the DNS test module (#12)

* Add network orchestrator repository

* cleanup duplicate start and install scripts

* Temporary fix for python dependencies

* Remove duplicate python requirements

* remove duplicate conf files

* remove remote-net option

* cleanp unecessary files

* Add dns test module
Fix test module build process

* Add mac address of device under test to test container
Update dns test to use mac address filter

* Update dns module tests

* Change result output

* logging update

* Update test module for better reusability

* Load in module config to test module

* logging cleanup

* Update baseline module to new template
Misc cleanup

* Add ability to disable individual tests

* remove duplicate readme

* Update device directories

* Remove local folder

* Update device template
Update test module to work with new device config file format

* Change test module network config options
Do not start network services for modules not configured for network

* Refactor

---------

* Add baseline and pylint tests (#25)

* Discover devices on the network (#22)

* Discover devices on the network

* Add defaults when missing from config
Implement monitor wait period from config

* Add steady state monitor
Remove duplicate callback registrations

* Load devices into network orchestrator during testrun start

---------

Co-authored-by: jhughesbiot <[email protected]>

* Build dependencies first (#21)

* Build dependencies first

* Remove debug message

* Add depend on option to test modules

* Re-add single interface option

* Import subprocess

---------

Co-authored-by: jhughesbiot <[email protected]>

* Port scan test module (#23)

* Add network orchestrator repository

* cleanup duplicate start and install scripts

* Temporary fix for python dependencies

* Remove duplicate python requirements

* remove duplicate conf files

* remove remote-net option

* cleanp unecessary files

* Add dns test module
Fix test module build process

* Add mac address of device under test to test container
Update dns test to use mac address filter

* Update dns module tests

* Change result output

* logging update

* Update test module for better reusability

* Load in module config to test module

* logging cleanup

* Update baseline module to new template
Misc cleanup

* Add ability to disable individual tests

* remove duplicate readme

* Update device directories

* Remove local folder

* Update device template
Update test module to work with new device config file format

* Change test module network config options
Do not start network services for modules not configured for network

* Initial nmap test module add
Add device ip resolving to base module
Add network mounting for test modules

* Update ipv4 device resolving in test modules

* Map in ip subnets and remove hard coded references

* Add ftp port test

* Add ability to pass config for individual tests within a module
Update nmap module scan to run tests based on config

* Add full module check for compliance

* Add all tcp port scans to config

* Update nmap commands to match existing DAQ tests
Add udp scanning and tests

* logging cleanup

* Update TCP port scanning range
Update logging

* Merge device config into module config
Update device template

* fix merge issues

* Update timeouts
Add multi-threading for multiple scanns to run simultaneously
Add option to use scan scripts for services

* Fix merge issues

* Fix device configs

* Remove unecessary files

* Cleanup duplicate properties

* Cleanup install script

* Formatting (#26)

* Fix pylint issues in net orc

* more pylint fixes

* fix listener lint issues

* fix logger lint issues

* fix validator lint issues

* fix util lint issues

* Update base network module linting issues

* Cleanup linter issues for dhcp modules
Remove old code testing code

* change to single quote delimeter

* Cleanup linter issues for ntp module

* Cleanup linter issues for radius module

* Cleanup linter issues for template module

* fix linter issues with faux-dev

* Test results (#27)

* Collect all module test results

* Fix test modules without config options

* Add timestamp to test results

* Test results (#28)

* Collect all module test results

* Fix test modules without config options

* Add timestamp to test results

* Add attempt timing and device info to test results

* Ignore disabled test containers when generating results

* Fully skip modules that are disabled

* Fix pylint test and skip internet tests so CI passes (#29)

* disable internet checks for pass

* fix pylint test

* Increase pylint score (#31)

* More formatting fixes

* More formatting fixes

* More formatting fixes

* More formatting fixes

* Misc pylint fixes
Fix test module logger

---------

Co-authored-by: jhughesbiot <[email protected]>

* Pylint (#32)

* More formatting fixes

* More formatting fixes

* More formatting fixes

* More formatting fixes

* Misc pylint fixes
Fix test module logger

* remove unused files

* more formatting

* revert breaking pylint changes

* more formatting

* fix results file

* More formatting

* ovs module formatting

---------

Co-authored-by: Jacob Boddey <[email protected]>

* Add license header (#36)

* More formatting fixes

* More formatting fixes

* More formatting fixes

* More formatting fixes

* Misc pylint fixes
Fix test module logger

* remove unused files

* more formatting

* revert breaking pylint changes

* more formatting

* fix results file

* More formatting

* ovs module formatting

* Add ovs control into network orchestrator

* Add verification methods for the base network

* Add network validation and misc logging updates

* remove ovs module

* add license header to all python files

---------

Co-authored-by: Jacob Boddey <[email protected]>
Co-authored-by: SuperJonotron <[email protected]>

* Ovs (#35)

* More formatting fixes

* More formatting fixes

* More formatting fixes

* More formatting fixes

* Misc pylint fixes
Fix test module logger

* remove unused files

* more formatting

* revert breaking pylint changes

* more formatting

* fix results file

* More formatting

* ovs module formatting

* Add ovs control into network orchestrator

* Add verification methods for the base network

* Add network validation and misc logging updates

* remove ovs module

---------

Co-authored-by: Jacob Boddey <[email protected]>
Co-authored-by: SuperJonotron <[email protected]>

* remove ovs files added back in during merge

* Nmap (#38)

* More formatting fixes

* More formatting fixes

* More formatting fixes

* More formatting fixes

* Misc pylint fixes
Fix test module logger

* remove unused files

* more formatting

* revert breaking pylint changes

* more formatting

* fix results file

* More formatting

* ovs module formatting

* Add ovs control into network orchestrator

* Add verification methods for the base network

* Add network validation and misc logging updates

* remove ovs module

* add license header to all python files

* Update tcp scans to speed up full port range scan
Add version checking
Implement ssh version checking

* Add unknown port checks
Match unknown ports to existing services
Add unknown ports without existing services to results file

---------

Co-authored-by: Jacob Boddey <[email protected]>
Co-authored-by: SuperJonotron <[email protected]>

* Create startup capture (#37)

* Connection (#40)

* Initial add of connection test module with ping test

* Update host user resolving

* Update host user resolving for validator

* add get user method to validator

* Conn mac oui (#42)

* Initial add of connection test module with ping test

* Update host user resolving

* Update host user resolving for validator

* add get user method to validator

* Add mac_oui test
Add option to return test result and details of test for reporting

* Con mac address (#43)

* Initial add of connection test module with ping test

* Update host user resolving

* Update host user resolving for validator

* add get user method to validator

* Add mac_oui test
Add option to return test result and details of test for reporting

* Add connection.mac_address test

* Dns (#44)

* Add MDNS test

* Update existing mdns logging to be more consistent with other tests

* Add startup and monitor captures

* File permissions (#45)

* Fix validator file permissions

* Fix test module permissions

* Fix device capture file permissions

* Fix device results permissions

* Add connection single ip test (#47)

* Nmap results (#49)

* Update processing of nmap results to use xml output and json conversions for stability

* Update matching with regex to prevent wrong service matches and duplicate processing for partial matches

* Update max port scan range

* Framework restructure (#50)

* Restructure framework and modules

* Fix CI paths

* Fix base module

* Add build script

* Remove build logs

* Update base and template docker files to fit the new format
Implement a template option on network modules
Fix skipping of base image build

* remove base image build in ci

* Remove group from chown

---------

Co-authored-by: jhughesbiot <[email protected]>

* Ip control (#51)

* Add initial work for ip control module

* Implement ip control module with additional cleanup methods

* Update link check to not use error stream

* Add error checking around container network configurations

* Add network cleanup for namespaces and links

* formatting

* Move config to /local (#52)

* Move config to /local

* Fix testing config

* Fix ovs_control config location

* Fix faux dev config location

* Add documentation (#53)

* Sync dev to main (#56)

* Merge dev into main (Sprint 7 and 8) (#33)

* Implement test orchestrator (#4)

* Initial work on test-orchestrator

* Ignore runtime folder

* Update runtime directory for test modules

* Fix logging
Add initial framework for running tests

* logging and misc cleanup

* logging changes

* Add a stop hook after all tests complete

* Refactor test_orc code

* Add arg passing
Add option to use locally cloned via install or remote via main project network orchestrator

* Fix baseline module
Fix orchestrator exiting only after timeout

* Add result file to baseline test module
Change result format to match closer to design doc

* Refactor pylint

* Skip test module if it failed to start

* Refactor

* Check for valid log level

---------

Co-authored-by: Jacob Boddey <[email protected]>

* Add issue report templates (#7)

* Add issue templates

* Update README.md

* Discover devices on the network (#5)

* Test run sync (#8)

* Initial work on test-orchestrator

* Ignore runtime folder

* Update runtime directory for test modules

* Fix logging
Add initial framework for running tests

* logging and misc cleanup

* logging changes

* Add a stop hook after all tests complete

* Refactor test_orc code

* Add arg passing
Add option to use locally cloned via install or remote via main project network orchestrator

* Fix baseline module
Fix orchestrator exiting only after timeout

* Add result file to baseline test module
Change result format to match closer to design doc

* Refactor pylint

* Skip test module if it failed to start

* Refactor

* Check for valid log level

* Add config file arg
Misc changes to network start procedure

* fix merge issues

* Update runner and test orch procedure
Add useful runtiem args

* Restructure test run startup process
Misc updates to work with net orch updates

* Refactor

---------

* Quick refactor (#9)

* Fix duplicate sleep calls

* Add net orc (#11)

* Add network orchestrator repository

* cleanup duplicate start and install scripts

* Temporary fix for python dependencies

* Remove duplicate python requirements

* remove duplicate conf files

* remove remote-net option

* cleanp unecessary files

* Add the DNS test module (#12)

* Add network orchestrator repository

* cleanup duplicate start and install scripts

* Temporary fix for python dependencies

* Remove duplicate python requirements

* remove duplicate conf files

* remove remote-net option

* cleanp unecessary files

* Add dns test module
Fix test module build process

* Add mac address of device under test to test container
Update dns test to use mac address filter

* Update dns module tests

* Change result output

* logging update

* Update test module for better reusability

* Load in module config to test module

* logging cleanup

* Update baseline module to new template
Misc cleanup

* Add ability to disable individual tests

* remove duplicate readme

* Update device directories

* Remove local folder

* Update device template
Update test module to work with new device config file format

* Change test module network config options
Do not start network services for modules not configured for network

* Refactor

---------

* Add baseline and pylint tests (#25)

* Discover devices on the network (#22)

* Discover devices on the network

* Add defaults when missing from config
Implement monitor wait period from config

* Add steady state monitor
Remove duplicate callback registrations

* Load devices into network orchestrator during testrun start

---------

Co-authored-by: jhughesbiot <[email protected]>

* Build dependencies first (#21)

* Build dependencies first

* Remove debug message

* Add depend on option to test modules

* Re-add single interface option

* Import subprocess

---------

Co-authored-by: jhughesbiot <[email protected]>

* Port scan test module (#23)

* Add network orchestrator repository

* cleanup duplicate start and install scripts

* Temporary fix for python dependencies

* Remove duplicate python requirements

* remove duplicate conf files

* remove remote-net option

* cleanp unecessary files

* Add dns test module
Fix test module build process

* Add mac address of device under test to test container
Update dns test to use mac address filter

* Update dns module tests

* Change result output

* logging update

* Update test module for better reusability

* Load in module config to test module

* logging cleanup

* Update baseline module to new template
Misc cleanup

* Add ability to disable individual tests

* remove duplicate readme

* Update device directories

* Remove local folder

* Update device template
Update test module to work with new device config file format

* Change test module network config options
Do not start network services for modules not configured for network

* Initial nmap test module add
Add device ip resolving to base module
Add network mounting for test modules

* Update ipv4 device resolving in test modules

* Map in ip subnets and remove hard coded references

* Add ftp port test

* Add ability to pass config for individual tests within a module
Update nmap module scan to run tests based on config

* Add full module check for compliance

* Add all tcp port scans to config

* Update nmap commands to match existing DAQ tests
Add udp scanning and tests

* logging cleanup

* Update TCP port scanning range
Update logging

* Merge device config into module config
Update device template

* fix merge issues

* Update timeouts
Add multi-threading for multiple scanns to run simultaneously
Add option to use scan scripts for services

* Fix merge issues

* Fix device configs

* Remove unecessary files

* Cleanup duplicate properties

* Cleanup install script

* Formatting (#26)

* Fix pylint issues in net orc

* more pylint fixes

* fix listener lint issues

* fix logger lint issues

* fix validator lint issues

* fix util lint issues

* Update base network module linting issues

* Cleanup linter issues for dhcp modules
Remove old code testing code

* change to single quote delimeter

* Cleanup linter issues for ntp module

* Cleanup linter issues for radius module

* Cleanup linter issues for template module

* fix linter issues with faux-dev

* Test results (#27)

* Collect all module test results

* Fix test modules without config options

* Add timestamp to test results

* Test results (#28)

* Collect all module test results

* Fix test modules without config options

* Add timestamp to test results

* Add attempt timing and device info to test results

* Ignore disabled test containers when generating results

* Fully skip modules that are disabled

* Fix pylint test and skip internet tests so CI passes (#29)

* disable internet checks for pass

* fix pylint test

* Increase pylint score (#31)

* More formatting fixes

* More formatting fixes

* More formatting fixes

* More formatting fixes

* Misc pylint fixes
Fix test module logger

---------

Co-authored-by: jhughesbiot <[email protected]>

* Pylint (#32)

* More formatting fixes

* More formatting fixes

* More formatting fixes

* More formatting fixes

* Misc pylint fixes
Fix test module logger

* remove unused files

* more formatting

* revert breaking pylint changes

* more formatting

* fix results file

* More formatting

* ovs module formatting

---------

Co-authored-by: Jacob Boddey <[email protected]>

* Add license header (#36)

* More formatting fixes

* More formatting fixes

* More formatting fixes

* More formatting fixes

* Misc pylint fixes
Fix test module logger

* remove unused files

* more formatting

* revert breaking pylint changes

* more formatting

* fix results file

* More formatting

* ovs module formatting

* Add ovs control into network orchestrator

* Add verification methods for the base network

* Add network validation and misc logging updates

* remove ovs module

* add license header to all python files

---------

Co-authored-by: Jacob Boddey <[email protected]>
Co-authored-by: SuperJonotron <[email protected]>

* Ovs (#35)

* More formatting fixes

* More formatting fixes

* More formatting fixes

* More formatting fixes

* Misc pylint fixes
Fix test module logger

* remove unused files

* more formatting

* revert breaking pylint changes

* more formatting

* fix results file

* More formatting

* ovs module formatting

* Add ovs control into network orchestrator

* Add verification methods for the base network

* Add network validation and misc logging updates

* remove ovs module

---------

Co-authored-by: Jacob Boddey <[email protected]>
Co-authored-by: SuperJonotron <[email protected]>

* remove ovs files added back in during merge

* Nmap (#38)

* More formatting fixes

* More formatting fixes

* More formatting fixes

* More formatting fixes

* Misc pylint fixes
Fix test module logger

* remove unused files

* more formatting

* revert breaking pylint changes

* more formatting

* fix results file

* More formatting

* ovs module formatting

* Add ovs control into network orchestrator

* Add verification methods for the base network

* Add network validation and misc logging updates

* remove ovs module

* add license header to all python files

* Update tcp scans to speed up full port range scan
Add version checking
Implement ssh version checking

* Add unknown port checks
Match unknown ports to existing services
Add unknown ports without existing services to results file

---------

Co-authored-by: Jacob Boddey <[email protected]>
Co-authored-by: SuperJonotron <[email protected]>

* Create startup capture (#37)

* Connection (#40)

* Initial add of connection test module with ping test

* Update host user resolving

* Update host user resolving for validator

* add get user method to validator

* Conn mac oui (#42)

* Initial add of connection test module with ping test

* Update host user resolving

* Update host user resolving for validator

* add get user method to validator

* Add mac_oui test
Add option to return test result and details of test for reporting

* Con mac address (#43)

* Initial add of connection test module with ping test

* Update host user resolving

* Update host user resolving for validator

* add get user method to validator

* Add mac_oui test
Add option to return test result and details of test for reporting

* Add connection.mac_address test

* Dns (#44)

* Add MDNS test

* Update existing mdns logging to be more consistent with other tests

* Add startup and monitor captures

* File permissions (#45)

* Fix validator file permissions

* Fix test module permissions

* Fix device capture file permissions

* Fix device results permissions

* Add connection single ip test (#47)

* Nmap results (#49)

* Update processing of nmap results to use xml output and json conversions for stability

* Update matching with regex to prevent wrong service matches and duplicate processing for partial matches

* Update max port scan range

* Framework restructure (#50)

* Restructure framework and modules

* Fix CI paths

* Fix base module

* Add build script

* Remove build logs

* Update base and template docker files to fit the new format
Implement a template option on network modules
Fix skipping of base image build

* remove base image build in ci

* Remove group from chown

---------

Co-authored-by: jhughesbiot <[email protected]>

* Ip control (#51)

* Add initial work for ip control module

* Implement ip control module with additional cleanup methods

* Update link check to not use error stream

* Add error checking around container network configurations

* Add network cleanup for namespaces and links

* formatting

* Move config to /local (#52)

* Move config to /local

* Fix testing config

* Fix ovs_control config location

* Fix faux dev config location

* Add documentation (#53)

---------

Co-authored-by: jhughesbiot <[email protected]>
Co-authored-by: jhughesbiot <[email protected]>
Co-authored-by: Noureddine <[email protected]>
Co-authored-by: SuperJonotron <[email protected]>

* Sprint 8 Hotfix (#54)

* Fix connection results.json

* Re add try/catch

* Fix log level

* Debug test module load order

* Add depends on to nmap module

* Remove logging change

---------

Co-authored-by: jhughesbiot <[email protected]>
Co-authored-by: jhughesbiot <[email protected]>
Co-authored-by: Noureddine <[email protected]>
Co-authored-by: SuperJonotron <[email protected]>

* Fix missing results on udp tests when tcp ports are also defined (#59)

* Add licence header (#61)

* Resolve merge conflict

* Add network docs (#63)

* Add network docs

* Rename to readme

* Add link to template module

* Dhcp (#64)

* Add initial work for ip control module

* Implement ip control module with additional cleanup methods

* Update link check to not use error stream

* Add error checking around container network configurations

* Add network cleanup for namespaces and links

* formatting

* initial work on adding grpc functions for dhcp tests

* rework code to allow for better usage and unit testing

* working poc for test containers and grpc client to dhcp-1

* Move grpc client code into base image

* Move grpc proto builds outside of dockerfile into module startup script

* Setup pythonpath var in test module base startup process
misc cleanup

* pylinting and logging updates

* Add python path resolving to network modules
Update grpc path to prevent conflicts
misc pylinting

* Change lease resolving method to fix pylint issue

* cleanup unit tests

* cleanup unit tests

* Add grpc updates to dhcp2 module
Update dhcp_config to deal with missing optional variables

* Add grpc updates to dhcp2 module
Update dhcp_config to deal with missing optional variables

* fix line endings

* misc cleanup

* Dhcp (#67)

* Add initial work for ip control module

* Implement ip control module with additional cleanup methods

* Update link check to not use error stream

* Add error checking around container network configurations

* Add network cleanup for namespaces and links

* formatting

* initial work on adding grpc functions for dhcp tests

* rework code to allow for better usage and unit testing

* working poc for test containers and grpc client to dhcp-1

* Move grpc client code into base image

* Move grpc proto builds outside of dockerfile into module startup script

* Setup pythonpath var in test module base startup process
misc cleanup

* pylinting and logging updates

* Add python path resolving to network modules
Update grpc path to prevent conflicts
misc pylinting

* Change lease resolving method to fix pylint issue

* cleanup unit tests

* cleanup unit tests

* Add grpc updates to dhcp2 module
Update dhcp_config to deal with missing optional variables

* Add grpc updates to dhcp2 module
Update dhcp_config to deal with missing optional variables

* fix line endings

* misc cleanup

* Move isc-dhcp-server and radvd to services
Move DHCP server monitoring and booting to python script

* Add grpc methods to interact with dhcp_server module
Update dhcp_server to control radvd server directly from calls
Fix radvd service status method

* Add updates to dhcp2 module
Update radvd service

* Add license headers

* Add connection.dhcp_address test (#68)

* Add NTP tests (#60)

* Add ntp support test

* Add extra log message

* Modify descriptions

* Pylint

* Pylint (#69)

---------

Co-authored-by: jhughesbiot <[email protected]>

* Add ipv6 tests (#65)

* Add ipv6 tests

* Check for ND_NS

---------

Co-authored-by: jhughesbiot <[email protected]>
Co-authored-by: jhughesbiot <[email protected]>
Co-authored-by: Noureddine <[email protected]>
Co-authored-by: SuperJonotron <[email protected]>

* Connection private address (#71)

* Add ntp support test

* Add extra log message

* Modify descriptions

* Pylint

* formatting

* Change isc-dhcp service setup
Fix dhcpd logging
Add start and stop methods to grpc dhcp client
Add dhcp2 client
Inttial private_addr test

* Add max lease time
Add unit tests

* fix last commit

* finish initial work on test

* pylinting

* Breakup test and allow better failure reporting

* restore network after test

* Wait for device to get a lease from original dhcp range after network restore

* pylinting

* Fix ipv6 tests

---------

Co-authored-by: Jacob Boddey <[email protected]>

* fix windows line ending

* Fix python import

* move isc-dhcp service commands to their own class
update logging
pylinting

* fix dhcp1

* Initial CI testing for tests (#72)

* Fix radvd conf

---------

Co-authored-by: jhughesbiot <[email protected]>
Co-authored-by: jhughesbiot <[email protected]>
Co-authored-by: Noureddine <[email protected]>
Co-authored-by: SuperJonotron <[email protected]>

* Fix testing command

* Disable API on testing

* Add API session

* Remove old method

* Remove local vars

* Replace old var

* Add device config

* Add device configs

* Fix paths

* Change MAC address

* Revert mac

* Fix copy path

* Debug loading devices

* Remove reference

* Changes

* Re-add checks to prevent null values

* Fix variable

* Fix

* Use dict instead of string

* Try without json conversion

* Container output to log

* Undo changes to nmap module

* Add post devices route

---------

Co-authored-by: jhughesbiot <[email protected]>
Co-authored-by: jhughesbiot <[email protected]>
Co-authored-by: Noureddine <[email protected]>
Co-authored-by: SuperJonotron <[email protected]>

* Dhcp tests (#81)

* Separate dhcp control methods into their own module
Implement ip change test
Add place holder for dhcp failover test

* Stabilize network before leaving ip_change test
Add dhcp_failover test

* fix regression issue with individual test enable/disable setting

* fix gitignore

* Merge tls tests into dev (#80)

* initial add of security module and tls tests

* Fix server test and implement 1.3 version

* pylinting

* More work on client tests

* Add client tls tests
Add unit tets
Add common python code to base test module

* re-enable dhcp unit tests disabled…
  • Loading branch information
9 people authored Apr 6, 2024
1 parent 254effd commit ef210eb
Show file tree
Hide file tree
Showing 245 changed files with 229,625 additions and 8,140 deletions.
27 changes: 27 additions & 0 deletions .github/workflows/package.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
name: Testrun package

on:
pull_request:
branches:
- '**'
push:
branches:
- 'dev'

jobs:
testrun_package:
permissions: {}
name: Package
runs-on: ubuntu-22.04
timeout-minutes: 5
steps:
- name: Checkout source
uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1
- name: Package Testrun
shell: bash {0}
run: cmd/package
- name: Archive package
uses: actions/upload-artifact@694cdabd8bdb0f10b2cea11669e1bf5453eed0a6 # v4.2.0
with:
name: testrun_package
path: testrun*.deb
84 changes: 25 additions & 59 deletions .github/workflows/testing.yml
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
name: Testrun test suite
permissions: {}

on:
pull_request:
Expand All @@ -17,48 +18,13 @@ jobs:
- name: Install dependencies
shell: bash {0}
run: cmd/prepare
- name: Package Testrun
shell: bash {0}
run: cmd/package
- name: Install Testrun
shell: bash {0}
timeout-minutes: 10
run: sudo dpkg -i testrun*.deb
run: TESTRUN_DIR=. cmd/install
- name: Run baseline tests
shell: bash {0}
run: testing/baseline/test_baseline

testrun_tests:
permissions: {}
name: Tests
runs-on: ubuntu-20.04
needs: testrun_baseline
timeout-minutes: 45
steps:
- name: Checkout source
uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1
- name: Install dependencies
shell: bash {0}
run: cmd/prepare
- name: Package Testrun
shell: bash {0}
run: cmd/package
- name: Install Testrun
shell: bash {0}
run: sudo dpkg -i testrun*.deb
- name: Run tests
shell: bash {0}
run: testing/tests/test_tests
- name: Archive runtime results
if: ${{ always() }}
run: sudo tar --exclude-vcs -czf runtime.tgz /usr/local/testrun/runtime/
- name: Upload runtime results
uses: actions/upload-artifact@694cdabd8bdb0f10b2cea11669e1bf5453eed0a6 # v4.2.0
if: ${{ always() }}
with:
if-no-files-found: error
name: runtime_tests_${{ github.run_id }}
path: runtime.tgz

testrun_api:
permissions: {}
Expand All @@ -71,19 +37,16 @@ jobs:
- name: Install dependencies
shell: bash {0}
run: cmd/prepare
- name: Package Testrun
shell: bash {0}
run: cmd/package
- name: Install Testrun
shell: bash {0}
run: sudo dpkg -i testrun*.deb
run: TESTRUN_DIR=. cmd/install
timeout-minutes: 30
- name: Run tests
shell: bash {0}
run: testing/api/test_api
- name: Archive runtime results
if: ${{ always() }}
run: sudo tar --exclude-vcs -czf runtime.tgz /usr/local/testrun/runtime/ /usr/local/testrun/local/
run: sudo tar --exclude-vcs -czf runtime.tgz runtime/ local/
- name: Upload runtime results
uses: actions/upload-artifact@694cdabd8bdb0f10b2cea11669e1bf5453eed0a6 # v4.2.0
if: ${{ always() }}
Expand All @@ -104,23 +67,6 @@ jobs:
shell: bash {0}
run: testing/pylint/test_pylint

testrun_package:
permissions: {}
name: Package
runs-on: ubuntu-22.04
timeout-minutes: 5
steps:
- name: Checkout source
uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1
- name: Package Testrun
shell: bash {0}
run: cmd/package
- name: Archive package
uses: actions/upload-artifact@694cdabd8bdb0f10b2cea11669e1bf5453eed0a6 # v4.2.0
with:
name: testrun_installer
path: testrun*.deb

testrun_ui:
permissions: {}
name: UI
Expand All @@ -130,7 +76,7 @@ jobs:
- name: Install Node
uses: actions/setup-node@b39b52d1213e96004bfcb1c61a8a6fa8ab84f3e8 # v4.0.1
with:
node-version: 18.10.0
node-version: 18.13.0
- name: Install Chromium Browser
run: sudo apt install chromium-browser
- name: Install dependencies
Expand All @@ -143,3 +89,23 @@ jobs:
env:
CI: true
working-directory: ./modules/ui

linters_ui:
permissions: {}
name: ESLint
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1
- name: Install Node
uses: actions/setup-node@b39b52d1213e96004bfcb1c61a8a6fa8ab84f3e8 # v4.0.1
with:
node-version: 18.13.0
- name: Install dependencies
run: npm install && npm ci
working-directory: ./modules/ui
- name: Run ESLint
run: npm run lint
working-directory: ./modules/ui
- name: Run format check
run: npm run format
working-directory: ./modules/ui
8 changes: 7 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,13 @@ build/

# Ignore generated files from unit tests
testing/unit_test/temp/
testing/unit/dns/output/
testing/unit/nmap/output/
testing/unit/ntp/output/
testing/unit/tls/output/
testing/unit/report/output/

*.deb
make/DEBIAN/postinst
make/DEBIAN/postinst

testrun.log
16 changes: 8 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,6 @@
[![CodeQL](https://github.com/google/testrun/actions/workflows/github-code-scanning/codeql/badge.svg?branch=main)](https://github.com/google/testrun/actions/workflows/github-code-scanning/codeql)
[![Testrun test suite](https://github.com/google/testrun/actions/workflows/testing.yml/badge.svg?branch=main&event=push)](https://github.com/google/testrun/actions/workflows/testing.yml)

<strong>Disclaimer</strong>: Testrun uses Google Analytics to learn about how our users use Testrun. By installing and running Testrun, you understand and accept the Terms of Service found [here](https://policies.google.com/technologies/partner-sites).

## Introduction :wave:
Testrun automates specific test cases to verify network and security functionality in IoT devices. It is an open source tool which allows manufacturers of IP capable devices to test their devices for the purposes of Device Qualification within the BOS program.

Expand Down Expand Up @@ -42,7 +40,10 @@ When manual testing or configuration changes are required, Testrun will provide
Once you have met the hardware and software requirements, you can get started with Testrun by following the [Get started guide](docs/get_started.md).

## Roadmap :chart_with_upwards_trend:
Testrun will constantly evolve to further support end-users by automating device network behaviour against industry standards.
Testrun will constantly evolve to further support end-users by automating device network behaviour against industry standards. For further information on upcoming features, check out the [Roadmap](docs/roadmap.pdf).

## Accessibility :busts_in_silhouette:
We are proud of our tool and strive to provide an enjoyable experience for all of our users. Testrun goes through rigorous accessibility testing at each release. You can read more about [Google and Accessibility here](https://www.google.co.uk/accessibility). You are welcome to submit a new issue and provide feedback on our implementations.

## Issue reporting :triangular_flag_on_post:
If the application has come across a problem at any point during setup or use, please raise an issue under the [issues tab](https://github.com/auto-iot/test-run/issues). Issue templates exist for both bug reports and feature requests. If neither of these are appropriate for your issue, raise a blank issue instead.
Expand All @@ -51,7 +52,7 @@ If the application has come across a problem at any point during setup or use, p
The contributing requirements can be found in [CONTRIBUTING.md](CONTRIBUTING.md). In short, checkout the [Google CLA](https://cla.developers.google.com/) site to get started.

## FAQ :raising_hand:
1) I have an issue with installing/upgrading Testrun, what do I do?
1) I have an issue whilst installing/upgrading Testrun, what do I do?

Sometimes, issues may arise when installing or upgrading Testrun - this may happen due to one of many reasons due to the nature of the application. However, most of the time, it can be resolved by following a full Testrun re-install by using these commands:
- ```sudo docker system prune -a```
Expand All @@ -70,12 +71,11 @@ The contributing requirements can be found in [CONTRIBUTING.md](CONTRIBUTING.md)
- IPv6 SLAAC
- DNS
- NTPv4
- 802.1x Port Based Authentication


4) Can I run Testrun on a virtual machine?

Testrun can be virtualized if the 2x ethernet adapters are passed through to a Virtual Box VM as a USB device rather than managed network adapters. A full guide will be provided once virtualization of Testrun has been fully tested.
Testrun can be virtualized if the 2x ethernet adapters are passed through to a VirtualBox VM as a USB device rather than managed network adapters. A full guide will be provided once virtualization of Testrun has been fully tested.

5) Can I connect multiple devices to Testrun?
5) Can I connect multiple devices to Testrun?

In short, Yes you can. The way in which multiple devices could be tested simultaneously is yet to be decided. However, if you simply want to add field/peer devices during runtime (even another laptop performing manual testing) then you may connect the USB ethernet adapter to an unmanaged switch.
17 changes: 12 additions & 5 deletions bin/testrun
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@
# See the License for the specific language governing permissions and
# limitations under the License.

# Check that user is root
if [[ "$EUID" -ne 0 ]]; then
echo "Must run as root. Use sudo $0"
exit 1
Expand All @@ -22,11 +23,17 @@ fi
# Ensure that /var/run/netns folder exists
sudo mkdir -p /var/run/netns

export TESTRUNPATH=/usr/local/testrun
cd $TESTRUNPATH
# Get the directory of the running script
FILE_PATH=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )

# Create device folder if it doesn't exist
mkdir -p local/devices
# Check if Testrun was installed as a package
if [ $FILE_PATH == "/usr/bin" ] ; then
export TESTRUN_PATH="/usr/local/testrun"
else
export TESTRUN_PATH=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")"/.. && pwd)
fi

cd $TESTRUN_PATH

# Remove existing runtime data
rm -rf runtime/*
Expand All @@ -35,7 +42,7 @@ rm -rf runtime/*
source venv/bin/activate

# Set the PYTHONPATH to include the "src" directory
export PYTHONPATH="$TESTRUNPATH/framework/python/src"
export PYTHONPATH="$TESTRUN_PATH/framework/python/src"
python -u framework/python/src/core/test_runner.py $@ 2>&1 | tee testrun.log

deactivate
29 changes: 24 additions & 5 deletions cmd/build
Original file line number Diff line number Diff line change
Expand Up @@ -38,16 +38,25 @@ echo Building docker images

# Build user interface
echo Building user interface
mkdir -p build/ui
docker build -t test-run/ui -f modules/ui/ui.Dockerfile . 2>&1 | tee build/ui/ui.log
if docker build -t test-run/ui -f modules/ui/ui.Dockerfile . ; then
echo Successully built the user interface
else
echo An error occured whilst building the user interface
exit 1
fi

# Build network modules
echo Building network modules
mkdir -p build/network
for dir in modules/network/* ; do
module=$(basename $dir)
echo Building network module $module...
docker build -f modules/network/$module/$module.Dockerfile -t test-run/$module . 2>&1 | tee build/network/$module.log
if docker build -f modules/network/$module/$module.Dockerfile -t test-run/$module . ; then
echo Successfully built container for network $module
else
echo An error occured whilst building container for network module $module
exit 1
fi
done

# Build validators
Expand All @@ -56,7 +65,12 @@ mkdir -p build/devices
for dir in modules/devices/* ; do
module=$(basename $dir)
echo Building validator module $module...
docker build -f modules/devices/$module/$module.Dockerfile -t test-run/$module . 2>&1 | tee build/devices/$module.log
if docker build -f modules/devices/$module/$module.Dockerfile -t test-run/$module . ; then
echo Successfully built container for device module $module
else
echo An error occured whilst building container for device module $module
exit 1
fi
done

# Build test modules
Expand All @@ -65,7 +79,12 @@ mkdir -p build/test
for dir in modules/test/* ; do
module=$(basename $dir)
echo Building test module $module...
docker build -f modules/test/$module/$module.Dockerfile -t test-run/$module-test . 2>&1 | tee build/test/$module.log
if docker build -f modules/test/$module/$module.Dockerfile -t test-run/$module-test . ; then
echo Successfully built container for test module $module
else
echo An error occured whilst building container for test module $module
exit 1
fi
done

echo Finished building modules
30 changes: 27 additions & 3 deletions cmd/install
Original file line number Diff line number Diff line change
Expand Up @@ -16,28 +16,52 @@

echo Installing application dependencies

TESTRUN_DIR=/usr/local/testrun
# Collect command line arguments
while getopts ":l" option; do
case $option in
l) # Install Testrun in local directory
TESTRUN_DIR=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")"/.. && pwd)
esac
done

# Check if TESTRUN_DIR has been set, otherwise install in /usr/local/testrun
if [[ -z "${TESTRUN_DIR}" ]]; then
TESTRUN_DIR=/usr/local/testrun
else
TESTRUN_DIR="${TESTRUN_DIR}"
fi

echo Installing Testrun at $TESTRUN_DIR

# Create the folder if it doesn't exist
mkdir -p $TESTRUN_DIR
cd $TESTRUN_DIR

# Activate the Python virtual environment
python3 -m venv venv

source venv/bin/activate

# Install Python dependencies in virtual environment
pip3 install -r framework/requirements.txt

# Copy the default configuration
cp -n local/system.json.example local/system.json

# Set file permissions
# Set file permissions on system config
# This does not work on GitHub actions
if logname ; then
USER_NAME=$(logname)
sudo chown "$USER_NAME" local/system.json
fi

# Exit out of python virtual environment
deactivate

# Build docker images
sudo cmd/build

# Create local folders
mkdir -p local/devices
mkdir -p local/root_certs

echo Finished installing Testrun
5 changes: 4 additions & 1 deletion cmd/package
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,9 @@

MAKE_SRC_DIR=make

# Delete existing make files
rm -rf $MAKE_SRC_DIR/usr

# Copy testrun script to /bin
mkdir -p $MAKE_SRC_DIR/bin
cp bin/testrun $MAKE_SRC_DIR/bin/testrun
Expand Down Expand Up @@ -53,4 +56,4 @@ cp -r {framework,modules} $MAKE_SRC_DIR/usr/local/testrun
dpkg-deb --build --root-owner-group make

# Rename the .deb file
mv make.deb testrun_1-1-3_amd64.deb
mv make.deb testrun_1-2_amd64.deb
Loading

0 comments on commit ef210eb

Please sign in to comment.